2024-04-26T17:37:24.440 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T17:37:24.441 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} --name leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi --verbose --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895 -- /home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895/orig.config.yaml 2024-04-26T17:37:24.504 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T17:37:24.618 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895 branch: wip-lusov-quiescer-fixes description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} email: null first_in_suite: false job_id: '7674895' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer-fixes ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 0 mountpoint: /tmp/cephfs check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 1 mountpoint: /tmp/cephfs mntopts: - ms_mode=secure - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer-fixes sha1: bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 9496 sha1: bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 sleep_before_teardown: 0 subset: 200/111111 suite: fs suite_branch: wip-lusov-quiescer-fixes suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKchA0vnuabghk5dSl6KV2FE+12Bwgp17BdGnkCb0tTWzpmNszg1u7Q7j8iJHp4Bhb4V/7TrUY4qk0M6mr0npEo= smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKjBYkFsJUrWXkYzp5wxoasVGvBqd1aYVjEqZ3dBP5RYMX2/SmE5wNdWfKxA7COTjobTnJWa0WsLXiZ2h3+7M/s= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAANIvgyydTKYZ2uktji9hiKZsB4kJq9b3D4reCbloMNIgZoe1zzYPTwotBeHaFW30GdAiHd2UNK0aEqUJ3ly3Q= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 32 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_17:24:23 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T17:37:24.618 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/qa; will attempt to use it 2024-04-26T17:37:24.619 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/qa/tasks 2024-04-26T17:37:24.619 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T17:37:24.621 INFO:teuthology.task.internal:Checking packages... 2024-04-26T17:37:24.646 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1' 2024-04-26T17:37:24.646 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T17:37:24.646 INFO:teuthology.packaging:ref: None 2024-04-26T17:37:24.646 INFO:teuthology.packaging:tag: None 2024-04-26T17:37:24.646 INFO:teuthology.packaging:branch: wip-lusov-quiescer-fixes 2024-04-26T17:37:24.646 INFO:teuthology.packaging:sha1: bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T17:37:24.646 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer-fixes 2024-04-26T17:37:24.798 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3267.gbc94a2a9 2024-04-26T17:37:24.800 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T17:37:24.810 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T17:37:24.811 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T17:37:24.839 INFO:teuthology.task.internal:Saving configuration 2024-04-26T17:37:24.863 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T17:37:24.874 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T17:37:24.896 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 17:32:23.309686', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKchA0vnuabghk5dSl6KV2FE+12Bwgp17BdGnkCb0tTWzpmNszg1u7Q7j8iJHp4Bhb4V/7TrUY4qk0M6mr0npEo='} 2024-04-26T17:37:24.913 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 17:32:23.313337', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKjBYkFsJUrWXkYzp5wxoasVGvBqd1aYVjEqZ3dBP5RYMX2/SmE5wNdWfKxA7COTjobTnJWa0WsLXiZ2h3+7M/s='} 2024-04-26T17:37:24.931 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 17:32:23.311679', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAANIvgyydTKYZ2uktji9hiKZsB4kJq9b3D4reCbloMNIgZoe1zzYPTwotBeHaFW30GdAiHd2UNK0aEqUJ3ly3Q='} 2024-04-26T17:37:24.931 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T17:37:24.943 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-26T17:37:24.943 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-26T17:37:24.943 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-26T17:37:24.943 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T17:37:25.052 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd86e91b0d0>, signals=[15]) 2024-04-26T17:37:25.052 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T17:37:25.058 INFO:teuthology.task.internal:Opening connections... 2024-04-26T17:37:25.058 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-04-26T17:37:25.060 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T17:37:25.139 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2024-04-26T17:37:25.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T17:37:25.213 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T17:37:25.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T17:37:25.288 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T17:37:25.295 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-04-26T17:37:25.314 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-04-26T17:37:25.314 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-04-26T17:37:25.372 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2024-04-26T17:37:25.372 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2024-04-26T17:37:25.372 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2024-04-26T17:37:25.372 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2024-04-26T17:37:25.372 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2024-04-26T17:37:25.372 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2024-04-26T17:37:25.372 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T17:37:25.372 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2024-04-26T17:37:25.372 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2024-04-26T17:37:25.372 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T17:37:25.373 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2024-04-26T17:37:25.373 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T17:37:25.373 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T17:37:25.373 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T17:37:25.373 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-04-26T17:37:25.395 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2024-04-26T17:37:25.413 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2024-04-26T17:37:25.413 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2024-04-26T17:37:25.471 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2024-04-26T17:37:25.471 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="9" 2024-04-26T17:37:25.471 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2024-04-26T17:37:25.471 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2024-04-26T17:37:25.471 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="9" 2024-04-26T17:37:25.471 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el9" 2024-04-26T17:37:25.472 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T17:37:25.472 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2024-04-26T17:37:25.472 INFO:teuthology.orchestra.run.smithi097.stdout:LOGO="fedora-logo-icon" 2024-04-26T17:37:25.472 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T17:37:25.472 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2024-04-26T17:37:25.472 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T17:37:25.472 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T17:37:25.472 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T17:37:25.472 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2024-04-26T17:37:25.495 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-04-26T17:37:25.513 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-04-26T17:37:25.513 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-04-26T17:37:25.570 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-04-26T17:37:25.570 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-04-26T17:37:25.570 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-04-26T17:37:25.570 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-04-26T17:37:25.570 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-04-26T17:37:25.570 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-04-26T17:37:25.570 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T17:37:25.570 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-04-26T17:37:25.571 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-04-26T17:37:25.571 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T17:37:25.571 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-04-26T17:37:25.571 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T17:37:25.571 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T17:37:25.571 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T17:37:25.571 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-04-26T17:37:25.595 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T17:37:25.608 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T17:37:25.617 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T17:37:25.617 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-04-26T17:37:25.620 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2024-04-26T17:37:25.622 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-04-26T17:37:25.639 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T17:37:25.647 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T17:37:25.647 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-04-26T17:37:25.678 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2024-04-26T17:37:25.680 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-04-26T17:37:25.704 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T17:37:25.768 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T17:37:25.789 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-26T17:37:25.789 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-26T17:37:25.789 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-26T17:37:25.790 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T17:37:25.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T17:37:25.807 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-04-26T17:37:25.864 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-437.el9.x86_64 2024-04-26T17:37:25.864 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.14.0-437.el9.x86_64 2024-04-26T17:37:25.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2024-04-26T17:37:28.191 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-26T17:37:32.427 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 6.1 MB/s | 19 MB 00:03 2024-04-26T17:37:36.536 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.2 MB 00:00 2024-04-26T17:37:37.953 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 16 kB 00:00 2024-04-26T17:37:39.082 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:01 2024-04-26T17:37:44.254 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-04-26T17:37:46.219 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-26T17:37:46.219 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-26T17:37:46.300 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-26T17:37:46.301 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-26T17:37:46.301 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-26T17:37:46.402 DEBUG:teuthology.orchestra.run.smithi031:> echo no | sudo yum reinstall kernel || true 2024-04-26T17:37:46.922 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:02 ago on Fri 26 Apr 2024 05:37:44 PM UTC. 2024-04-26T17:37:47.047 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-26T17:37:47.056 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-26T17:37:47.056 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-04-26T17:37:47.056 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-26T17:37:47.056 INFO:teuthology.orchestra.run.smithi031.stdout:Reinstalling: 2024-04-26T17:37:47.056 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-26T17:37:47.056 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:37:47.056 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-26T17:37:47.056 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-26T17:37:47.056 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:37:47.056 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 255 k 2024-04-26T17:37:47.057 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 0 2024-04-26T17:37:47.057 INFO:teuthology.orchestra.run.smithi031.stderr:Operation aborted. 2024-04-26T17:37:47.108 INFO:teuthology.orchestra.run.smithi031.stdout:Is this ok [y/N]: 2024-04-26T17:37:47.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum reinstall -y kernel || true 2024-04-26T17:37:47.623 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:03 ago on Fri 26 Apr 2024 05:37:44 PM UTC. 2024-04-26T17:37:47.747 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-26T17:37:47.747 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-26T17:37:47.747 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-04-26T17:37:47.747 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-26T17:37:47.747 INFO:teuthology.orchestra.run.smithi031.stdout:Reinstalling: 2024-04-26T17:37:47.747 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-26T17:37:47.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:37:47.747 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-26T17:37:47.747 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-26T17:37:47.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:37:47.748 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 255 k 2024-04-26T17:37:47.748 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 0 2024-04-26T17:37:47.748 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-26T17:37:47.907 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-437.el9.x86_64.rpm 2.7 MB/s | 255 kB 00:00 2024-04-26T17:37:47.907 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-26T17:37:47.907 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.6 MB/s | 255 kB 00:00 2024-04-26T17:37:47.927 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-26T17:37:48.137 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-26T17:37:48.137 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-26T17:37:48.142 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-26T17:37:48.143 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-26T17:37:48.274 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-26T17:37:48.357 INFO:teuthology.orchestra.run.smithi031.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-26T17:37:48.358 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T17:37:49.042 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T17:37:49.042 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-26T17:37:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T17:37:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:37:49.708 INFO:teuthology.orchestra.run.smithi031.stdout:Reinstalled: 2024-04-26T17:37:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-26T17:37:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:37:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-26T17:37:49.893 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-04-26T17:37:49.942 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-26T17:37:49.942 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-26T17:37:49.942 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-26T17:37:49.942 DEBUG:teuthology.parallel:result is None 2024-04-26T17:37:49.943 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T17:37:49.952 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T17:37:49.952 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T17:37:49.956 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T17:37:49.958 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T17:37:49.976 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T17:37:49.985 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T17:37:49.997 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T17:37:49.997 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T17:37:50.010 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T17:37:50.016 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T17:37:50.052 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T17:37:50.062 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T17:37:50.062 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T17:37:50.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T17:37:50.093 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T17:37:50.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T17:37:50.111 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T17:37:50.113 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T17:37:50.136 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T17:37:50.145 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T17:37:50.163 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T17:37:50.168 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T17:37:50.184 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T17:37:50.197 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T17:37:50.215 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T17:37:50.216 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T17:37:50.224 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T17:37:50.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T17:37:50.226 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T17:37:50.242 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T17:37:50.284 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T17:37:50.296 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T17:37:50.296 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T17:37:50.298 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T17:37:50.308 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T17:37:50.340 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T17:37:50.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T17:37:50.451 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T17:37:50.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T17:37:50.573 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:37:50.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T17:37:50.637 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T17:37:50.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T17:37:50.727 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T17:37:50.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T17:37:50.850 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-26T17:37:50.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T17:37:50.915 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T17:37:50.953 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T17:37:51.009 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T17:37:51.089 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T17:37:51.138 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T17:37:51.138 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T17:37:51.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-04-26T17:37:51.207 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2024-04-26T17:37:51.209 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-04-26T17:37:51.242 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T17:37:51.245 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T17:37:51.287 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T17:37:51.803 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T17:37:51.812 INFO:teuthology.task.internal:Starting timer... 2024-04-26T17:37:51.812 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T17:37:51.830 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T17:37:51.844 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-26T17:37:51.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2024-04-26T17:37:51.909 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2024-04-26T17:37:51.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo service auditd rotate 2024-04-26T17:37:51.961 INFO:teuthology.orchestra.run.smithi097.stdout:Rotating logs: 2024-04-26T17:37:51.963 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-04-26T17:37:52.013 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-04-26T17:37:52.015 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T17:37:52.016 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2024-04-26T17:37:52.037 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2024-04-26T17:37:52.037 DEBUG:teuthology.orchestra.run.smithi097:> /usr/sbin/getenforce 2024-04-26T17:37:52.061 INFO:teuthology.orchestra.run.smithi097.stdout:Permissive 2024-04-26T17:37:52.061 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-04-26T17:37:52.083 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-04-26T17:37:52.083 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi097.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-04-26T17:37:52.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T17:37:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T17:37:52.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T17:37:52.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T17:37:52.137 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T17:37:52.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T17:37:52.164 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T17:37:52.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2024-04-26T17:37:52.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo /usr/sbin/setenforce permissive 2024-04-26T17:37:52.225 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-04-26T17:37:52.252 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T17:37:52.263 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T17:37:52.282 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T17:37:52.283 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi097.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T17:41:43.188 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi097.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-04-26T17:41:43.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-26T17:41:43.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T17:41:43.269 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-04-26T17:41:43.350 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-26T17:41:43.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-26T17:41:43.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T17:41:43.427 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-04-26T17:41:43.510 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-26T17:41:43.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-26T17:41:43.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T17:41:43.585 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-04-26T17:41:43.661 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-26T17:41:43.662 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T17:41:43.674 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T17:41:43.675 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T17:41:43.675 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T17:41:43.678 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T17:41:43.678 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T17:41:43.681 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T17:41:43.681 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T17:41:43.708 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T17:41:43.709 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T17:41:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T17:41:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T17:41:43.730 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T17:41:43.744 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: ntpd: command not found 2024-04-26T17:41:43.745 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T17:41:43.750 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2024-04-26T17:41:43.755 INFO:teuthology.orchestra.run.smithi097.stdout:506 Cannot talk to daemon 2024-04-26T17:41:43.762 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2024-04-26T17:41:43.768 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T17:41:43.768 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-04-26T17:41:43.776 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T17:41:43.777 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-04-26T17:41:43.780 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T17:41:43.790 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T17:41:43.791 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T17:41:43.803 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T17:41:43.835 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-04-26T17:41:43.839 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T17:41:43.839 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-04-26T17:41:43.839 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T17:41:43.839 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T17:41:43.839 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T17:41:43.839 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T17:41:43.847 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-04-26T17:41:43.850 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T17:41:43.851 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-04-26T17:41:43.851 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T17:41:43.851 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T17:41:43.851 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T17:41:43.851 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T17:41:43.856 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-04-26T17:41:43.860 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T17:41:43.860 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-04-26T17:41:43.860 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T17:41:43.860 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T17:41:43.861 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T17:41:43.861 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T17:41:43.861 INFO:teuthology.run_tasks:Running task install... 2024-04-26T17:41:43.870 DEBUG:teuthology.task.install:project ceph 2024-04-26T17:41:43.870 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1'}} 2024-04-26T17:41:43.870 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1'} 2024-04-26T17:41:43.870 INFO:teuthology.task.install:Using flavor: default 2024-04-26T17:41:43.876 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T17:41:43.876 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-26T17:41:43.876 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1', 'tag': None, 'wait_for_package': False} 2024-04-26T17:41:43.877 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T17:41:43.878 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1', 'tag': None, 'wait_for_package': False} 2024-04-26T17:41:43.879 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T17:41:43.880 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1', 'tag': None, 'wait_for_package': False} 2024-04-26T17:41:43.880 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T17:41:44.021 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/ 2024-04-26T17:41:44.022 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3267.gbc94a2a9 2024-04-26T17:41:44.028 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/ 2024-04-26T17:41:44.028 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3267.gbc94a2a9 2024-04-26T17:41:44.095 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/ 2024-04-26T17:41:44.095 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3267.gbc94a2a9 2024-04-26T17:41:44.130 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T17:41:44.130 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:41:44.130 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T17:41:44.143 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T17:41:44.143 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-26T17:41:44.143 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T17:41:44.157 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T17:41:44.157 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T17:41:44.167 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T17:41:44.167 DEBUG:teuthology.orchestra.run.smithi097:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T17:41:44.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T17:41:44.232 DEBUG:teuthology.orchestra.run.smithi097:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T17:41:44.260 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T17:41:44.260 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T17:41:44.260 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T17:41:44.285 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T17:41:44.286 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T17:41:44.298 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T17:41:44.306 DEBUG:teuthology.orchestra.run.smithi097:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T17:41:44.349 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T17:41:44.362 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2024-04-26T17:41:44.364 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2024-04-26T17:41:44.368 INFO:teuthology.orchestra.run.smithi097.stdout:check_obsoletes = 1 2024-04-26T17:41:44.370 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum clean all 2024-04-26T17:41:44.422 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T17:41:44.485 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-04-26T17:41:44.486 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-04-26T17:41:44.594 INFO:teuthology.orchestra.run.smithi097.stdout:48 files removed 2024-04-26T17:41:44.598 INFO:teuthology.orchestra.run.smithi031.stdout:48 files removed 2024-04-26T17:41:44.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T17:41:44.625 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T17:41:44.713 INFO:teuthology.orchestra.run.smithi186.stdout:48 files removed 2024-04-26T17:41:44.740 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T17:41:45.317 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 200 kB/s | 84 kB 00:00 2024-04-26T17:41:45.350 INFO:teuthology.orchestra.run.smithi097.stdout:ceph packages for x86_64 178 kB/s | 84 kB 00:00 2024-04-26T17:41:45.414 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 204 kB/s | 84 kB 00:00 2024-04-26T17:41:45.734 INFO:teuthology.orchestra.run.smithi097.stdout:ceph noarch packages 34 kB/s | 12 kB 00:00 2024-04-26T17:41:45.751 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-04-26T17:41:45.773 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 37 kB/s | 12 kB 00:00 2024-04-26T17:41:46.082 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 6.9 kB/s | 1.9 kB 00:00 2024-04-26T17:41:46.151 INFO:teuthology.orchestra.run.smithi097.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-04-26T17:41:46.187 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-04-26T17:41:46.665 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-26T17:41:46.710 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-26T17:41:46.920 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-26T17:41:48.666 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-26T17:41:50.796 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 6.8 MB/s | 19 MB 00:02 2024-04-26T17:41:53.027 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 3.6 MB/s | 19 MB 00:05 2024-04-26T17:41:53.309 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 5.6 MB/s | 6.2 MB 00:01 2024-04-26T17:41:55.009 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-04-26T17:41:55.805 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.2 MB 00:01 2024-04-26T17:41:56.236 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-26T17:41:57.489 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-26T17:41:57.703 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 6.2 MB 00:01 2024-04-26T17:41:58.582 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-04-26T17:41:59.095 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 16 kB 00:00 2024-04-26T17:42:01.200 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-26T17:42:01.260 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-26T17:42:03.395 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T17:42:03.396 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T17:42:03.407 INFO:teuthology.orchestra.run.smithi186.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T17:42:03.407 INFO:teuthology.orchestra.run.smithi186.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T17:42:03.408 INFO:teuthology.orchestra.run.smithi186.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T17:42:03.409 INFO:teuthology.orchestra.run.smithi186.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T17:42:03.411 INFO:teuthology.orchestra.run.smithi186.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T17:42:03.413 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T17:42:03.414 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T17:42:03.474 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-26T17:42:03.487 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-04-26T17:42:03.488 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-26T17:42:03.488 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-04-26T17:42:03.488 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-26T17:42:03.488 INFO:teuthology.orchestra.run.smithi186.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T17:42:03.489 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 6.6 k 2024-04-26T17:42:03.489 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 5.4 M 2024-04-26T17:42:03.489 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 890 k 2024-04-26T17:42:03.489 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 145 k 2024-04-26T17:42:03.489 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 1.5 M 2024-04-26T17:42:03.490 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 142 k 2024-04-26T17:42:03.490 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 3.5 M 2024-04-26T17:42:03.490 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 7.4 M 2024-04-26T17:42:03.490 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 49 k 2024-04-26T17:42:03.491 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 11 M 2024-04-26T17:42:03.491 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 49 M 2024-04-26T17:42:03.491 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 273 k 2024-04-26T17:42:03.491 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 762 k 2024-04-26T17:42:03.491 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 224 k 2024-04-26T17:42:03.492 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 27 k 2024-04-26T17:42:03.492 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 18 k 2024-04-26T17:42:03.492 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T17:42:03.492 INFO:teuthology.orchestra.run.smithi186.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T17:42:03.492 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T17:42:03.492 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 32 k 2024-04-26T17:42:03.493 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 746 k 2024-04-26T17:42:03.493 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 127 k 2024-04-26T17:42:03.493 INFO:teuthology.orchestra.run.smithi186.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T17:42:03.493 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T17:42:03.493 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T17:42:03.494 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T17:42:03.494 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T17:42:03.494 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T17:42:03.494 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 162 k 2024-04-26T17:42:03.494 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 320 k 2024-04-26T17:42:03.494 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 298 k 2024-04-26T17:42:03.495 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 99 k 2024-04-26T17:42:03.495 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 85 k 2024-04-26T17:42:03.495 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.1 M 2024-04-26T17:42:03.495 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 170 k 2024-04-26T17:42:03.495 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T17:42:03.495 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-04-26T17:42:03.495 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.4 M 2024-04-26T17:42:03.496 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.1 M 2024-04-26T17:42:03.496 INFO:teuthology.orchestra.run.smithi186.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T17:42:03.496 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T17:42:03.496 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-26T17:42:03.496 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T17:42:03.496 INFO:teuthology.orchestra.run.smithi186.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T17:42:03.497 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T17:42:03.497 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 21 M 2024-04-26T17:42:03.497 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 34 k 2024-04-26T17:42:03.497 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 2.4 M 2024-04-26T17:42:03.497 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 249 k 2024-04-26T17:42:03.497 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 4.7 M 2024-04-26T17:42:03.498 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 17 M 2024-04-26T17:42:03.498 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 16 k 2024-04-26T17:42:03.498 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 25 k 2024-04-26T17:42:03.498 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T17:42:03.498 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T17:42:03.499 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T17:42:03.499 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T17:42:03.499 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T17:42:03.499 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T17:42:03.499 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T17:42:03.499 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T17:42:03.500 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 163 k 2024-04-26T17:42:03.500 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T17:42:03.500 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T17:42:03.500 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T17:42:03.500 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T17:42:03.500 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T17:42:03.501 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 501 k 2024-04-26T17:42:03.501 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T17:42:03.501 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 5.4 M 2024-04-26T17:42:03.501 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T17:42:03.501 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T17:42:03.501 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T17:42:03.501 INFO:teuthology.orchestra.run.smithi186.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T17:42:03.502 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T17:42:03.502 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T17:42:03.502 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T17:42:03.502 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T17:42:03.502 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T17:42:03.502 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T17:42:03.503 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T17:42:03.503 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T17:42:03.503 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T17:42:03.503 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T17:42:03.503 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T17:42:03.503 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T17:42:03.504 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T17:42:03.504 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 45 k 2024-04-26T17:42:03.504 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 135 k 2024-04-26T17:42:03.504 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T17:42:03.504 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T17:42:03.504 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T17:42:03.505 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T17:42:03.505 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T17:42:03.505 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T17:42:03.505 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T17:42:03.505 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T17:42:03.505 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T17:42:03.506 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T17:42:03.506 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T17:42:03.506 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T17:42:03.506 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T17:42:03.506 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T17:42:03.506 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T17:42:03.507 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T17:42:03.507 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T17:42:03.507 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T17:42:03.507 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T17:42:03.507 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T17:42:03.507 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T17:42:03.508 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T17:42:03.508 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T17:42:03.508 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T17:42:03.508 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T17:42:03.508 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T17:42:03.508 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T17:42:03.509 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T17:42:03.509 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T17:42:03.509 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T17:42:03.509 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T17:42:03.509 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T17:42:03.509 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T17:42:03.510 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T17:42:03.510 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T17:42:03.510 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T17:42:03.510 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T17:42:03.510 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T17:42:03.510 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T17:42:03.511 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T17:42:03.511 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T17:42:03.511 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T17:42:03.511 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T17:42:03.511 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T17:42:03.511 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T17:42:03.512 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T17:42:03.512 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T17:42:03.512 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T17:42:03.512 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T17:42:03.512 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T17:42:03.512 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T17:42:03.513 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T17:42:03.513 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T17:42:03.513 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T17:42:03.513 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T17:42:03.513 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T17:42:03.513 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T17:42:03.513 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T17:42:03.514 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T17:42:03.514 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T17:42:03.514 INFO:teuthology.orchestra.run.smithi186.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T17:42:03.514 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T17:42:03.514 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T17:42:03.514 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-04-26T17:42:03.515 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T17:42:03.515 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:42:03.515 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-26T17:42:03.515 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-04-26T17:42:03.515 INFO:teuthology.orchestra.run.smithi186.stdout:Install 148 Packages 2024-04-26T17:42:03.515 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 4 Packages 2024-04-26T17:42:03.515 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:42:03.516 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 222 M 2024-04-26T17:42:03.517 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-26T17:42:03.815 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-04-26T17:42:05.053 INFO:teuthology.orchestra.run.smithi186.stdout:(1/152): ceph-19.0.0-3267.gbc94a2a9.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-04-26T17:42:05.379 INFO:teuthology.orchestra.run.smithi186.stdout:(2/152): ceph-fuse-19.0.0-3267.gbc94a2a9.el9.x8 2.7 MB/s | 890 kB 00:00 2024-04-26T17:42:05.463 INFO:teuthology.orchestra.run.smithi186.stdout:(3/152): ceph-immutable-object-cache-19.0.0-326 1.7 MB/s | 145 kB 00:00 2024-04-26T17:42:05.988 INFO:teuthology.orchestra.run.smithi186.stdout:(4/152): ceph-mds-19.0.0-3267.gbc94a2a9.el9.x86 4.5 MB/s | 2.4 MB 00:00 2024-04-26T17:42:06.008 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T17:42:06.008 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T17:42:06.019 INFO:teuthology.orchestra.run.smithi031.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T17:42:06.020 INFO:teuthology.orchestra.run.smithi031.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T17:42:06.021 INFO:teuthology.orchestra.run.smithi031.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T17:42:06.022 INFO:teuthology.orchestra.run.smithi031.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T17:42:06.023 INFO:teuthology.orchestra.run.smithi031.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T17:42:06.026 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T17:42:06.026 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T17:42:06.054 INFO:teuthology.orchestra.run.smithi186.stdout:(5/152): ceph-base-19.0.0-3267.gbc94a2a9.el9.x8 3.8 MB/s | 5.4 MB 00:01 2024-04-26T17:42:06.092 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-26T17:42:06.102 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-04-26T17:42:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-26T17:42:06.102 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-04-26T17:42:06.102 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-26T17:42:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T17:42:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 6.6 k 2024-04-26T17:42:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 5.4 M 2024-04-26T17:42:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 890 k 2024-04-26T17:42:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 145 k 2024-04-26T17:42:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 1.5 M 2024-04-26T17:42:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 142 k 2024-04-26T17:42:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 3.5 M 2024-04-26T17:42:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 7.4 M 2024-04-26T17:42:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 49 k 2024-04-26T17:42:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 11 M 2024-04-26T17:42:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 49 M 2024-04-26T17:42:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 273 k 2024-04-26T17:42:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 762 k 2024-04-26T17:42:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 224 k 2024-04-26T17:42:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-shell x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 27 k 2024-04-26T17:42:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 18 k 2024-04-26T17:42:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T17:42:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T17:42:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T17:42:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 32 k 2024-04-26T17:42:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 746 k 2024-04-26T17:42:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 127 k 2024-04-26T17:42:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T17:42:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T17:42:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T17:42:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T17:42:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T17:42:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T17:42:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 162 k 2024-04-26T17:42:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 320 k 2024-04-26T17:42:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 298 k 2024-04-26T17:42:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 99 k 2024-04-26T17:42:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 85 k 2024-04-26T17:42:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.1 M 2024-04-26T17:42:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 170 k 2024-04-26T17:42:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T17:42:06.105 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2024-04-26T17:42:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.4 M 2024-04-26T17:42:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.1 M 2024-04-26T17:42:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T17:42:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T17:42:06.105 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-26T17:42:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T17:42:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T17:42:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T17:42:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 21 M 2024-04-26T17:42:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 34 k 2024-04-26T17:42:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 2.4 M 2024-04-26T17:42:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 249 k 2024-04-26T17:42:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 4.7 M 2024-04-26T17:42:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 17 M 2024-04-26T17:42:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 16 k 2024-04-26T17:42:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 25 k 2024-04-26T17:42:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T17:42:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T17:42:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T17:42:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T17:42:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T17:42:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T17:42:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T17:42:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T17:42:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 163 k 2024-04-26T17:42:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T17:42:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T17:42:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T17:42:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T17:42:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T17:42:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 501 k 2024-04-26T17:42:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T17:42:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 5.4 M 2024-04-26T17:42:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T17:42:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T17:42:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T17:42:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T17:42:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T17:42:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T17:42:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T17:42:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T17:42:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T17:42:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T17:42:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T17:42:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T17:42:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T17:42:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T17:42:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T17:42:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T17:42:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T17:42:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 45 k 2024-04-26T17:42:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 135 k 2024-04-26T17:42:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T17:42:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T17:42:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T17:42:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T17:42:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T17:42:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T17:42:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T17:42:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T17:42:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T17:42:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T17:42:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T17:42:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T17:42:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T17:42:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T17:42:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T17:42:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T17:42:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T17:42:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T17:42:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T17:42:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T17:42:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T17:42:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T17:42:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T17:42:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T17:42:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T17:42:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T17:42:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T17:42:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T17:42:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T17:42:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T17:42:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T17:42:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T17:42:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T17:42:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T17:42:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T17:42:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T17:42:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T17:42:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T17:42:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T17:42:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T17:42:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T17:42:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T17:42:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T17:42:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T17:42:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T17:42:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T17:42:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T17:42:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T17:42:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T17:42:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T17:42:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T17:42:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T17:42:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T17:42:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T17:42:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T17:42:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T17:42:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T17:42:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T17:42:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T17:42:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T17:42:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T17:42:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T17:42:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T17:42:06.113 INFO:teuthology.orchestra.run.smithi031.stdout:Installing weak dependencies: 2024-04-26T17:42:06.114 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T17:42:06.114 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:42:06.114 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-26T17:42:06.114 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-04-26T17:42:06.114 INFO:teuthology.orchestra.run.smithi031.stdout:Install 148 Packages 2024-04-26T17:42:06.114 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 4 Packages 2024-04-26T17:42:06.114 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:42:06.114 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 222 M 2024-04-26T17:42:06.114 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-26T17:42:06.197 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-26T17:42:06.330 INFO:teuthology.orchestra.run.smithi186.stdout:(6/152): ceph-mgr-19.0.0-3267.gbc94a2a9.el9.x86 4.4 MB/s | 1.5 MB 00:00 2024-04-26T17:42:06.839 INFO:teuthology.orchestra.run.smithi186.stdout:(7/152): ceph-common-19.0.0-3267.gbc94a2a9.el9. 9.6 MB/s | 21 MB 00:02 2024-04-26T17:42:06.905 INFO:teuthology.orchestra.run.smithi186.stdout:(8/152): ceph-mon-19.0.0-3267.gbc94a2a9.el9.x86 5.5 MB/s | 4.7 MB 00:00 2024-04-26T17:42:07.014 INFO:teuthology.orchestra.run.smithi186.stdout:(9/152): ceph-selinux-19.0.0-3267.gbc94a2a9.el9 232 kB/s | 25 kB 00:00 2024-04-26T17:42:07.091 INFO:teuthology.orchestra.run.smithi031.stdout:(1/152): ceph-19.0.0-3267.gbc94a2a9.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-04-26T17:42:08.181 INFO:teuthology.orchestra.run.smithi186.stdout:(10/152): ceph-radosgw-19.0.0-3267.gbc94a2a9.el 7.9 MB/s | 11 MB 00:01 2024-04-26T17:42:08.183 INFO:teuthology.orchestra.run.smithi031.stdout:(2/152): ceph-fuse-19.0.0-3267.gbc94a2a9.el9.x8 815 kB/s | 890 kB 00:01 2024-04-26T17:42:08.272 INFO:teuthology.orchestra.run.smithi186.stdout:(11/152): cephfs-mirror-19.0.0-3267.gbc94a2a9.e 2.4 MB/s | 224 kB 00:00 2024-04-26T17:42:08.298 INFO:teuthology.orchestra.run.smithi097.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T17:42:08.299 INFO:teuthology.orchestra.run.smithi097.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T17:42:08.309 INFO:teuthology.orchestra.run.smithi097.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T17:42:08.310 INFO:teuthology.orchestra.run.smithi097.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T17:42:08.311 INFO:teuthology.orchestra.run.smithi097.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T17:42:08.312 INFO:teuthology.orchestra.run.smithi097.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T17:42:08.313 INFO:teuthology.orchestra.run.smithi097.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T17:42:08.315 INFO:teuthology.orchestra.run.smithi097.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T17:42:08.316 INFO:teuthology.orchestra.run.smithi097.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T17:42:08.349 INFO:teuthology.orchestra.run.smithi031.stdout:(3/152): ceph-immutable-object-cache-19.0.0-326 869 kB/s | 145 kB 00:00 2024-04-26T17:42:08.356 INFO:teuthology.orchestra.run.smithi186.stdout:(12/152): cephfs-shell-19.0.0-3267.gbc94a2a9.el 324 kB/s | 27 kB 00:00 2024-04-26T17:42:08.376 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-26T17:42:08.395 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================= 2024-04-26T17:42:08.395 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-04-26T17:42:08.395 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================= 2024-04-26T17:42:08.395 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-26T17:42:08.395 INFO:teuthology.orchestra.run.smithi097.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T17:42:08.395 INFO:teuthology.orchestra.run.smithi097.stdout: ceph x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 6.6 k 2024-04-26T17:42:08.396 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 5.4 M 2024-04-26T17:42:08.396 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 890 k 2024-04-26T17:42:08.396 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 145 k 2024-04-26T17:42:08.396 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 1.5 M 2024-04-26T17:42:08.396 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 142 k 2024-04-26T17:42:08.396 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 3.5 M 2024-04-26T17:42:08.396 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 7.4 M 2024-04-26T17:42:08.396 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 49 k 2024-04-26T17:42:08.396 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 11 M 2024-04-26T17:42:08.396 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 49 M 2024-04-26T17:42:08.396 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 273 k 2024-04-26T17:42:08.396 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 762 k 2024-04-26T17:42:08.396 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-mirror x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 224 k 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-shell x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 27 k 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-top noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 18 k 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 32 k 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 746 k 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 127 k 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T17:42:08.397 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 162 k 2024-04-26T17:42:08.398 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 320 k 2024-04-26T17:42:08.398 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 298 k 2024-04-26T17:42:08.398 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 99 k 2024-04-26T17:42:08.398 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 85 k 2024-04-26T17:42:08.398 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.1 M 2024-04-26T17:42:08.398 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 170 k 2024-04-26T17:42:08.398 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T17:42:08.398 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2024-04-26T17:42:08.398 INFO:teuthology.orchestra.run.smithi097.stdout: librados2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.4 M 2024-04-26T17:42:08.398 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 3.1 M 2024-04-26T17:42:08.398 INFO:teuthology.orchestra.run.smithi097.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T17:42:08.398 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T17:42:08.399 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-04-26T17:42:08.399 INFO:teuthology.orchestra.run.smithi097.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T17:42:08.399 INFO:teuthology.orchestra.run.smithi097.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T17:42:08.399 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T17:42:08.399 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 21 M 2024-04-26T17:42:08.399 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 34 k 2024-04-26T17:42:08.399 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 2.4 M 2024-04-26T17:42:08.399 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 249 k 2024-04-26T17:42:08.399 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 4.7 M 2024-04-26T17:42:08.399 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 17 M 2024-04-26T17:42:08.399 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3267.gbc94a2a9.el9 ceph-noarch 16 k 2024-04-26T17:42:08.399 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 25 k 2024-04-26T17:42:08.399 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T17:42:08.399 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T17:42:08.400 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T17:42:08.400 INFO:teuthology.orchestra.run.smithi097.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T17:42:08.400 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T17:42:08.400 INFO:teuthology.orchestra.run.smithi097.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T17:42:08.400 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T17:42:08.400 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T17:42:08.400 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 163 k 2024-04-26T17:42:08.400 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T17:42:08.400 INFO:teuthology.orchestra.run.smithi097.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T17:42:08.400 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T17:42:08.400 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T17:42:08.400 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T17:42:08.400 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 501 k 2024-04-26T17:42:08.400 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T17:42:08.401 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2 x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 5.4 M 2024-04-26T17:42:08.401 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T17:42:08.401 INFO:teuthology.orchestra.run.smithi097.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T17:42:08.401 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T17:42:08.401 INFO:teuthology.orchestra.run.smithi097.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T17:42:08.401 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T17:42:08.401 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T17:42:08.401 INFO:teuthology.orchestra.run.smithi097.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T17:42:08.401 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T17:42:08.401 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T17:42:08.401 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T17:42:08.401 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T17:42:08.401 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T17:42:08.401 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T17:42:08.402 INFO:teuthology.orchestra.run.smithi097.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T17:42:08.402 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T17:42:08.402 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T17:42:08.402 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T17:42:08.402 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 45 k 2024-04-26T17:42:08.402 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common x86_64 2:19.0.0-3267.gbc94a2a9.el9 ceph 135 k 2024-04-26T17:42:08.402 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T17:42:08.402 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T17:42:08.402 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T17:42:08.402 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T17:42:08.402 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T17:42:08.402 INFO:teuthology.orchestra.run.smithi097.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T17:42:08.402 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T17:42:08.402 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T17:42:08.403 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T17:42:08.404 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T17:42:08.405 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T17:42:08.405 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T17:42:08.405 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T17:42:08.405 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T17:42:08.405 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T17:42:08.405 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T17:42:08.405 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T17:42:08.405 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T17:42:08.405 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T17:42:08.405 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T17:42:08.405 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T17:42:08.405 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T17:42:08.405 INFO:teuthology.orchestra.run.smithi097.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T17:42:08.406 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T17:42:08.406 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T17:42:08.406 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T17:42:08.406 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T17:42:08.406 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T17:42:08.406 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T17:42:08.406 INFO:teuthology.orchestra.run.smithi097.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T17:42:08.406 INFO:teuthology.orchestra.run.smithi097.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T17:42:08.406 INFO:teuthology.orchestra.run.smithi097.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T17:42:08.406 INFO:teuthology.orchestra.run.smithi097.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T17:42:08.406 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T17:42:08.406 INFO:teuthology.orchestra.run.smithi097.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T17:42:08.406 INFO:teuthology.orchestra.run.smithi097.stdout:Installing weak dependencies: 2024-04-26T17:42:08.407 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T17:42:08.407 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:42:08.407 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-26T17:42:08.407 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================= 2024-04-26T17:42:08.407 INFO:teuthology.orchestra.run.smithi097.stdout:Install 148 Packages 2024-04-26T17:42:08.407 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 4 Packages 2024-04-26T17:42:08.407 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:42:08.407 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 222 M 2024-04-26T17:42:08.408 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-26T17:42:08.423 INFO:teuthology.orchestra.run.smithi186.stdout:(13/152): libcephfs-devel-19.0.0-3267.gbc94a2a9 475 kB/s | 32 kB 00:00 2024-04-26T17:42:08.682 INFO:teuthology.orchestra.run.smithi186.stdout:(14/152): libcephfs2-19.0.0-3267.gbc94a2a9.el9. 2.8 MB/s | 746 kB 00:00 2024-04-26T17:42:08.774 INFO:teuthology.orchestra.run.smithi186.stdout:(15/152): libcephsqlite-19.0.0-3267.gbc94a2a9.e 1.7 MB/s | 163 kB 00:00 2024-04-26T17:42:08.865 INFO:teuthology.orchestra.run.smithi186.stdout:(16/152): librados-devel-19.0.0-3267.gbc94a2a9. 1.3 MB/s | 127 kB 00:00 2024-04-26T17:42:09.116 INFO:teuthology.orchestra.run.smithi186.stdout:(17/152): libradosstriper1-19.0.0-3267.gbc94a2a 2.0 MB/s | 501 kB 00:00 2024-04-26T17:42:09.756 INFO:teuthology.orchestra.run.smithi097.stdout:(1/152): ceph-19.0.0-3267.gbc94a2a9.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-04-26T17:42:09.842 INFO:teuthology.orchestra.run.smithi031.stdout:(4/152): ceph-mds-19.0.0-3267.gbc94a2a9.el9.x86 1.6 MB/s | 2.4 MB 00:01 2024-04-26T17:42:10.340 INFO:teuthology.orchestra.run.smithi097.stdout:(2/152): ceph-fuse-19.0.0-3267.gbc94a2a9.el9.x8 1.5 MB/s | 890 kB 00:00 2024-04-26T17:42:10.440 INFO:teuthology.orchestra.run.smithi097.stdout:(3/152): ceph-immutable-object-cache-19.0.0-326 1.4 MB/s | 145 kB 00:00 2024-04-26T17:42:10.674 INFO:teuthology.orchestra.run.smithi186.stdout:(18/152): ceph-osd-19.0.0-3267.gbc94a2a9.el9.x8 3.9 MB/s | 17 MB 00:04 2024-04-26T17:42:10.775 INFO:teuthology.orchestra.run.smithi031.stdout:(5/152): ceph-mgr-19.0.0-3267.gbc94a2a9.el9.x86 1.6 MB/s | 1.5 MB 00:00 2024-04-26T17:42:10.799 INFO:teuthology.orchestra.run.smithi186.stdout:(19/152): python3-ceph-argparse-19.0.0-3267.gbc 362 kB/s | 45 kB 00:00 2024-04-26T17:42:10.923 INFO:teuthology.orchestra.run.smithi097.stdout:(4/152): ceph-mds-19.0.0-3267.gbc94a2a9.el9.x86 4.9 MB/s | 2.4 MB 00:00 2024-04-26T17:42:10.983 INFO:teuthology.orchestra.run.smithi186.stdout:(20/152): python3-ceph-common-19.0.0-3267.gbc94 736 kB/s | 135 kB 00:00 2024-04-26T17:42:11.067 INFO:teuthology.orchestra.run.smithi031.stdout:(6/152): ceph-base-19.0.0-3267.gbc94a2a9.el9.x8 1.2 MB/s | 5.4 MB 00:04 2024-04-26T17:42:11.175 INFO:teuthology.orchestra.run.smithi186.stdout:(21/152): python3-cephfs-19.0.0-3267.gbc94a2a9. 844 kB/s | 162 kB 00:00 2024-04-26T17:42:11.282 INFO:teuthology.orchestra.run.smithi097.stdout:(5/152): ceph-mgr-19.0.0-3267.gbc94a2a9.el9.x86 4.2 MB/s | 1.5 MB 00:00 2024-04-26T17:42:11.492 INFO:teuthology.orchestra.run.smithi186.stdout:(22/152): python3-rados-19.0.0-3267.gbc94a2a9.e 1.0 MB/s | 320 kB 00:00 2024-04-26T17:42:11.684 INFO:teuthology.orchestra.run.smithi186.stdout:(23/152): librgw2-19.0.0-3267.gbc94a2a9.el9.x86 2.1 MB/s | 5.4 MB 00:02 2024-04-26T17:42:11.710 INFO:teuthology.orchestra.run.smithi186.stdout:(24/152): python3-rbd-19.0.0-3267.gbc94a2a9.el9 1.3 MB/s | 298 kB 00:00 2024-04-26T17:42:11.752 INFO:teuthology.orchestra.run.smithi186.stdout:(25/152): python3-rgw-19.0.0-3267.gbc94a2a9.el9 1.5 MB/s | 99 kB 00:00 2024-04-26T17:42:11.793 INFO:teuthology.orchestra.run.smithi186.stdout:(26/152): rbd-fuse-19.0.0-3267.gbc94a2a9.el9.x8 1.0 MB/s | 85 kB 00:00 2024-04-26T17:42:11.952 INFO:teuthology.orchestra.run.smithi186.stdout:(27/152): rbd-nbd-19.0.0-3267.gbc94a2a9.el9.x86 1.1 MB/s | 170 kB 00:00 2024-04-26T17:42:12.085 INFO:teuthology.orchestra.run.smithi186.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3267.g 256 kB/s | 34 kB 00:00 2024-04-26T17:42:12.116 INFO:teuthology.orchestra.run.smithi097.stdout:(6/152): ceph-mon-19.0.0-3267.gbc94a2a9.el9.x86 5.6 MB/s | 4.7 MB 00:00 2024-04-26T17:42:12.183 INFO:teuthology.orchestra.run.smithi097.stdout:(7/152): ceph-base-19.0.0-3267.gbc94a2a9.el9.x8 1.9 MB/s | 5.4 MB 00:02 2024-04-26T17:42:12.186 INFO:teuthology.orchestra.run.smithi186.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3267.gbc94a2a 1.4 MB/s | 142 kB 00:00 2024-04-26T17:42:12.809 INFO:teuthology.orchestra.run.smithi031.stdout:(7/152): ceph-mon-19.0.0-3267.gbc94a2a9.el9.x86 2.3 MB/s | 4.7 MB 00:02 2024-04-26T17:42:13.061 INFO:teuthology.orchestra.run.smithi186.stdout:(30/152): rbd-mirror-19.0.0-3267.gbc94a2a9.el9. 2.4 MB/s | 3.1 MB 00:01 2024-04-26T17:42:13.650 INFO:teuthology.orchestra.run.smithi031.stdout:(8/152): ceph-common-19.0.0-3267.gbc94a2a9.el9. 3.1 MB/s | 21 MB 00:06 2024-04-26T17:42:13.734 INFO:teuthology.orchestra.run.smithi031.stdout:(9/152): ceph-selinux-19.0.0-3267.gbc94a2a9.el9 304 kB/s | 25 kB 00:00 2024-04-26T17:42:14.148 INFO:teuthology.orchestra.run.smithi186.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3267.gbc94a 1.8 MB/s | 3.5 MB 00:01 2024-04-26T17:42:14.340 INFO:teuthology.orchestra.run.smithi186.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3267.gbc 1.3 MB/s | 249 kB 00:00 2024-04-26T17:42:14.456 INFO:teuthology.orchestra.run.smithi186.stdout:(33/152): ceph-mgr-rook-19.0.0-3267.gbc94a2a9.e 423 kB/s | 49 kB 00:00 2024-04-26T17:42:14.564 INFO:teuthology.orchestra.run.smithi186.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3267.gb 152 kB/s | 16 kB 00:00 2024-04-26T17:42:14.823 INFO:teuthology.orchestra.run.smithi186.stdout:(35/152): ceph-volume-19.0.0-3267.gbc94a2a9.el9 1.0 MB/s | 273 kB 00:00 2024-04-26T17:42:15.232 INFO:teuthology.orchestra.run.smithi097.stdout:(8/152): ceph-osd-19.0.0-3267.gbc94a2a9.el9.x86 5.4 MB/s | 17 MB 00:03 2024-04-26T17:42:15.290 INFO:teuthology.orchestra.run.smithi186.stdout:(36/152): cephadm-19.0.0-3267.gbc94a2a9.el9.noa 1.6 MB/s | 762 kB 00:00 2024-04-26T17:42:15.335 INFO:teuthology.orchestra.run.smithi097.stdout:(9/152): ceph-selinux-19.0.0-3267.gbc94a2a9.el9 247 kB/s | 25 kB 00:00 2024-04-26T17:42:15.373 INFO:teuthology.orchestra.run.smithi186.stdout:(37/152): cephfs-top-19.0.0-3267.gbc94a2a9.el9. 219 kB/s | 18 kB 00:00 2024-04-26T17:42:15.431 INFO:teuthology.orchestra.run.smithi186.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 80 kB 00:00 2024-04-26T17:42:15.557 INFO:teuthology.orchestra.run.smithi186.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 6.3 MB/s | 803 kB 00:00 2024-04-26T17:42:15.624 INFO:teuthology.orchestra.run.smithi186.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-26T17:42:15.652 INFO:teuthology.orchestra.run.smithi097.stdout:(10/152): ceph-radosgw-19.0.0-3267.gbc94a2a9.el 3.1 MB/s | 11 MB 00:03 2024-04-26T17:42:15.699 INFO:teuthology.orchestra.run.smithi186.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-04-26T17:42:15.785 INFO:teuthology.orchestra.run.smithi097.stdout:(11/152): cephfs-mirror-19.0.0-3267.gbc94a2a9.e 1.6 MB/s | 224 kB 00:00 2024-04-26T17:42:15.808 INFO:teuthology.orchestra.run.smithi186.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-04-26T17:42:15.875 INFO:teuthology.orchestra.run.smithi186.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-26T17:42:15.894 INFO:teuthology.orchestra.run.smithi097.stdout:(12/152): cephfs-shell-19.0.0-3267.gbc94a2a9.el 250 kB/s | 27 kB 00:00 2024-04-26T17:42:15.958 INFO:teuthology.orchestra.run.smithi186.stdout:(44/152): ceph-mgr-diskprediction-local-19.0.0- 2.6 MB/s | 7.4 MB 00:02 2024-04-26T17:42:15.986 INFO:teuthology.orchestra.run.smithi097.stdout:(13/152): libcephfs-devel-19.0.0-3267.gbc94a2a9 346 kB/s | 32 kB 00:00 2024-04-26T17:42:16.017 INFO:teuthology.orchestra.run.smithi186.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 952 kB/s | 135 kB 00:00 2024-04-26T17:42:16.042 INFO:teuthology.orchestra.run.smithi186.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-04-26T17:42:16.109 INFO:teuthology.orchestra.run.smithi186.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-04-26T17:42:16.159 INFO:teuthology.orchestra.run.smithi186.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 5.8 MB/s | 689 kB 00:00 2024-04-26T17:42:16.220 INFO:teuthology.orchestra.run.smithi097.stdout:(14/152): libcephfs2-19.0.0-3267.gbc94a2a9.el9. 3.1 MB/s | 746 kB 00:00 2024-04-26T17:42:16.226 INFO:teuthology.orchestra.run.smithi186.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-26T17:42:16.260 INFO:teuthology.orchestra.run.smithi186.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 6.1 MB/s | 943 kB 00:00 2024-04-26T17:42:16.302 INFO:teuthology.orchestra.run.smithi186.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 286 kB/s | 21 kB 00:00 2024-04-26T17:42:16.303 INFO:teuthology.orchestra.run.smithi097.stdout:(15/152): libcephsqlite-19.0.0-3267.gbc94a2a9.e 1.9 MB/s | 163 kB 00:00 2024-04-26T17:42:16.352 INFO:teuthology.orchestra.run.smithi186.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 654 kB/s | 33 kB 00:00 2024-04-26T17:42:16.377 INFO:teuthology.orchestra.run.smithi186.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-04-26T17:42:16.403 INFO:teuthology.orchestra.run.smithi097.stdout:(16/152): librados-devel-19.0.0-3267.gbc94a2a9. 1.2 MB/s | 127 kB 00:00 2024-04-26T17:42:16.428 INFO:teuthology.orchestra.run.smithi186.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-04-26T17:42:16.469 INFO:teuthology.orchestra.run.smithi186.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 268 kB/s | 11 kB 00:00 2024-04-26T17:42:16.545 INFO:teuthology.orchestra.run.smithi186.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-26T17:42:16.578 INFO:teuthology.orchestra.run.smithi097.stdout:(17/152): libradosstriper1-19.0.0-3267.gbc94a2a 2.8 MB/s | 501 kB 00:00 2024-04-26T17:42:16.620 INFO:teuthology.orchestra.run.smithi186.stdout:(57/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-04-26T17:42:16.645 INFO:teuthology.orchestra.run.smithi186.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 451 kB/s | 45 kB 00:00 2024-04-26T17:42:16.729 INFO:teuthology.orchestra.run.smithi186.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-04-26T17:42:16.754 INFO:teuthology.orchestra.run.smithi186.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 5.3 MB/s | 582 kB 00:00 2024-04-26T17:42:16.796 INFO:teuthology.orchestra.run.smithi186.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 380 kB/s | 25 kB 00:00 2024-04-26T17:42:16.822 INFO:teuthology.orchestra.run.smithi186.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 706 kB/s | 47 kB 00:00 2024-04-26T17:42:16.897 INFO:teuthology.orchestra.run.smithi186.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 188 kB 00:00 2024-04-26T17:42:16.930 INFO:teuthology.orchestra.run.smithi186.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-04-26T17:42:16.956 INFO:teuthology.orchestra.run.smithi186.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 602 kB/s | 35 kB 00:00 2024-04-26T17:42:17.223 INFO:teuthology.orchestra.run.smithi186.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 16 MB/s | 4.7 MB 00:00 2024-04-26T17:42:17.275 INFO:teuthology.orchestra.run.smithi031.stdout:(10/152): ceph-radosgw-19.0.0-3267.gbc94a2a9.el 2.4 MB/s | 11 MB 00:04 2024-04-26T17:42:17.289 INFO:teuthology.orchestra.run.smithi186.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-26T17:42:17.398 INFO:teuthology.orchestra.run.smithi186.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 8.2 MB/s | 911 kB 00:00 2024-04-26T17:42:17.417 INFO:teuthology.orchestra.run.smithi031.stdout:(11/152): cephfs-mirror-19.0.0-3267.gbc94a2a9.e 1.6 MB/s | 224 kB 00:00 2024-04-26T17:42:17.448 INFO:teuthology.orchestra.run.smithi186.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-04-26T17:42:17.490 INFO:teuthology.orchestra.run.smithi186.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-04-26T17:42:17.500 INFO:teuthology.orchestra.run.smithi031.stdout:(12/152): cephfs-shell-19.0.0-3267.gbc94a2a9.el 325 kB/s | 27 kB 00:00 2024-04-26T17:42:17.599 INFO:teuthology.orchestra.run.smithi186.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-04-26T17:42:17.609 INFO:teuthology.orchestra.run.smithi031.stdout:(13/152): libcephfs-devel-19.0.0-3267.gbc94a2a9 293 kB/s | 32 kB 00:00 2024-04-26T17:42:17.658 INFO:teuthology.orchestra.run.smithi186.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-26T17:42:17.791 INFO:teuthology.orchestra.run.smithi186.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 17 MB/s | 5.8 MB 00:00 2024-04-26T17:42:17.875 INFO:teuthology.orchestra.run.smithi186.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-04-26T17:42:17.893 INFO:teuthology.orchestra.run.smithi031.stdout:(14/152): libcephfs2-19.0.0-3267.gbc94a2a9.el9. 2.6 MB/s | 746 kB 00:00 2024-04-26T17:42:17.958 INFO:teuthology.orchestra.run.smithi186.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-04-26T17:42:17.976 INFO:teuthology.orchestra.run.smithi031.stdout:(15/152): libcephsqlite-19.0.0-3267.gbc94a2a9.e 1.9 MB/s | 163 kB 00:00 2024-04-26T17:42:17.987 INFO:teuthology.orchestra.run.smithi097.stdout:(18/152): librgw2-19.0.0-3267.gbc94a2a9.el9.x86 3.8 MB/s | 5.4 MB 00:01 2024-04-26T17:42:18.050 INFO:teuthology.orchestra.run.smithi186.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-04-26T17:42:18.051 INFO:teuthology.orchestra.run.smithi031.stdout:(16/152): librados-devel-19.0.0-3267.gbc94a2a9. 1.7 MB/s | 127 kB 00:00 2024-04-26T17:42:18.070 INFO:teuthology.orchestra.run.smithi097.stdout:(19/152): python3-ceph-argparse-19.0.0-3267.gbc 543 kB/s | 45 kB 00:00 2024-04-26T17:42:18.075 INFO:teuthology.orchestra.run.smithi186.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 219 kB/s | 26 kB 00:00 2024-04-26T17:42:18.117 INFO:teuthology.orchestra.run.smithi186.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 287 kB/s | 19 kB 00:00 2024-04-26T17:42:18.154 INFO:teuthology.orchestra.run.smithi097.stdout:(20/152): python3-ceph-common-19.0.0-3267.gbc94 1.6 MB/s | 135 kB 00:00 2024-04-26T17:42:18.168 INFO:teuthology.orchestra.run.smithi186.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-04-26T17:42:18.193 INFO:teuthology.orchestra.run.smithi186.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 462 kB/s | 35 kB 00:00 2024-04-26T17:42:18.237 INFO:teuthology.orchestra.run.smithi097.stdout:(21/152): python3-cephfs-19.0.0-3267.gbc94a2a9. 1.9 MB/s | 162 kB 00:00 2024-04-26T17:42:18.252 INFO:teuthology.orchestra.run.smithi031.stdout:(17/152): libradosstriper1-19.0.0-3267.gbc94a2a 2.4 MB/s | 501 kB 00:00 2024-04-26T17:42:18.268 INFO:teuthology.orchestra.run.smithi186.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.3 MB/s | 250 kB 00:00 2024-04-26T17:42:18.343 INFO:teuthology.orchestra.run.smithi186.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-04-26T17:42:18.371 INFO:teuthology.orchestra.run.smithi097.stdout:(22/152): python3-rados-19.0.0-3267.gbc94a2a9.e 2.3 MB/s | 320 kB 00:00 2024-04-26T17:42:18.410 INFO:teuthology.orchestra.run.smithi186.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-04-26T17:42:18.513 INFO:teuthology.orchestra.run.smithi097.stdout:(23/152): python3-rbd-19.0.0-3267.gbc94a2a9.el9 2.1 MB/s | 298 kB 00:00 2024-04-26T17:42:18.519 INFO:teuthology.orchestra.run.smithi186.stdout:(84/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 5.1 MB 00:00 2024-04-26T17:42:18.544 INFO:teuthology.orchestra.run.smithi186.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 316 kB/s | 42 kB 00:00 2024-04-26T17:42:18.597 INFO:teuthology.orchestra.run.smithi097.stdout:(24/152): python3-rgw-19.0.0-3267.gbc94a2a9.el9 1.2 MB/s | 99 kB 00:00 2024-04-26T17:42:18.611 INFO:teuthology.orchestra.run.smithi186.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 2.9 MB/s | 269 kB 00:00 2024-04-26T17:42:18.637 INFO:teuthology.orchestra.run.smithi186.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-04-26T17:42:18.680 INFO:teuthology.orchestra.run.smithi097.stdout:(25/152): rbd-fuse-19.0.0-3267.gbc94a2a9.el9.x8 1.0 MB/s | 85 kB 00:00 2024-04-26T17:42:18.695 INFO:teuthology.orchestra.run.smithi186.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 878 kB/s | 51 kB 00:00 2024-04-26T17:42:18.720 INFO:teuthology.orchestra.run.smithi186.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-04-26T17:42:18.771 INFO:teuthology.orchestra.run.smithi186.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 712 kB/s | 54 kB 00:00 2024-04-26T17:42:18.829 INFO:teuthology.orchestra.run.smithi186.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 714 kB/s | 42 kB 00:00 2024-04-26T17:42:18.896 INFO:teuthology.orchestra.run.smithi186.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 662 kB/s | 44 kB 00:00 2024-04-26T17:42:19.005 INFO:teuthology.orchestra.run.smithi186.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-04-26T17:42:19.056 INFO:teuthology.orchestra.run.smithi186.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 719 kB/s | 36 kB 00:00 2024-04-26T17:42:19.061 INFO:teuthology.orchestra.run.smithi031.stdout:(18/152): ceph-osd-19.0.0-3267.gbc94a2a9.el9.x8 2.1 MB/s | 17 MB 00:07 2024-04-26T17:42:19.114 INFO:teuthology.orchestra.run.smithi186.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 811 kB/s | 47 kB 00:00 2024-04-26T17:42:19.145 INFO:teuthology.orchestra.run.smithi031.stdout:(19/152): python3-ceph-argparse-19.0.0-3267.gbc 544 kB/s | 45 kB 00:00 2024-04-26T17:42:19.222 INFO:teuthology.orchestra.run.smithi186.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.8 MB/s | 305 kB 00:00 2024-04-26T17:42:19.228 INFO:teuthology.orchestra.run.smithi031.stdout:(20/152): python3-ceph-common-19.0.0-3267.gbc94 1.6 MB/s | 135 kB 00:00 2024-04-26T17:42:19.303 INFO:teuthology.orchestra.run.smithi031.stdout:(21/152): python3-cephfs-19.0.0-3267.gbc94a2a9. 2.1 MB/s | 162 kB 00:00 2024-04-26T17:42:19.382 INFO:teuthology.orchestra.run.smithi186.stdout:(97/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-04-26T17:42:19.449 INFO:teuthology.orchestra.run.smithi186.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 256 kB/s | 58 kB 00:00 2024-04-26T17:42:19.462 INFO:teuthology.orchestra.run.smithi031.stdout:(22/152): python3-rados-19.0.0-3267.gbc94a2a9.e 2.0 MB/s | 320 kB 00:00 2024-04-26T17:42:19.472 INFO:teuthology.orchestra.run.smithi097.stdout:(26/152): rbd-mirror-19.0.0-3267.gbc94a2a9.el9. 3.9 MB/s | 3.1 MB 00:00 2024-04-26T17:42:19.482 INFO:teuthology.orchestra.run.smithi186.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 938 kB/s | 94 kB 00:00 2024-04-26T17:42:19.541 INFO:teuthology.orchestra.run.smithi186.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 694 kB/s | 64 kB 00:00 2024-04-26T17:42:19.556 INFO:teuthology.orchestra.run.smithi097.stdout:(27/152): rbd-nbd-19.0.0-3267.gbc94a2a9.el9.x86 2.0 MB/s | 170 kB 00:00 2024-04-26T17:42:19.565 INFO:teuthology.orchestra.run.smithi186.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 536 kB/s | 45 kB 00:00 2024-04-26T17:42:19.604 INFO:teuthology.orchestra.run.smithi031.stdout:(23/152): python3-rbd-19.0.0-3267.gbc94a2a9.el9 2.1 MB/s | 298 kB 00:00 2024-04-26T17:42:19.607 INFO:teuthology.orchestra.run.smithi186.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 334 kB/s | 22 kB 00:00 2024-04-26T17:42:19.640 INFO:teuthology.orchestra.run.smithi097.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3267.g 410 kB/s | 34 kB 00:00 2024-04-26T17:42:19.686 INFO:teuthology.orchestra.run.smithi186.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 7.2 MB/s | 863 kB 00:00 2024-04-26T17:42:19.687 INFO:teuthology.orchestra.run.smithi031.stdout:(24/152): python3-rgw-19.0.0-3267.gbc94a2a9.el9 1.2 MB/s | 99 kB 00:00 2024-04-26T17:42:19.740 INFO:teuthology.orchestra.run.smithi097.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3267.gbc94a2a 1.4 MB/s | 142 kB 00:00 2024-04-26T17:42:19.771 INFO:teuthology.orchestra.run.smithi031.stdout:(25/152): rbd-fuse-19.0.0-3267.gbc94a2a9.el9.x8 1.0 MB/s | 85 kB 00:00 2024-04-26T17:42:19.946 INFO:teuthology.orchestra.run.smithi031.stdout:(26/152): librgw2-19.0.0-3267.gbc94a2a9.el9.x86 3.2 MB/s | 5.4 MB 00:01 2024-04-26T17:42:20.121 INFO:teuthology.orchestra.run.smithi031.stdout:(27/152): rbd-nbd-19.0.0-3267.gbc94a2a9.el9.x86 973 kB/s | 170 kB 00:00 2024-04-26T17:42:20.213 INFO:teuthology.orchestra.run.smithi031.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3267.g 373 kB/s | 34 kB 00:00 2024-04-26T17:42:20.306 INFO:teuthology.orchestra.run.smithi031.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3267.gbc94a2a 1.5 MB/s | 142 kB 00:00 2024-04-26T17:42:20.583 INFO:teuthology.orchestra.run.smithi186.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 343 kB/s | 309 kB 00:00 2024-04-26T17:42:20.692 INFO:teuthology.orchestra.run.smithi186.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 506 kB/s | 548 kB 00:01 2024-04-26T17:42:20.699 INFO:teuthology.orchestra.run.smithi097.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3267.gbc94a 3.7 MB/s | 3.5 MB 00:00 2024-04-26T17:42:20.717 INFO:teuthology.orchestra.run.smithi186.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 146 kB/s | 19 kB 00:00 2024-04-26T17:42:20.825 INFO:teuthology.orchestra.run.smithi186.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-26T17:42:20.897 INFO:teuthology.orchestra.run.smithi031.stdout:(30/152): rbd-mirror-19.0.0-3267.gbc94a2a9.el9. 2.7 MB/s | 3.1 MB 00:01 2024-04-26T17:42:20.934 INFO:teuthology.orchestra.run.smithi186.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-04-26T17:42:21.126 INFO:teuthology.orchestra.run.smithi186.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 352 kB/s | 67 kB 00:00 2024-04-26T17:42:21.393 INFO:teuthology.orchestra.run.smithi186.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 567 kB/s | 151 kB 00:00 2024-04-26T17:42:21.424 INFO:teuthology.orchestra.run.smithi097.stdout:(31/152): ceph-common-19.0.0-3267.gbc94a2a9.el9 1.7 MB/s | 21 MB 00:12 2024-04-26T17:42:21.431 INFO:teuthology.orchestra.run.smithi031.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3267.gbc94a 3.1 MB/s | 3.5 MB 00:01 2024-04-26T17:42:21.524 INFO:teuthology.orchestra.run.smithi097.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3267.gbc 2.4 MB/s | 249 kB 00:00 2024-04-26T17:42:21.539 INFO:teuthology.orchestra.run.smithi031.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3267.gbc 2.2 MB/s | 249 kB 00:00 2024-04-26T17:42:21.623 INFO:teuthology.orchestra.run.smithi031.stdout:(33/152): ceph-mgr-rook-19.0.0-3267.gbc94a2a9.e 592 kB/s | 49 kB 00:00 2024-04-26T17:42:21.641 INFO:teuthology.orchestra.run.smithi097.stdout:(33/152): ceph-mgr-rook-19.0.0-3267.gbc94a2a9.e 422 kB/s | 49 kB 00:00 2024-04-26T17:42:21.698 INFO:teuthology.orchestra.run.smithi031.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3267.gb 220 kB/s | 16 kB 00:00 2024-04-26T17:42:21.725 INFO:teuthology.orchestra.run.smithi097.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3267.gb 198 kB/s | 16 kB 00:00 2024-04-26T17:42:21.882 INFO:teuthology.orchestra.run.smithi031.stdout:(35/152): ceph-volume-19.0.0-3267.gbc94a2a9.el9 1.5 MB/s | 273 kB 00:00 2024-04-26T17:42:21.900 INFO:teuthology.orchestra.run.smithi097.stdout:(35/152): ceph-volume-19.0.0-3267.gbc94a2a9.el9 1.5 MB/s | 273 kB 00:00 2024-04-26T17:42:22.157 INFO:teuthology.orchestra.run.smithi031.stdout:(36/152): cephadm-19.0.0-3267.gbc94a2a9.el9.noa 2.7 MB/s | 762 kB 00:00 2024-04-26T17:42:22.175 INFO:teuthology.orchestra.run.smithi097.stdout:(36/152): cephadm-19.0.0-3267.gbc94a2a9.el9.noa 2.7 MB/s | 762 kB 00:00 2024-04-26T17:42:22.241 INFO:teuthology.orchestra.run.smithi031.stdout:(37/152): cephfs-top-19.0.0-3267.gbc94a2a9.el9. 219 kB/s | 18 kB 00:00 2024-04-26T17:42:22.276 INFO:teuthology.orchestra.run.smithi097.stdout:(37/152): cephfs-top-19.0.0-3267.gbc94a2a9.el9. 182 kB/s | 18 kB 00:00 2024-04-26T17:42:22.383 INFO:teuthology.orchestra.run.smithi031.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 568 kB/s | 80 kB 00:00 2024-04-26T17:42:22.418 INFO:teuthology.orchestra.run.smithi097.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 567 kB/s | 80 kB 00:00 2024-04-26T17:42:22.485 INFO:teuthology.orchestra.run.smithi186.stdout:(111/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 768 kB/s | 839 kB 00:01 2024-04-26T17:42:22.618 INFO:teuthology.orchestra.run.smithi097.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-04-26T17:42:22.641 INFO:teuthology.orchestra.run.smithi031.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-04-26T17:42:22.750 INFO:teuthology.orchestra.run.smithi031.stdout:(40/152): ceph-mgr-diskprediction-local-19.0.0- 4.0 MB/s | 7.4 MB 00:01 2024-04-26T17:42:22.752 INFO:teuthology.orchestra.run.smithi097.stdout:(40/152): ceph-mgr-diskprediction-local-19.0.0- 3.6 MB/s | 7.4 MB 00:02 2024-04-26T17:42:22.775 INFO:teuthology.orchestra.run.smithi031.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-04-26T17:42:22.777 INFO:teuthology.orchestra.run.smithi097.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-26T17:42:22.950 INFO:teuthology.orchestra.run.smithi031.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-04-26T17:42:23.001 INFO:teuthology.orchestra.run.smithi031.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-26T17:42:23.051 INFO:teuthology.orchestra.run.smithi031.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-26T17:42:23.110 INFO:teuthology.orchestra.run.smithi031.stdout:(45/152): python3-cryptography-36.0.1-4.el9.x86 3.7 MB/s | 1.2 MB 00:00 2024-04-26T17:42:23.134 INFO:teuthology.orchestra.run.smithi031.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-04-26T17:42:23.193 INFO:teuthology.orchestra.run.smithi031.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-04-26T17:42:23.194 INFO:teuthology.orchestra.run.smithi097.stdout:(42/152): ceph-test-19.0.0-3267.gbc94a2a9.el9.x 6.3 MB/s | 49 MB 00:07 2024-04-26T17:42:23.244 INFO:teuthology.orchestra.run.smithi186.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 723 kB/s | 548 kB 00:00 2024-04-26T17:42:23.277 INFO:teuthology.orchestra.run.smithi031.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 4.7 MB/s | 689 kB 00:00 2024-04-26T17:42:23.335 INFO:teuthology.orchestra.run.smithi031.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 6.5 MB/s | 943 kB 00:00 2024-04-26T17:42:23.344 INFO:teuthology.orchestra.run.smithi097.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 2.2 MB/s | 1.2 MB 00:00 2024-04-26T17:42:23.352 INFO:teuthology.orchestra.run.smithi186.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 402 kB/s | 43 kB 00:00 2024-04-26T17:42:23.369 INFO:teuthology.orchestra.run.smithi031.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 107 kB 00:00 2024-04-26T17:42:23.370 INFO:teuthology.orchestra.run.smithi097.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 609 kB/s | 106 kB 00:00 2024-04-26T17:42:23.395 INFO:teuthology.orchestra.run.smithi097.stdout:(45/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 394 kB/s | 253 kB 00:00 2024-04-26T17:42:23.402 INFO:teuthology.orchestra.run.smithi031.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 322 kB/s | 21 kB 00:00 2024-04-26T17:42:23.444 INFO:teuthology.orchestra.run.smithi031.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 786 kB/s | 33 kB 00:00 2024-04-26T17:42:23.453 INFO:teuthology.orchestra.run.smithi097.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-04-26T17:42:23.460 INFO:teuthology.orchestra.run.smithi186.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-04-26T17:42:23.478 INFO:teuthology.orchestra.run.smithi031.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-04-26T17:42:23.479 INFO:teuthology.orchestra.run.smithi097.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-04-26T17:42:23.520 INFO:teuthology.orchestra.run.smithi031.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 441 kB/s | 18 kB 00:00 2024-04-26T17:42:23.537 INFO:teuthology.orchestra.run.smithi097.stdout:(48/152): python3-pycparser-2.20-6.el9.noarch.r 702 kB/s | 135 kB 00:00 2024-04-26T17:42:23.554 INFO:teuthology.orchestra.run.smithi031.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 334 kB/s | 11 kB 00:00 2024-04-26T17:42:23.594 INFO:teuthology.orchestra.run.smithi186.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 112 kB/s | 15 kB 00:00 2024-04-26T17:42:23.596 INFO:teuthology.orchestra.run.smithi097.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-26T17:42:23.637 INFO:teuthology.orchestra.run.smithi031.stdout:(56/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-04-26T17:42:23.663 INFO:teuthology.orchestra.run.smithi097.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 689 kB 00:00 2024-04-26T17:42:23.679 INFO:teuthology.orchestra.run.smithi031.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-26T17:42:23.688 INFO:teuthology.orchestra.run.smithi097.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 234 kB/s | 21 kB 00:00 2024-04-26T17:42:23.722 INFO:teuthology.orchestra.run.smithi097.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 3.8 MB/s | 943 kB 00:00 2024-04-26T17:42:23.729 INFO:teuthology.orchestra.run.smithi031.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 493 kB/s | 45 kB 00:00 2024-04-26T17:42:23.747 INFO:teuthology.orchestra.run.smithi097.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 560 kB/s | 33 kB 00:00 2024-04-26T17:42:23.780 INFO:teuthology.orchestra.run.smithi097.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-04-26T17:42:23.796 INFO:teuthology.orchestra.run.smithi031.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-04-26T17:42:23.805 INFO:teuthology.orchestra.run.smithi097.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 311 kB/s | 18 kB 00:00 2024-04-26T17:42:23.831 INFO:teuthology.orchestra.run.smithi097.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-04-26T17:42:23.838 INFO:teuthology.orchestra.run.smithi031.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 5.2 MB/s | 582 kB 00:00 2024-04-26T17:42:23.861 INFO:teuthology.orchestra.run.smithi186.stdout:(116/152): python3-cheroot-8.6.0-4.el9.noarch.r 645 kB/s | 172 kB 00:00 2024-04-26T17:42:23.880 INFO:teuthology.orchestra.run.smithi031.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 304 kB/s | 25 kB 00:00 2024-04-26T17:42:23.881 INFO:teuthology.orchestra.run.smithi097.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 905 kB/s | 45 kB 00:00 2024-04-26T17:42:23.906 INFO:teuthology.orchestra.run.smithi031.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 710 kB/s | 47 kB 00:00 2024-04-26T17:42:23.906 INFO:teuthology.orchestra.run.smithi097.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-04-26T17:42:23.973 INFO:teuthology.orchestra.run.smithi031.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-04-26T17:42:23.998 INFO:teuthology.orchestra.run.smithi031.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-04-26T17:42:24.023 INFO:teuthology.orchestra.run.smithi097.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 662 kB 00:00 2024-04-26T17:42:24.040 INFO:teuthology.orchestra.run.smithi031.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 528 kB/s | 35 kB 00:00 2024-04-26T17:42:24.074 INFO:teuthology.orchestra.run.smithi097.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 3.4 MB/s | 582 kB 00:00 2024-04-26T17:42:24.099 INFO:teuthology.orchestra.run.smithi097.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 339 kB/s | 25 kB 00:00 2024-04-26T17:42:24.124 INFO:teuthology.orchestra.run.smithi097.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-04-26T17:42:24.183 INFO:teuthology.orchestra.run.smithi097.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-04-26T17:42:24.208 INFO:teuthology.orchestra.run.smithi097.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-26T17:42:24.233 INFO:teuthology.orchestra.run.smithi097.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-04-26T17:42:24.370 INFO:teuthology.orchestra.run.smithi186.stdout:(117/152): python3-cherrypy-18.6.1-2.el9.noarch 705 kB/s | 358 kB 00:00 2024-04-26T17:42:24.434 INFO:teuthology.orchestra.run.smithi097.stdout:(66/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.2 MB/s | 3.0 MB 00:00 2024-04-26T17:42:24.524 INFO:teuthology.orchestra.run.smithi031.stdout:(66/152): ceph-test-19.0.0-3267.gbc94a2a9.el9.x 4.6 MB/s | 49 MB 00:10 2024-04-26T17:42:24.607 INFO:teuthology.orchestra.run.smithi031.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 7.8 MB/s | 4.7 MB 00:00 2024-04-26T17:42:24.732 INFO:teuthology.orchestra.run.smithi031.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 6.4 MB/s | 4.4 MB 00:00 2024-04-26T17:42:24.751 INFO:teuthology.orchestra.run.smithi097.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 5.0 MB/s | 1.6 MB 00:00 2024-04-26T17:42:24.816 INFO:teuthology.orchestra.run.smithi031.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-26T17:42:24.849 INFO:teuthology.orchestra.run.smithi031.stdout:(70/152): postgresql-13.14-1.el9.x86_64.rpm 4.9 MB/s | 1.6 MB 00:00 2024-04-26T17:42:24.883 INFO:teuthology.orchestra.run.smithi031.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 3.2 MB/s | 911 kB 00:00 2024-04-26T17:42:24.886 INFO:teuthology.orchestra.run.smithi186.stdout:(118/152): ceph-test-19.0.0-3267.gbc94a2a9.el9. 2.7 MB/s | 49 MB 00:17 2024-04-26T17:42:24.934 INFO:teuthology.orchestra.run.smithi097.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 4.9 MB/s | 911 kB 00:00 2024-04-26T17:42:24.950 INFO:teuthology.orchestra.run.smithi031.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-26T17:42:24.962 INFO:teuthology.orchestra.run.smithi186.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 512 kB/s | 303 kB 00:00 2024-04-26T17:42:25.001 INFO:teuthology.orchestra.run.smithi097.stdout:(69/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 5.8 MB/s | 4.4 MB 00:00 2024-04-26T17:42:25.008 INFO:teuthology.orchestra.run.smithi031.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-04-26T17:42:25.027 INFO:teuthology.orchestra.run.smithi097.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-04-26T17:42:25.075 INFO:teuthology.orchestra.run.smithi031.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-04-26T17:42:25.227 INFO:teuthology.orchestra.run.smithi097.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-04-26T17:42:25.267 INFO:teuthology.orchestra.run.smithi031.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.3 MB/s | 248 kB 00:00 2024-04-26T17:42:25.285 INFO:teuthology.orchestra.run.smithi097.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-26T17:42:25.304 INFO:teuthology.orchestra.run.smithi186.stdout:(120/152): python3-google-auth-2.29.0-1.el9.noa 602 kB/s | 206 kB 00:00 2024-04-26T17:42:25.334 INFO:teuthology.orchestra.run.smithi031.stdout:(76/152): postgresql-server-13.14-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-04-26T17:42:25.337 INFO:teuthology.orchestra.run.smithi186.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 78 kB/s | 35 kB 00:00 2024-04-26T17:42:25.376 INFO:teuthology.orchestra.run.smithi031.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 237 kB/s | 26 kB 00:00 2024-04-26T17:42:25.401 INFO:teuthology.orchestra.run.smithi031.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 288 kB/s | 19 kB 00:00 2024-04-26T17:42:25.436 INFO:teuthology.orchestra.run.smithi097.stdout:(73/152): openblas-openmp-0.3.21-2.el9.x86_64.r 3.9 MB/s | 4.7 MB 00:01 2024-04-26T17:42:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:(79/152): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-04-26T17:42:25.519 INFO:teuthology.orchestra.run.smithi031.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-04-26T17:42:25.519 INFO:teuthology.orchestra.run.smithi097.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-04-26T17:42:25.544 INFO:teuthology.orchestra.run.smithi031.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 244 kB/s | 35 kB 00:00 2024-04-26T17:42:25.603 INFO:teuthology.orchestra.run.smithi097.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-04-26T17:42:25.636 INFO:teuthology.orchestra.run.smithi031.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-04-26T17:42:25.653 INFO:teuthology.orchestra.run.smithi097.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 514 kB/s | 26 kB 00:00 2024-04-26T17:42:25.661 INFO:teuthology.orchestra.run.smithi031.stdout:(83/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-04-26T17:42:25.687 INFO:teuthology.orchestra.run.smithi186.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 413 kB/s | 144 kB 00:00 2024-04-26T17:42:25.703 INFO:teuthology.orchestra.run.smithi097.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-04-26T17:42:25.711 INFO:teuthology.orchestra.run.smithi031.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 77 kB 00:00 2024-04-26T17:42:25.728 INFO:teuthology.orchestra.run.smithi031.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 631 kB/s | 42 kB 00:00 2024-04-26T17:42:25.778 INFO:teuthology.orchestra.run.smithi097.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-26T17:42:25.796 INFO:teuthology.orchestra.run.smithi186.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-26T17:42:25.803 INFO:teuthology.orchestra.run.smithi031.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-26T17:42:25.829 INFO:teuthology.orchestra.run.smithi031.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 2.2 MB/s | 269 kB 00:00 2024-04-26T17:42:25.830 INFO:teuthology.orchestra.run.smithi097.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 693 kB/s | 35 kB 00:00 2024-04-26T17:42:25.895 INFO:teuthology.orchestra.run.smithi031.stdout:(88/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-26T17:42:25.896 INFO:teuthology.orchestra.run.smithi186.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-04-26T17:42:25.945 INFO:teuthology.orchestra.run.smithi031.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 436 kB/s | 51 kB 00:00 2024-04-26T17:42:25.962 INFO:teuthology.orchestra.run.smithi097.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 6.0 MB/s | 5.8 MB 00:00 2024-04-26T17:42:25.971 INFO:teuthology.orchestra.run.smithi031.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 715 kB/s | 54 kB 00:00 2024-04-26T17:42:25.996 INFO:teuthology.orchestra.run.smithi031.stdout:(91/152): python3-pyasn1-modules-0.4.8-6.el9.no 1.4 MB/s | 279 kB 00:00 2024-04-26T17:42:26.005 INFO:teuthology.orchestra.run.smithi186.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2024-04-26T17:42:26.046 INFO:teuthology.orchestra.run.smithi097.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 250 kB 00:00 2024-04-26T17:42:26.046 INFO:teuthology.orchestra.run.smithi031.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-04-26T17:42:26.072 INFO:teuthology.orchestra.run.smithi031.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 585 kB/s | 44 kB 00:00 2024-04-26T17:42:26.122 INFO:teuthology.orchestra.run.smithi031.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 721 kB/s | 36 kB 00:00 2024-04-26T17:42:26.129 INFO:teuthology.orchestra.run.smithi097.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 222 kB 00:00 2024-04-26T17:42:26.147 INFO:teuthology.orchestra.run.smithi186.stdout:(126/152): libarrow-9.0.0-11.el9.x86_64.rpm 829 kB/s | 4.4 MB 00:05 2024-04-26T17:42:26.147 INFO:teuthology.orchestra.run.smithi031.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-04-26T17:42:26.172 INFO:teuthology.orchestra.run.smithi186.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 116 kB/s | 19 kB 00:00 2024-04-26T17:42:26.180 INFO:teuthology.orchestra.run.smithi097.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-26T17:42:26.197 INFO:teuthology.orchestra.run.smithi031.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-04-26T17:42:26.230 INFO:teuthology.orchestra.run.smithi097.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 42 kB 00:00 2024-04-26T17:42:26.248 INFO:teuthology.orchestra.run.smithi186.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-26T17:42:26.256 INFO:teuthology.orchestra.run.smithi031.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 305 kB 00:00 2024-04-26T17:42:26.281 INFO:teuthology.orchestra.run.smithi031.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 692 kB/s | 58 kB 00:00 2024-04-26T17:42:26.314 INFO:teuthology.orchestra.run.smithi097.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 3.2 MB/s | 269 kB 00:00 2024-04-26T17:42:26.348 INFO:teuthology.orchestra.run.smithi186.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 464 kB/s | 46 kB 00:00 2024-04-26T17:42:26.357 INFO:teuthology.orchestra.run.smithi031.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 935 kB/s | 94 kB 00:00 2024-04-26T17:42:26.364 INFO:teuthology.orchestra.run.smithi097.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-26T17:42:26.382 INFO:teuthology.orchestra.run.smithi031.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 635 kB/s | 64 kB 00:00 2024-04-26T17:42:26.432 INFO:teuthology.orchestra.run.smithi031.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 597 kB/s | 45 kB 00:00 2024-04-26T17:42:26.458 INFO:teuthology.orchestra.run.smithi031.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-04-26T17:42:26.464 INFO:teuthology.orchestra.run.smithi097.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 279 kB 00:00 2024-04-26T17:42:26.531 INFO:teuthology.orchestra.run.smithi186.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 430 kB/s | 79 kB 00:00 2024-04-26T17:42:26.539 INFO:teuthology.orchestra.run.smithi097.stdout:(88/152): python3-babel-2.9.1-2.el9.noarch.rpm 4.8 MB/s | 6.0 MB 00:01 2024-04-26T17:42:26.565 INFO:teuthology.orchestra.run.smithi097.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 509 kB/s | 51 kB 00:00 2024-04-26T17:42:26.590 INFO:teuthology.orchestra.run.smithi097.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-26T17:42:26.640 INFO:teuthology.orchestra.run.smithi097.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 833 kB/s | 42 kB 00:00 2024-04-26T17:42:26.715 INFO:teuthology.orchestra.run.smithi186.stdout:(131/152): python3-natsort-7.1.1-5.el9.noarch.r 315 kB/s | 58 kB 00:00 2024-04-26T17:42:26.732 INFO:teuthology.orchestra.run.smithi097.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 482 kB/s | 44 kB 00:00 2024-04-26T17:42:26.741 INFO:teuthology.orchestra.run.smithi031.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-04-26T17:42:26.774 INFO:teuthology.orchestra.run.smithi031.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-04-26T17:42:26.816 INFO:teuthology.orchestra.run.smithi097.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-04-26T17:42:26.875 INFO:teuthology.orchestra.run.smithi097.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 617 kB/s | 36 kB 00:00 2024-04-26T17:42:26.933 INFO:teuthology.orchestra.run.smithi031.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 123 kB/s | 19 kB 00:00 2024-04-26T17:42:26.958 INFO:teuthology.orchestra.run.smithi031.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-26T17:42:26.966 INFO:teuthology.orchestra.run.smithi097.stdout:(95/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.5 MB/s | 5.1 MB 00:01 2024-04-26T17:42:26.992 INFO:teuthology.orchestra.run.smithi097.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 405 kB/s | 47 kB 00:00 2024-04-26T17:42:27.025 INFO:teuthology.orchestra.run.smithi031.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-04-26T17:42:27.050 INFO:teuthology.orchestra.run.smithi097.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 994 kB/s | 58 kB 00:00 2024-04-26T17:42:27.059 INFO:teuthology.orchestra.run.smithi031.stdout:(108/152): gperftools-libs-2.9.1-2.el9.x86_64.r 973 kB/s | 309 kB 00:00 2024-04-26T17:42:27.084 INFO:teuthology.orchestra.run.smithi097.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.5 MB/s | 305 kB 00:00 2024-04-26T17:42:27.092 INFO:teuthology.orchestra.run.smithi031.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 725 kB/s | 49 kB 00:00 2024-04-26T17:42:27.109 INFO:teuthology.orchestra.run.smithi097.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-26T17:42:27.126 INFO:teuthology.orchestra.run.smithi031.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-26T17:42:27.143 INFO:teuthology.orchestra.run.smithi097.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-26T17:42:27.168 INFO:teuthology.orchestra.run.smithi031.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-26T17:42:27.168 INFO:teuthology.orchestra.run.smithi097.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 766 kB/s | 45 kB 00:00 2024-04-26T17:42:27.193 INFO:teuthology.orchestra.run.smithi097.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 443 kB/s | 22 kB 00:00 2024-04-26T17:42:27.195 INFO:teuthology.orchestra.run.smithi186.stdout:(132/152): python3-pecan-1.4.2-3.el9.noarch.rpm 568 kB/s | 272 kB 00:00 2024-04-26T17:42:27.293 INFO:teuthology.orchestra.run.smithi031.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-04-26T17:42:27.360 INFO:teuthology.orchestra.run.smithi097.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 4.4 MB/s | 863 kB 00:00 2024-04-26T17:42:27.366 INFO:teuthology.orchestra.run.smithi186.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 96 kB/s | 16 kB 00:00 2024-04-26T17:42:27.377 INFO:teuthology.orchestra.run.smithi031.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 522 kB/s | 43 kB 00:00 2024-04-26T17:42:27.410 INFO:teuthology.orchestra.run.smithi031.stdout:(114/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.9 MB/s | 839 kB 00:00 2024-04-26T17:42:27.477 INFO:teuthology.orchestra.run.smithi031.stdout:(115/152): libarrow-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 4.4 MB 00:00 2024-04-26T17:42:27.503 INFO:teuthology.orchestra.run.smithi031.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-04-26T17:42:27.528 INFO:teuthology.orchestra.run.smithi031.stdout:(117/152): python3-cachetools-4.2.4-1.el9.noarc 213 kB/s | 32 kB 00:00 2024-04-26T17:42:27.553 INFO:teuthology.orchestra.run.smithi031.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-26T17:42:27.557 INFO:teuthology.orchestra.run.smithi186.stdout:(134/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 470 kB/s | 90 kB 00:00 2024-04-26T17:42:27.578 INFO:teuthology.orchestra.run.smithi031.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-04-26T17:42:27.604 INFO:teuthology.orchestra.run.smithi031.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-04-26T17:42:27.624 INFO:teuthology.orchestra.run.smithi186.stdout:(135/152): python3-kubernetes-26.1.0-2.el9.noar 720 kB/s | 1.0 MB 00:01 2024-04-26T17:42:27.629 INFO:teuthology.orchestra.run.smithi031.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 469 kB/s | 35 kB 00:00 2024-04-26T17:42:27.654 INFO:teuthology.orchestra.run.smithi031.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-26T17:42:27.661 INFO:teuthology.orchestra.run.smithi097.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-26T17:42:27.666 INFO:teuthology.orchestra.run.smithi186.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 259 kB/s | 28 kB 00:00 2024-04-26T17:42:27.705 INFO:teuthology.orchestra.run.smithi031.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-04-26T17:42:27.730 INFO:teuthology.orchestra.run.smithi031.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-04-26T17:42:27.733 INFO:teuthology.orchestra.run.smithi186.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-04-26T17:42:27.736 INFO:teuthology.orchestra.run.smithi097.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 260 kB/s | 19 kB 00:00 2024-04-26T17:42:27.778 INFO:teuthology.orchestra.run.smithi097.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 740 kB/s | 309 kB 00:00 2024-04-26T17:42:27.788 INFO:teuthology.orchestra.run.smithi031.stdout:(125/152): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-04-26T17:42:27.792 INFO:teuthology.orchestra.run.smithi186.stdout:(138/152): python3-grpcio-1.46.7-10.el9.x86_64. 839 kB/s | 2.0 MB 00:02 2024-04-26T17:42:27.813 INFO:teuthology.orchestra.run.smithi031.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 162 kB/s | 18 kB 00:00 2024-04-26T17:42:27.830 INFO:teuthology.orchestra.run.smithi031.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-04-26T17:42:27.853 INFO:teuthology.orchestra.run.smithi097.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-04-26T17:42:27.856 INFO:teuthology.orchestra.run.smithi031.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 290 kB/s | 19 kB 00:00 2024-04-26T17:42:27.897 INFO:teuthology.orchestra.run.smithi031.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 295 kB/s | 20 kB 00:00 2024-04-26T17:42:27.900 INFO:teuthology.orchestra.run.smithi186.stdout:(139/152): python3-tempora-5.0.0-2.el9.noarch.r 330 kB/s | 36 kB 00:00 2024-04-26T17:42:27.926 INFO:teuthology.orchestra.run.smithi186.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 307 kB/s | 59 kB 00:00 2024-04-26T17:42:27.931 INFO:teuthology.orchestra.run.smithi031.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-04-26T17:42:27.945 INFO:teuthology.orchestra.run.smithi097.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-04-26T17:42:27.951 INFO:teuthology.orchestra.run.smithi186.stdout:(141/152): python3-routes-2.5.1-5.el9.noarch.rp 663 kB/s | 188 kB 00:00 2024-04-26T17:42:27.956 INFO:teuthology.orchestra.run.smithi031.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 461 kB/s | 46 kB 00:00 2024-04-26T17:42:27.982 INFO:teuthology.orchestra.run.smithi031.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 940 kB/s | 79 kB 00:00 2024-04-26T17:42:28.007 INFO:teuthology.orchestra.run.smithi031.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 765 kB/s | 58 kB 00:00 2024-04-26T17:42:28.009 INFO:teuthology.orchestra.run.smithi186.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 469 kB/s | 51 kB 00:00 2024-04-26T17:42:28.032 INFO:teuthology.orchestra.run.smithi031.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-26T17:42:28.054 INFO:teuthology.orchestra.run.smithi097.stdout:(109/152): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-26T17:42:28.057 INFO:teuthology.orchestra.run.smithi031.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-04-26T17:42:28.079 INFO:teuthology.orchestra.run.smithi097.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 505 kB/s | 67 kB 00:00 2024-04-26T17:42:28.083 INFO:teuthology.orchestra.run.smithi031.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-26T17:42:28.108 INFO:teuthology.orchestra.run.smithi031.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-04-26T17:42:28.129 INFO:teuthology.orchestra.run.smithi097.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-26T17:42:28.133 INFO:teuthology.orchestra.run.smithi031.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 409 kB/s | 31 kB 00:00 2024-04-26T17:42:28.135 INFO:teuthology.orchestra.run.smithi186.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 488 kB/s | 90 kB 00:00 2024-04-26T17:42:28.158 INFO:teuthology.orchestra.run.smithi031.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-26T17:42:28.184 INFO:teuthology.orchestra.run.smithi031.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 785 kB/s | 59 kB 00:00 2024-04-26T17:42:28.209 INFO:teuthology.orchestra.run.smithi031.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-04-26T17:42:28.213 INFO:teuthology.orchestra.run.smithi097.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-04-26T17:42:28.234 INFO:teuthology.orchestra.run.smithi031.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 675 kB/s | 51 kB 00:00 2024-04-26T17:42:28.235 INFO:teuthology.orchestra.run.smithi186.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2024-04-26T17:42:28.255 INFO:teuthology.orchestra.run.smithi097.stdout:(113/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.7 MB/s | 839 kB 00:00 2024-04-26T17:42:28.259 INFO:teuthology.orchestra.run.smithi031.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-26T17:42:28.277 INFO:teuthology.orchestra.run.smithi186.stdout:(145/152): python3-webob-1.8.7-6.el9.noarch.rpm 654 kB/s | 230 kB 00:00 2024-04-26T17:42:28.280 INFO:teuthology.orchestra.run.smithi097.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 648 kB/s | 43 kB 00:00 2024-04-26T17:42:28.285 INFO:teuthology.orchestra.run.smithi031.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-26T17:42:28.310 INFO:teuthology.orchestra.run.smithi031.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-04-26T17:42:28.322 INFO:teuthology.orchestra.run.smithi097.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 481 kB/s | 32 kB 00:00 2024-04-26T17:42:28.335 INFO:teuthology.orchestra.run.smithi031.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-04-26T17:42:28.347 INFO:teuthology.orchestra.run.smithi097.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-04-26T17:42:28.360 INFO:teuthology.orchestra.run.smithi031.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-26T17:42:28.398 INFO:teuthology.orchestra.run.smithi097.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-26T17:42:28.419 INFO:teuthology.orchestra.run.smithi031.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-26T17:42:28.423 INFO:teuthology.orchestra.run.smithi097.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-26T17:42:28.482 INFO:teuthology.orchestra.run.smithi097.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-04-26T17:42:28.498 INFO:teuthology.orchestra.run.smithi097.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 469 kB/s | 35 kB 00:00 2024-04-26T17:42:28.502 INFO:teuthology.orchestra.run.smithi186.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 718 kB/s | 192 kB 00:00 2024-04-26T17:42:28.565 INFO:teuthology.orchestra.run.smithi097.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.4 MB/s | 206 kB 00:00 2024-04-26T17:42:28.615 INFO:teuthology.orchestra.run.smithi097.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 17 MB/s | 2.0 MB 00:00 2024-04-26T17:42:28.644 INFO:teuthology.orchestra.run.smithi186.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 673 kB/s | 427 kB 00:00 2024-04-26T17:42:28.649 INFO:teuthology.orchestra.run.smithi097.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-04-26T17:42:28.693 INFO:teuthology.orchestra.run.smithi097.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-04-26T17:42:28.725 INFO:teuthology.orchestra.run.smithi097.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-04-26T17:42:28.744 INFO:teuthology.orchestra.run.smithi031.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 4.2 MB/s | 1.3 MB 00:00 2024-04-26T17:42:28.767 INFO:teuthology.orchestra.run.smithi097.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-04-26T17:42:28.800 INFO:teuthology.orchestra.run.smithi097.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-04-26T17:42:28.842 INFO:teuthology.orchestra.run.smithi097.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-04-26T17:42:28.903 INFO:teuthology.orchestra.run.smithi031.stdout:(150/152): librbd1-19.0.0-3267.gbc94a2a9.el9.x8 5.8 MB/s | 3.1 MB 00:00 2024-04-26T17:42:28.918 INFO:teuthology.orchestra.run.smithi097.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-04-26T17:42:28.943 INFO:teuthology.orchestra.run.smithi097.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 462 kB/s | 46 kB 00:00 2024-04-26T17:42:28.985 INFO:teuthology.orchestra.run.smithi097.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-04-26T17:42:29.019 INFO:teuthology.orchestra.run.smithi097.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 766 kB/s | 58 kB 00:00 2024-04-26T17:42:29.028 INFO:teuthology.orchestra.run.smithi031.stdout:(151/152): librados2-19.0.0-3267.gbc94a2a9.el9. 4.9 MB/s | 3.4 MB 00:00 2024-04-26T17:42:29.069 INFO:teuthology.orchestra.run.smithi097.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-26T17:42:29.094 INFO:teuthology.orchestra.run.smithi097.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-04-26T17:42:29.120 INFO:teuthology.orchestra.run.smithi031.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 6.4 MB/s | 2.4 MB 00:00 2024-04-26T17:42:29.122 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-26T17:42:29.123 INFO:teuthology.orchestra.run.smithi031.stdout:Total 9.6 MB/s | 222 MB 00:23 2024-04-26T17:42:29.144 INFO:teuthology.orchestra.run.smithi097.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-26T17:42:29.169 INFO:teuthology.orchestra.run.smithi186.stdout:(148/152): librbd1-19.0.0-3267.gbc94a2a9.el9.x8 6.0 MB/s | 3.1 MB 00:00 2024-04-26T17:42:29.170 INFO:teuthology.orchestra.run.smithi097.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 376 kB/s | 28 kB 00:00 2024-04-26T17:42:29.278 INFO:teuthology.orchestra.run.smithi186.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 13 MB/s | 1.3 MB 00:00 2024-04-26T17:42:29.328 INFO:teuthology.orchestra.run.smithi097.stdout:(137/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.9 MB/s | 16 MB 00:02 2024-04-26T17:42:29.354 INFO:teuthology.orchestra.run.smithi097.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 147 kB/s | 31 kB 00:00 2024-04-26T17:42:29.379 INFO:teuthology.orchestra.run.smithi097.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 903 kB/s | 188 kB 00:00 2024-04-26T17:42:29.420 INFO:teuthology.orchestra.run.smithi186.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 17 MB/s | 2.4 MB 00:00 2024-04-26T17:42:29.429 INFO:teuthology.orchestra.run.smithi097.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 477 kB/s | 36 kB 00:00 2024-04-26T17:42:29.479 INFO:teuthology.orchestra.run.smithi097.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 700 kB/s | 51 kB 00:00 2024-04-26T17:42:29.513 INFO:teuthology.orchestra.run.smithi097.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-26T17:42:29.555 INFO:teuthology.orchestra.run.smithi097.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-26T17:42:29.597 INFO:teuthology.orchestra.run.smithi097.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-04-26T17:42:29.620 INFO:teuthology.orchestra.run.smithi186.stdout:(151/152): librados2-19.0.0-3267.gbc94a2a9.el9. 3.0 MB/s | 3.4 MB 00:01 2024-04-26T17:42:29.630 INFO:teuthology.orchestra.run.smithi097.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-04-26T17:42:29.656 INFO:teuthology.orchestra.run.smithi097.stdout:(146/152): python3-rsa-4.9-2.el9.noarch.rpm 181 kB/s | 59 kB 00:00 2024-04-26T17:42:29.681 INFO:teuthology.orchestra.run.smithi097.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-26T17:42:29.723 INFO:teuthology.orchestra.run.smithi097.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-26T17:42:30.023 INFO:teuthology.orchestra.run.smithi097.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 4.5 MB/s | 1.3 MB 00:00 2024-04-26T17:42:30.190 INFO:teuthology.orchestra.run.smithi097.stdout:(150/152): librados2-19.0.0-3267.gbc94a2a9.el9. 6.4 MB/s | 3.4 MB 00:00 2024-04-26T17:42:30.222 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-26T17:42:30.381 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-26T17:42:30.381 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-26T17:42:30.387 INFO:teuthology.orchestra.run.smithi186.stdout:(152/152): thrift-0.15.0-2.el9.x86_64.rpm 768 kB/s | 1.6 MB 00:02 2024-04-26T17:42:30.393 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-26T17:42:30.393 INFO:teuthology.orchestra.run.smithi186.stdout:Total 8.2 MB/s | 222 MB 00:26 2024-04-26T17:42:30.449 INFO:teuthology.orchestra.run.smithi097.stdout:(151/152): librbd1-19.0.0-3267.gbc94a2a9.el9.x8 4.1 MB/s | 3.1 MB 00:00 2024-04-26T17:42:30.599 INFO:teuthology.orchestra.run.smithi097.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 4.2 MB/s | 2.4 MB 00:00 2024-04-26T17:42:30.604 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-26T17:42:30.605 INFO:teuthology.orchestra.run.smithi097.stdout:Total 10 MB/s | 222 MB 00:22 2024-04-26T17:42:31.508 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-26T17:42:31.650 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-26T17:42:31.651 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-26T17:42:31.714 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-26T17:42:31.857 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-26T17:42:31.858 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-26T17:42:32.260 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-26T17:42:32.260 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-26T17:42:33.422 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-26T17:42:33.422 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-26T17:42:33.607 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-26T17:42:33.607 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-26T17:42:38.298 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-26T17:42:38.398 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T17:42:38.614 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T17:42:38.668 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T17:42:38.709 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T17:42:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T17:42:38.778 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T17:42:39.122 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T17:42:39.225 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T17:42:39.333 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T17:42:39.359 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T17:42:39.422 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-26T17:42:39.452 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T17:42:39.480 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 10/156 2024-04-26T17:42:39.507 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T17:42:39.551 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T17:42:39.608 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T17:42:39.641 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T17:42:39.642 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T17:42:39.667 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T17:42:39.695 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T17:42:39.709 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T17:42:39.713 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T17:42:39.760 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-26T17:42:39.819 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T17:42:39.848 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 14/156 2024-04-26T17:42:39.853 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T17:42:39.891 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T17:42:39.911 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 14/156 2024-04-26T17:42:39.933 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T17:42:39.945 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T17:42:39.958 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T17:42:39.978 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T17:42:40.004 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T17:42:40.038 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T17:42:40.047 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T17:42:40.143 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T17:42:40.158 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T17:42:40.190 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T17:42:40.265 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T17:42:40.293 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T17:42:40.302 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T17:42:40.373 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T17:42:40.392 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T17:42:40.399 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T17:42:40.431 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T17:42:40.431 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 10/156 2024-04-26T17:42:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T17:42:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T17:42:40.542 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T17:42:40.580 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T17:42:40.590 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T17:42:40.609 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T17:42:40.628 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T17:42:40.634 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T17:42:40.639 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T17:42:40.656 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T17:42:40.666 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T17:42:40.694 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T17:42:40.731 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T17:42:40.759 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T17:42:40.766 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T17:42:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 10/156 2024-04-26T17:42:40.782 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 14/156 2024-04-26T17:42:40.826 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T17:42:40.836 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 14/156 2024-04-26T17:42:40.855 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T17:42:40.858 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T17:42:40.881 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T17:42:40.899 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T17:42:40.916 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T17:42:40.965 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T17:42:40.978 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T17:42:40.993 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T17:42:41.009 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9. 31/156 2024-04-26T17:42:41.053 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el 32/156 2024-04-26T17:42:41.075 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T17:42:41.081 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_6 33/156 2024-04-26T17:42:41.097 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T17:42:41.107 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T17:42:41.119 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 14/156 2024-04-26T17:42:41.147 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T17:42:41.182 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 14/156 2024-04-26T17:42:41.203 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T17:42:41.205 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T17:42:41.216 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T17:42:41.226 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T17:42:41.283 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T17:42:41.313 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T17:42:41.348 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T17:42:41.383 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T17:42:41.428 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T17:42:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T17:42:41.457 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T17:42:41.470 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T17:42:41.483 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T17:42:41.581 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T17:42:41.676 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T17:42:41.689 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T17:42:41.719 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T17:42:41.749 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T17:42:41.763 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T17:42:41.799 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T17:42:41.830 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T17:42:41.837 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T17:42:41.878 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T17:42:41.922 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T17:42:41.924 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T17:42:41.954 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T17:42:41.982 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T17:42:41.983 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T17:42:42.014 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T17:42:42.041 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T17:42:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T17:42:42.053 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T17:42:42.105 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T17:42:42.118 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T17:42:42.130 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T17:42:42.135 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T17:42:42.152 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9. 31/156 2024-04-26T17:42:42.164 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T17:42:42.170 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T17:42:42.195 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el 32/156 2024-04-26T17:42:42.207 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T17:42:42.223 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_6 33/156 2024-04-26T17:42:42.245 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_6 43/156 2024-04-26T17:42:42.247 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T17:42:42.281 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9. 31/156 2024-04-26T17:42:42.287 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T17:42:42.293 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T17:42:42.324 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el 32/156 2024-04-26T17:42:42.339 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T17:42:42.353 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_6 33/156 2024-04-26T17:42:42.414 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T17:42:42.423 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T17:42:42.468 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T17:42:42.482 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T17:42:42.539 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T17:42:42.544 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T17:42:42.548 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T17:42:42.596 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T17:42:42.615 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T17:42:42.671 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T17:42:42.675 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T17:42:42.706 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T17:42:42.729 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T17:42:42.743 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T17:42:42.791 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T17:42:42.825 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T17:42:42.870 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T17:42:42.911 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T17:42:43.055 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T17:42:43.142 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T17:42:43.218 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T17:42:43.248 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T17:42:43.255 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T17:42:43.308 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T17:42:43.317 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T17:42:43.341 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T17:42:43.346 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_6 43/156 2024-04-26T17:42:43.363 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T17:42:43.369 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T17:42:43.387 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T17:42:43.405 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T17:42:43.412 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T17:42:43.451 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_6 43/156 2024-04-26T17:42:43.492 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T17:42:43.505 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T17:42:43.537 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T17:42:43.542 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T17:42:43.577 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T17:42:43.599 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T17:42:43.611 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T17:42:43.635 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T17:42:43.652 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T17:42:43.653 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T17:42:43.703 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T17:42:43.714 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T17:42:43.725 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T17:42:43.757 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T17:42:43.759 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T17:42:43.794 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T17:42:43.840 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T17:42:43.841 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T17:42:43.843 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T17:42:43.871 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T17:42:43.885 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T17:42:43.892 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T17:42:43.907 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T17:42:43.921 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T17:42:43.928 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T17:42:43.954 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T17:42:43.955 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T17:42:43.970 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T17:42:43.990 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T17:42:43.994 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T17:42:44.033 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T17:42:44.034 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T17:42:44.064 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T17:42:44.075 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T17:42:44.098 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T17:42:44.122 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T17:42:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T17:42:44.173 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T17:42:44.175 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T17:42:44.206 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T17:42:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T17:42:44.213 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T17:42:44.249 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T17:42:44.255 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T17:42:44.286 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T17:42:44.289 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T17:42:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T17:42:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T17:42:44.340 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T17:42:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T17:42:44.372 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T17:42:44.404 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T17:42:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T17:42:44.422 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T17:42:44.429 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T17:42:44.454 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T17:42:44.461 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T17:42:44.501 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T17:42:44.526 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T17:42:44.528 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T17:42:44.528 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T17:42:44.551 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T17:42:44.579 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T17:42:44.627 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T17:42:44.655 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T17:42:44.686 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T17:42:44.714 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T17:42:44.749 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T17:42:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T17:42:44.783 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T17:42:44.788 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T17:42:44.811 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T17:42:44.844 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T17:42:44.845 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T17:42:44.874 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T17:42:44.884 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T17:42:44.900 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T17:42:44.916 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T17:42:44.934 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T17:42:44.946 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T17:42:44.967 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T17:42:44.981 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T17:42:45.002 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T17:42:45.030 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T17:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T17:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T17:42:45.165 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T17:42:45.165 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T17:42:45.165 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T17:42:45.194 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T17:42:45.210 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T17:42:45.237 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T17:42:45.268 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T17:42:45.308 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T17:42:45.316 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T17:42:45.364 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T17:42:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T17:42:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T17:42:45.417 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T17:42:45.450 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T17:42:45.470 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T17:42:45.555 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T17:42:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T17:42:45.717 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T17:42:45.761 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T17:42:45.945 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T17:42:45.975 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T17:42:46.055 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T17:42:46.084 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T17:42:46.129 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T17:42:46.162 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T17:42:46.204 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T17:42:46.251 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T17:42:46.259 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T17:42:46.284 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T17:42:46.311 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T17:42:46.385 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T17:42:46.399 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T17:42:46.439 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T17:42:46.442 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T17:42:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T17:42:46.618 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T17:42:46.636 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T17:42:46.653 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T17:42:46.734 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T17:42:46.746 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T17:42:46.823 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T17:42:46.850 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T17:42:46.878 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T17:42:46.928 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T17:42:47.375 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T17:42:47.502 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T17:42:47.581 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T17:42:47.691 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T17:42:48.198 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T17:42:48.269 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T17:42:48.297 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T17:42:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T17:42:48.659 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T17:42:48.717 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 99/156 2024-04-26T17:42:48.747 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T17:42:48.809 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 99/156 2024-04-26T17:42:48.832 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T17:42:48.839 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 100/156 2024-04-26T17:42:48.859 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T17:42:48.875 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T17:42:48.893 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T17:42:48.968 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T17:42:49.054 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T17:42:49.082 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T17:42:49.115 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T17:42:49.206 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T17:42:49.248 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 99/156 2024-04-26T17:42:49.338 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 99/156 2024-04-26T17:42:49.369 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 100/156 2024-04-26T17:42:49.380 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T17:42:49.405 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T17:42:49.412 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 103/156 2024-04-26T17:42:49.427 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T17:42:49.470 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 99/156 2024-04-26T17:42:49.492 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 103/156 2024-04-26T17:42:49.513 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 104/156 2024-04-26T17:42:49.560 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 99/156 2024-04-26T17:42:49.583 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 100/156 2024-04-26T17:42:49.619 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T17:42:49.888 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T17:42:49.917 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 103/156 2024-04-26T17:42:49.988 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 103/156 2024-04-26T17:42:50.010 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 104/156 2024-04-26T17:42:50.099 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T17:42:50.131 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 103/156 2024-04-26T17:42:50.202 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 103/156 2024-04-26T17:42:50.226 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 104/156 2024-04-26T17:42:51.911 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T17:42:51.943 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T17:42:52.304 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T17:42:52.482 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T17:42:52.498 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T17:42:52.514 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T17:42:52.520 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T17:42:52.563 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 106/156 2024-04-26T17:42:52.643 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 106/156 2024-04-26T17:42:52.644 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T17:42:52.644 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T17:42:52.644 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:42:52.681 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 107/156 2024-04-26T17:42:53.048 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T17:42:53.069 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 105/156 2024-04-26T17:42:53.093 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 106/156 2024-04-26T17:42:53.115 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 106/156 2024-04-26T17:42:53.186 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 106/156 2024-04-26T17:42:53.187 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T17:42:53.187 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T17:42:53.187 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:42:53.208 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 106/156 2024-04-26T17:42:53.208 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T17:42:53.208 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T17:42:53.209 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:42:53.210 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 107/156 2024-04-26T17:42:53.241 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 107/156 2024-04-26T17:43:04.435 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 107/156 2024-04-26T17:43:04.435 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-04-26T17:43:04.435 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-04-26T17:43:04.435 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-04-26T17:43:04.436 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-04-26T17:43:04.436 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-04-26T17:43:04.436 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-04-26T17:43:04.436 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-04-26T17:43:04.436 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:43:04.606 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 108/156 2024-04-26T17:43:04.621 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 108/156 2024-04-26T17:43:04.621 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:04.621 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T17:43:04.621 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:43:04.707 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 107/156 2024-04-26T17:43:04.707 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2024-04-26T17:43:04.707 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2024-04-26T17:43:04.707 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2024-04-26T17:43:04.707 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2024-04-26T17:43:04.707 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2024-04-26T17:43:04.707 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2024-04-26T17:43:04.707 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2024-04-26T17:43:04.707 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:43:04.802 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 107/156 2024-04-26T17:43:04.803 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /sys 2024-04-26T17:43:04.803 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /proc 2024-04-26T17:43:04.803 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /mnt 2024-04-26T17:43:04.803 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /var/tmp 2024-04-26T17:43:04.803 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /home 2024-04-26T17:43:04.803 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /root 2024-04-26T17:43:04.803 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /tmp 2024-04-26T17:43:04.803 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:43:04.882 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 108/156 2024-04-26T17:43:04.900 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 108/156 2024-04-26T17:43:04.901 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:04.901 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T17:43:04.901 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:43:04.978 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 108/156 2024-04-26T17:43:04.994 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 108/156 2024-04-26T17:43:04.994 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:04.994 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T17:43:04.995 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:43:06.133 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 109/156 2024-04-26T17:43:06.159 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 109/156 2024-04-26T17:43:06.160 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:06.160 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T17:43:06.160 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T17:43:06.160 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T17:43:06.160 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:43:06.384 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 110/156 2024-04-26T17:43:06.406 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 110/156 2024-04-26T17:43:06.406 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:06.406 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T17:43:06.406 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T17:43:06.406 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T17:43:06.407 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:43:06.434 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 109/156 2024-04-26T17:43:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 109/156 2024-04-26T17:43:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 109/156 2024-04-26T17:43:06.461 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:06.461 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T17:43:06.461 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T17:43:06.461 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T17:43:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:43:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 109/156 2024-04-26T17:43:06.474 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:06.474 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T17:43:06.474 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T17:43:06.474 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T17:43:06.474 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:43:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 110/156 2024-04-26T17:43:06.698 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 110/156 2024-04-26T17:43:06.717 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 110/156 2024-04-26T17:43:06.717 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:06.717 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T17:43:06.717 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T17:43:06.717 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T17:43:06.717 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:43:06.721 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 110/156 2024-04-26T17:43:06.721 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:06.721 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T17:43:06.721 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T17:43:06.721 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T17:43:06.721 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:43:06.851 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 111/156 2024-04-26T17:43:06.874 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 111/156 2024-04-26T17:43:06.874 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:06.874 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T17:43:06.874 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T17:43:06.874 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T17:43:06.874 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:43:06.981 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T17:43:07.039 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T17:43:07.102 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T17:43:07.166 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 111/156 2024-04-26T17:43:07.177 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 111/156 2024-04-26T17:43:07.189 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 111/156 2024-04-26T17:43:07.189 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:07.189 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T17:43:07.189 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T17:43:07.189 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T17:43:07.189 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:43:07.201 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 111/156 2024-04-26T17:43:07.201 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:07.201 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T17:43:07.201 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T17:43:07.201 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T17:43:07.202 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:43:07.288 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T17:43:07.288 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T17:43:07.309 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T17:43:07.335 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T17:43:07.346 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T17:43:07.367 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T17:43:07.401 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T17:43:07.403 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T17:43:07.431 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T17:43:07.493 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T17:43:07.591 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T17:43:07.625 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T17:43:07.627 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T17:43:07.634 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T17:43:07.668 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T17:43:07.679 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T17:43:07.700 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T17:43:07.713 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T17:43:07.728 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T17:43:07.760 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T17:43:07.819 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T17:43:07.850 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T17:43:07.959 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T17:43:07.994 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T17:43:08.000 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T17:43:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T17:43:08.038 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T17:43:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T17:43:08.068 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T17:43:08.096 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T17:43:08.163 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 123/156 2024-04-26T17:43:08.185 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 123/156 2024-04-26T17:43:08.217 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.e 124/156 2024-04-26T17:43:08.279 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9. 125/156 2024-04-26T17:43:08.308 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 126/156 2024-04-26T17:43:08.492 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 123/156 2024-04-26T17:43:08.495 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 123/156 2024-04-26T17:43:08.519 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 123/156 2024-04-26T17:43:08.524 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 123/156 2024-04-26T17:43:08.552 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.e 124/156 2024-04-26T17:43:08.557 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.e 124/156 2024-04-26T17:43:08.619 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9. 125/156 2024-04-26T17:43:08.625 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9. 125/156 2024-04-26T17:43:08.650 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 126/156 2024-04-26T17:43:08.650 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 126/156 2024-04-26T17:43:09.353 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 126/156 2024-04-26T17:43:09.384 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 127/156 2024-04-26T17:43:09.690 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 126/156 2024-04-26T17:43:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 127/156 2024-04-26T17:43:09.754 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 126/156 2024-04-26T17:43:09.785 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 127/156 2024-04-26T17:43:10.353 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 127/156 2024-04-26T17:43:10.376 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 128/156 2024-04-26T17:43:10.479 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 128/156 2024-04-26T17:43:10.647 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el 129/156 2024-04-26T17:43:10.687 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 130/156 2024-04-26T17:43:10.690 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 127/156 2024-04-26T17:43:10.711 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 130/156 2024-04-26T17:43:10.711 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:10.711 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T17:43:10.711 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T17:43:10.711 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T17:43:10.712 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:43:10.716 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 128/156 2024-04-26T17:43:10.754 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 131/156 2024-04-26T17:43:10.765 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 131/156 2024-04-26T17:43:10.802 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 127/156 2024-04-26T17:43:10.820 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 128/156 2024-04-26T17:43:10.827 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 128/156 2024-04-26T17:43:10.864 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 132/156 2024-04-26T17:43:10.912 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 133/156 2024-04-26T17:43:10.936 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 128/156 2024-04-26T17:43:10.978 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el 129/156 2024-04-26T17:43:11.002 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 130/156 2024-04-26T17:43:11.025 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 130/156 2024-04-26T17:43:11.025 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:11.025 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T17:43:11.025 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T17:43:11.025 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T17:43:11.025 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:43:11.069 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 131/156 2024-04-26T17:43:11.080 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 131/156 2024-04-26T17:43:11.104 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el 129/156 2024-04-26T17:43:11.129 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 130/156 2024-04-26T17:43:11.153 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 130/156 2024-04-26T17:43:11.153 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:11.153 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T17:43:11.153 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T17:43:11.153 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T17:43:11.153 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:43:11.170 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 132/156 2024-04-26T17:43:11.204 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 131/156 2024-04-26T17:43:11.215 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 131/156 2024-04-26T17:43:11.218 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 133/156 2024-04-26T17:43:11.309 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 132/156 2024-04-26T17:43:11.362 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 133/156 2024-04-26T17:43:11.840 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T17:43:11.865 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 135/156 2024-04-26T17:43:11.888 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 135/156 2024-04-26T17:43:11.888 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:11.888 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T17:43:11.888 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T17:43:11.888 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T17:43:11.888 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:43:11.932 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 136/156 2024-04-26T17:43:11.954 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 136/156 2024-04-26T17:43:11.954 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:11.954 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T17:43:11.955 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:43:11.999 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 137/156 2024-04-26T17:43:12.021 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 137/156 2024-04-26T17:43:12.022 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:12.022 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T17:43:12.022 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T17:43:12.022 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T17:43:12.022 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:43:12.144 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T17:43:12.180 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 135/156 2024-04-26T17:43:12.203 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 135/156 2024-04-26T17:43:12.203 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:12.203 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T17:43:12.203 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T17:43:12.203 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T17:43:12.203 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:43:12.248 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 136/156 2024-04-26T17:43:12.269 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 136/156 2024-04-26T17:43:12.269 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:12.269 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T17:43:12.269 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:43:12.314 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 137/156 2024-04-26T17:43:12.325 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 138/156 2024-04-26T17:43:12.336 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T17:43:12.337 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 137/156 2024-04-26T17:43:12.337 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:12.337 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T17:43:12.337 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T17:43:12.337 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T17:43:12.337 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:43:12.348 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 138/156 2024-04-26T17:43:12.348 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:12.348 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T17:43:12.348 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T17:43:12.349 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T17:43:12.349 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:43:12.365 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 135/156 2024-04-26T17:43:12.389 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 135/156 2024-04-26T17:43:12.389 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:12.389 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T17:43:12.389 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T17:43:12.389 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T17:43:12.389 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:43:12.432 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 136/156 2024-04-26T17:43:12.455 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 136/156 2024-04-26T17:43:12.455 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:12.455 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T17:43:12.455 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:43:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 137/156 2024-04-26T17:43:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 137/156 2024-04-26T17:43:12.523 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:12.523 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T17:43:12.523 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T17:43:12.523 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T17:43:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:43:12.640 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 138/156 2024-04-26T17:43:12.662 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 138/156 2024-04-26T17:43:12.663 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:12.663 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T17:43:12.663 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T17:43:12.663 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T17:43:12.663 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:43:12.842 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 138/156 2024-04-26T17:43:12.865 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 138/156 2024-04-26T17:43:12.865 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T17:43:12.865 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T17:43:12.865 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T17:43:12.866 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T17:43:12.866 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:43:16.995 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 139/156 2024-04-26T17:43:17.036 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 140/156 2024-04-26T17:43:17.077 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 141/156 2024-04-26T17:43:17.109 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T17:43:17.137 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T17:43:17.277 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 139/156 2024-04-26T17:43:17.318 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 140/156 2024-04-26T17:43:17.321 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 144/156 2024-04-26T17:43:17.359 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 141/156 2024-04-26T17:43:17.366 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T17:43:17.391 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T17:43:17.396 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_ 146/156 2024-04-26T17:43:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T17:43:17.606 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 144/156 2024-04-26T17:43:17.648 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T17:43:17.678 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_ 146/156 2024-04-26T17:43:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 139/156 2024-04-26T17:43:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 140/156 2024-04-26T17:43:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 141/156 2024-04-26T17:43:17.843 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T17:43:17.870 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T17:43:18.019 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T17:43:18.060 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T17:43:18.075 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 144/156 2024-04-26T17:43:18.080 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T17:43:18.115 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T17:43:18.145 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_ 146/156 2024-04-26T17:43:18.251 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T17:43:18.292 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T17:43:18.312 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T17:43:18.717 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T17:43:18.759 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T17:43:18.780 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T17:43:18.937 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 148/156 2024-04-26T17:43:19.023 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T17:43:19.080 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T17:43:19.142 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 148/156 2024-04-26T17:43:19.163 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T17:43:19.201 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T17:43:19.201 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T17:43:19.230 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T17:43:19.270 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T17:43:19.286 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T17:43:19.319 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T17:43:19.319 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T17:43:19.354 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T17:43:19.392 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T17:43:19.392 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T17:43:19.406 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T17:43:19.475 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T17:43:19.476 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T17:43:19.500 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T17:43:19.501 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T17:43:19.588 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T17:43:19.658 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T17:43:19.698 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 148/156 2024-04-26T17:43:19.782 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T17:43:19.824 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T17:43:19.909 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T17:43:19.945 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T17:43:19.945 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T17:43:20.029 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T17:43:20.070 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T17:43:20.070 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T17:43:20.164 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T17:43:20.238 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T17:43:25.379 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T17:43:25.379 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 1/156 2024-04-26T17:43:25.380 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2/156 2024-04-26T17:43:25.380 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 3/156 2024-04-26T17:43:25.380 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 4/156 2024-04-26T17:43:25.380 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 5/156 2024-04-26T17:43:25.380 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 6/156 2024-04-26T17:43:25.380 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 7/156 2024-04-26T17:43:25.380 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T17:43:25.380 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T17:43:25.380 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 10/156 2024-04-26T17:43:25.380 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 11/156 2024-04-26T17:43:25.380 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 12/156 2024-04-26T17:43:25.380 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T17:43:25.380 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 14/156 2024-04-26T17:43:25.380 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_ 15/156 2024-04-26T17:43:25.381 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 16/156 2024-04-26T17:43:25.381 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 17/156 2024-04-26T17:43:25.381 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_6 18/156 2024-04-26T17:43:25.381 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 19/156 2024-04-26T17:43:25.381 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 20/156 2024-04-26T17:43:25.381 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el 21/156 2024-04-26T17:43:25.381 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9. 22/156 2024-04-26T17:43:25.381 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_6 23/156 2024-04-26T17:43:25.381 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 24/156 2024-04-26T17:43:25.381 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 25/156 2024-04-26T17:43:25.381 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 26/156 2024-04-26T17:43:25.381 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 27/156 2024-04-26T17:43:25.381 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 28/156 2024-04-26T17:43:25.382 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 29/156 2024-04-26T17:43:25.394 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9. 30/156 2024-04-26T17:43:25.394 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 31/156 2024-04-26T17:43:25.394 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 32/156 2024-04-26T17:43:25.394 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 33/156 2024-04-26T17:43:25.394 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el 34/156 2024-04-26T17:43:25.394 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 35/156 2024-04-26T17:43:25.394 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.e 36/156 2024-04-26T17:43:25.394 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 37/156 2024-04-26T17:43:25.395 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 38/156 2024-04-26T17:43:25.395 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 39/156 2024-04-26T17:43:25.395 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T17:43:25.395 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T17:43:25.395 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T17:43:25.395 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T17:43:25.395 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T17:43:25.395 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T17:43:25.395 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T17:43:25.395 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T17:43:25.395 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T17:43:25.395 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T17:43:25.395 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T17:43:25.396 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T17:43:25.396 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T17:43:25.396 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T17:43:25.396 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T17:43:25.396 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T17:43:25.396 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T17:43:25.396 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T17:43:25.396 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T17:43:25.396 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T17:43:25.396 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T17:43:25.396 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T17:43:25.396 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T17:43:25.396 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T17:43:25.397 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T17:43:25.397 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T17:43:25.397 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T17:43:25.397 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T17:43:25.397 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T17:43:25.397 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T17:43:25.397 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T17:43:25.397 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T17:43:25.397 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T17:43:25.397 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T17:43:25.397 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T17:43:25.397 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T17:43:25.398 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T17:43:25.398 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T17:43:25.398 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T17:43:25.398 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T17:43:25.398 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T17:43:25.398 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T17:43:25.398 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T17:43:25.398 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T17:43:25.398 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T17:43:25.398 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T17:43:25.398 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T17:43:25.398 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T17:43:25.398 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T17:43:25.399 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T17:43:25.399 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T17:43:25.399 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T17:43:25.399 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T17:43:25.399 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T17:43:25.399 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T17:43:25.399 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T17:43:25.399 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T17:43:25.399 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T17:43:25.399 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T17:43:25.399 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T17:43:25.399 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T17:43:25.399 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T17:43:25.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T17:43:25.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T17:43:25.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T17:43:25.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T17:43:25.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T17:43:25.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T17:43:25.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T17:43:25.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T17:43:25.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T17:43:25.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T17:43:25.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T17:43:25.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T17:43:25.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T17:43:25.400 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T17:43:25.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T17:43:25.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T17:43:25.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T17:43:25.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T17:43:25.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T17:43:25.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T17:43:25.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T17:43:25.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T17:43:25.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T17:43:25.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T17:43:25.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T17:43:25.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T17:43:25.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T17:43:25.401 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T17:43:25.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T17:43:25.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T17:43:25.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T17:43:25.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T17:43:25.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T17:43:25.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T17:43:25.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T17:43:25.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T17:43:25.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T17:43:25.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T17:43:25.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T17:43:25.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T17:43:25.402 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T17:43:25.403 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T17:43:25.403 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T17:43:25.403 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T17:43:25.403 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T17:43:25.403 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T17:43:25.403 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T17:43:25.403 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 149/156 2024-04-26T17:43:25.403 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T17:43:25.403 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 151/156 2024-04-26T17:43:25.403 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T17:43:25.403 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T17:43:25.403 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T17:43:25.403 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T17:43:25.479 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T17:43:25.480 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 1/156 2024-04-26T17:43:25.480 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2/156 2024-04-26T17:43:25.480 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 3/156 2024-04-26T17:43:25.480 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 4/156 2024-04-26T17:43:25.480 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 5/156 2024-04-26T17:43:25.480 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 6/156 2024-04-26T17:43:25.480 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 7/156 2024-04-26T17:43:25.480 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T17:43:25.480 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T17:43:25.480 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 10/156 2024-04-26T17:43:25.480 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 11/156 2024-04-26T17:43:25.480 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 12/156 2024-04-26T17:43:25.481 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T17:43:25.481 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 14/156 2024-04-26T17:43:25.481 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_ 15/156 2024-04-26T17:43:25.481 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 16/156 2024-04-26T17:43:25.481 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 17/156 2024-04-26T17:43:25.481 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_6 18/156 2024-04-26T17:43:25.481 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 19/156 2024-04-26T17:43:25.481 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 20/156 2024-04-26T17:43:25.481 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el 21/156 2024-04-26T17:43:25.481 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9. 22/156 2024-04-26T17:43:25.481 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_6 23/156 2024-04-26T17:43:25.481 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 24/156 2024-04-26T17:43:25.481 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 25/156 2024-04-26T17:43:25.481 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 26/156 2024-04-26T17:43:25.482 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 27/156 2024-04-26T17:43:25.482 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 28/156 2024-04-26T17:43:25.482 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 29/156 2024-04-26T17:43:25.482 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9. 30/156 2024-04-26T17:43:25.482 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 31/156 2024-04-26T17:43:25.482 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 32/156 2024-04-26T17:43:25.482 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 33/156 2024-04-26T17:43:25.482 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el 34/156 2024-04-26T17:43:25.482 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 35/156 2024-04-26T17:43:25.482 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.e 36/156 2024-04-26T17:43:25.482 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 37/156 2024-04-26T17:43:25.482 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 38/156 2024-04-26T17:43:25.482 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 39/156 2024-04-26T17:43:25.483 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T17:43:25.483 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T17:43:25.483 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T17:43:25.483 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T17:43:25.483 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T17:43:25.483 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T17:43:25.483 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T17:43:25.483 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T17:43:25.483 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T17:43:25.483 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T17:43:25.483 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T17:43:25.483 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T17:43:25.483 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T17:43:25.484 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T17:43:25.484 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T17:43:25.484 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T17:43:25.484 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T17:43:25.484 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T17:43:25.484 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T17:43:25.484 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T17:43:25.484 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T17:43:25.484 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T17:43:25.484 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T17:43:25.484 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T17:43:25.484 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T17:43:25.484 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T17:43:25.485 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T17:43:25.485 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T17:43:25.485 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T17:43:25.485 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T17:43:25.485 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T17:43:25.485 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T17:43:25.485 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T17:43:25.485 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T17:43:25.485 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T17:43:25.485 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T17:43:25.485 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T17:43:25.485 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T17:43:25.485 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T17:43:25.486 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T17:43:25.486 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T17:43:25.486 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T17:43:25.486 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T17:43:25.486 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T17:43:25.486 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T17:43:25.486 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T17:43:25.486 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T17:43:25.486 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T17:43:25.486 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T17:43:25.486 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T17:43:25.486 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T17:43:25.486 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T17:43:25.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T17:43:25.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T17:43:25.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T17:43:25.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T17:43:25.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T17:43:25.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T17:43:25.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T17:43:25.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T17:43:25.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T17:43:25.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T17:43:25.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T17:43:25.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T17:43:25.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T17:43:25.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T17:43:25.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T17:43:25.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T17:43:25.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T17:43:25.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T17:43:25.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T17:43:25.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T17:43:25.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T17:43:25.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T17:43:25.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T17:43:25.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T17:43:25.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T17:43:25.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T17:43:25.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T17:43:25.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T17:43:25.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T17:43:25.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T17:43:25.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T17:43:25.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T17:43:25.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T17:43:25.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T17:43:25.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T17:43:25.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T17:43:25.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T17:43:25.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T17:43:25.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T17:43:25.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T17:43:25.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T17:43:25.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T17:43:25.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T17:43:25.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T17:43:25.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T17:43:25.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T17:43:25.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T17:43:25.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T17:43:25.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T17:43:25.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T17:43:25.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T17:43:25.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T17:43:25.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T17:43:25.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T17:43:25.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T17:43:25.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T17:43:25.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T17:43:25.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 149/156 2024-04-26T17:43:25.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T17:43:25.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 151/156 2024-04-26T17:43:25.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T17:43:25.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T17:43:25.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T17:43:25.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T17:43:25.671 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T17:43:25.671 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 1/156 2024-04-26T17:43:25.671 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2/156 2024-04-26T17:43:25.671 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 3/156 2024-04-26T17:43:25.671 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 4/156 2024-04-26T17:43:25.671 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3267.gbc94a 5/156 2024-04-26T17:43:25.671 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 6/156 2024-04-26T17:43:25.672 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 7/156 2024-04-26T17:43:25.672 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 8/156 2024-04-26T17:43:25.672 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 9/156 2024-04-26T17:43:25.672 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 10/156 2024-04-26T17:43:25.672 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 11/156 2024-04-26T17:43:25.672 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 12/156 2024-04-26T17:43:25.672 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 13/156 2024-04-26T17:43:25.672 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 14/156 2024-04-26T17:43:25.672 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_ 15/156 2024-04-26T17:43:25.672 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 16/156 2024-04-26T17:43:25.672 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 17/156 2024-04-26T17:43:25.673 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_6 18/156 2024-04-26T17:43:25.673 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86 19/156 2024-04-26T17:43:25.673 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 20/156 2024-04-26T17:43:25.673 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el 21/156 2024-04-26T17:43:25.673 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9. 22/156 2024-04-26T17:43:25.673 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_6 23/156 2024-04-26T17:43:25.673 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 24/156 2024-04-26T17:43:25.673 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 25/156 2024-04-26T17:43:25.673 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 26/156 2024-04-26T17:43:25.673 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 27/156 2024-04-26T17:43:25.674 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 28/156 2024-04-26T17:43:25.674 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 29/156 2024-04-26T17:43:25.674 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9. 30/156 2024-04-26T17:43:25.687 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noa 31/156 2024-04-26T17:43:25.688 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.n 32/156 2024-04-26T17:43:25.688 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc9 33/156 2024-04-26T17:43:25.688 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el 34/156 2024-04-26T17:43:25.688 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 35/156 2024-04-26T17:43:25.688 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.e 36/156 2024-04-26T17:43:25.688 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 37/156 2024-04-26T17:43:25.688 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 38/156 2024-04-26T17:43:25.688 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 39/156 2024-04-26T17:43:25.688 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T17:43:25.688 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T17:43:25.688 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T17:43:25.689 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T17:43:25.689 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T17:43:25.689 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T17:43:25.689 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T17:43:25.689 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T17:43:25.689 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T17:43:25.689 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T17:43:25.689 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T17:43:25.689 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T17:43:25.689 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T17:43:25.690 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T17:43:25.690 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T17:43:25.690 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T17:43:25.690 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T17:43:25.690 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T17:43:25.690 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T17:43:25.690 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T17:43:25.690 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T17:43:25.690 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T17:43:25.690 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T17:43:25.691 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T17:43:25.691 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T17:43:25.691 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T17:43:25.691 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T17:43:25.691 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T17:43:25.691 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T17:43:25.691 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T17:43:25.691 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T17:43:25.691 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T17:43:25.691 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T17:43:25.691 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T17:43:25.692 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T17:43:25.692 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T17:43:25.692 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T17:43:25.692 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T17:43:25.692 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T17:43:25.692 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T17:43:25.692 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T17:43:25.692 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T17:43:25.692 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T17:43:25.692 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T17:43:25.692 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T17:43:25.693 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T17:43:25.693 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T17:43:25.693 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T17:43:25.693 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T17:43:25.693 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T17:43:25.693 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T17:43:25.693 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T17:43:25.693 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T17:43:25.693 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T17:43:25.693 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T17:43:25.694 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T17:43:25.694 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T17:43:25.694 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T17:43:25.694 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T17:43:25.694 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T17:43:25.694 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T17:43:25.694 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T17:43:25.694 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T17:43:25.694 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T17:43:25.694 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T17:43:25.694 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T17:43:25.694 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T17:43:25.695 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T17:43:25.695 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T17:43:25.695 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T17:43:25.695 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T17:43:25.695 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T17:43:25.695 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T17:43:25.695 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T17:43:25.695 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T17:43:25.695 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T17:43:25.695 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T17:43:25.696 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T17:43:25.696 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T17:43:25.696 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T17:43:25.696 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T17:43:25.696 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T17:43:25.696 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T17:43:25.696 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T17:43:25.696 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T17:43:25.696 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T17:43:25.696 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T17:43:25.697 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T17:43:25.697 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T17:43:25.697 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T17:43:25.697 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T17:43:25.697 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T17:43:25.697 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T17:43:25.697 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T17:43:25.697 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T17:43:25.697 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T17:43:25.697 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T17:43:25.697 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T17:43:25.698 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T17:43:25.698 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T17:43:25.698 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T17:43:25.698 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T17:43:25.698 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T17:43:25.698 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T17:43:25.698 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T17:43:25.698 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T17:43:25.698 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T17:43:25.699 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T17:43:25.699 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T17:43:25.699 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 149/156 2024-04-26T17:43:25.699 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T17:43:25.699 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 151/156 2024-04-26T17:43:25.699 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T17:43:25.699 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T17:43:25.699 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T17:43:25.699 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T17:43:27.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T17:43:27.464 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:43:27.465 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-04-26T17:43:27.465 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.465 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.465 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T17:43:27.465 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T17:43:27.465 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-26T17:43:27.465 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T17:43:27.465 INFO:teuthology.orchestra.run.smithi186.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T17:43:27.465 INFO:teuthology.orchestra.run.smithi186.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T17:43:27.465 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T17:43:27.465 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.465 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.466 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.466 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.466 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.466 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.466 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.466 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.466 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.466 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.466 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.466 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.466 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.466 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.467 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.467 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.467 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.467 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.467 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.467 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.467 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.467 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.467 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.467 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.467 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T17:43:27.467 INFO:teuthology.orchestra.run.smithi186.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T17:43:27.468 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T17:43:27.468 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T17:43:27.468 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T17:43:27.468 INFO:teuthology.orchestra.run.smithi186.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T17:43:27.468 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T17:43:27.468 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T17:43:27.468 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T17:43:27.468 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T17:43:27.468 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T17:43:27.468 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.468 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.468 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.469 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T17:43:27.469 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T17:43:27.469 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T17:43:27.469 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T17:43:27.469 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T17:43:27.469 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.469 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.469 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T17:43:27.469 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.469 INFO:teuthology.orchestra.run.smithi186.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T17:43:27.469 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T17:43:27.469 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T17:43:27.470 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T17:43:27.470 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T17:43:27.470 INFO:teuthology.orchestra.run.smithi186.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T17:43:27.470 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T17:43:27.470 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T17:43:27.470 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T17:43:27.470 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T17:43:27.470 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T17:43:27.470 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T17:43:27.470 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T17:43:27.470 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T17:43:27.470 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T17:43:27.471 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T17:43:27.471 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T17:43:27.471 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T17:43:27.471 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T17:43:27.471 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T17:43:27.471 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T17:43:27.471 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T17:43:27.471 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T17:43:27.471 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.471 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.472 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.472 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T17:43:27.472 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T17:43:27.472 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T17:43:27.472 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T17:43:27.472 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T17:43:27.473 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T17:43:27.473 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T17:43:27.473 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T17:43:27.473 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T17:43:27.473 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T17:43:27.473 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T17:43:27.474 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T17:43:27.474 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T17:43:27.474 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T17:43:27.474 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T17:43:27.474 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T17:43:27.474 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T17:43:27.474 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T17:43:27.475 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T17:43:27.475 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T17:43:27.475 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T17:43:27.475 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T17:43:27.475 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T17:43:27.476 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T17:43:27.476 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T17:43:27.476 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T17:43:27.476 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T17:43:27.476 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T17:43:27.476 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T17:43:27.476 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T17:43:27.477 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T17:43:27.477 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T17:43:27.477 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T17:43:27.477 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T17:43:27.477 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T17:43:27.477 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T17:43:27.478 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T17:43:27.478 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T17:43:27.478 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T17:43:27.478 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T17:43:27.478 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.478 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.479 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T17:43:27.479 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T17:43:27.479 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T17:43:27.479 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.479 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T17:43:27.479 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T17:43:27.480 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T17:43:27.480 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T17:43:27.480 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T17:43:27.480 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T17:43:27.480 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T17:43:27.480 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T17:43:27.481 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T17:43:27.481 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T17:43:27.481 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T17:43:27.481 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T17:43:27.481 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T17:43:27.481 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T17:43:27.482 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T17:43:27.482 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.482 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.482 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.482 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T17:43:27.482 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T17:43:27.483 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T17:43:27.483 INFO:teuthology.orchestra.run.smithi186.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T17:43:27.483 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T17:43:27.483 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T17:43:27.484 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T17:43:27.484 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:43:27.484 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-26T17:43:27.574 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T17:43:27.574 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:43:27.574 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2024-04-26T17:43:27.574 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.574 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.574 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T17:43:27.574 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T17:43:27.574 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-26T17:43:27.575 INFO:teuthology.orchestra.run.smithi097.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T17:43:27.575 INFO:teuthology.orchestra.run.smithi097.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T17:43:27.575 INFO:teuthology.orchestra.run.smithi097.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T17:43:27.575 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T17:43:27.575 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.575 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.575 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.575 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.575 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.575 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.575 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.575 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.575 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.576 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.576 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.576 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.576 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.576 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.576 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.576 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.576 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.576 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.576 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.576 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.576 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.576 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.577 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.577 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.577 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T17:43:27.577 INFO:teuthology.orchestra.run.smithi097.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T17:43:27.577 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T17:43:27.577 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T17:43:27.577 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T17:43:27.577 INFO:teuthology.orchestra.run.smithi097.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T17:43:27.577 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T17:43:27.577 INFO:teuthology.orchestra.run.smithi097.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T17:43:27.577 INFO:teuthology.orchestra.run.smithi097.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T17:43:27.577 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T17:43:27.577 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T17:43:27.577 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.578 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.578 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.578 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T17:43:27.578 INFO:teuthology.orchestra.run.smithi097.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T17:43:27.578 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T17:43:27.578 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T17:43:27.578 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T17:43:27.578 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.578 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.578 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T17:43:27.578 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.578 INFO:teuthology.orchestra.run.smithi097.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T17:43:27.579 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T17:43:27.579 INFO:teuthology.orchestra.run.smithi097.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T17:43:27.579 INFO:teuthology.orchestra.run.smithi097.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T17:43:27.579 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T17:43:27.579 INFO:teuthology.orchestra.run.smithi097.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T17:43:27.579 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T17:43:27.579 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T17:43:27.579 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T17:43:27.579 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T17:43:27.579 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T17:43:27.579 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T17:43:27.579 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T17:43:27.579 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T17:43:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T17:43:27.581 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T17:43:27.581 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T17:43:27.581 INFO:teuthology.orchestra.run.smithi097.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T17:43:27.581 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T17:43:27.581 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T17:43:27.581 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T17:43:27.581 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T17:43:27.581 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T17:43:27.581 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T17:43:27.581 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T17:43:27.581 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T17:43:27.581 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T17:43:27.581 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T17:43:27.582 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T17:43:27.582 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T17:43:27.582 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T17:43:27.582 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T17:43:27.582 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T17:43:27.582 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T17:43:27.582 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T17:43:27.582 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T17:43:27.582 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T17:43:27.582 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T17:43:27.583 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T17:43:27.583 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T17:43:27.583 INFO:teuthology.orchestra.run.smithi097.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T17:43:27.583 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T17:43:27.583 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T17:43:27.583 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T17:43:27.583 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T17:43:27.583 INFO:teuthology.orchestra.run.smithi097.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T17:43:27.583 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T17:43:27.583 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T17:43:27.583 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T17:43:27.583 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T17:43:27.583 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T17:43:27.584 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T17:43:27.584 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.584 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.584 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T17:43:27.584 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T17:43:27.584 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T17:43:27.584 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.584 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T17:43:27.584 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T17:43:27.584 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T17:43:27.584 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T17:43:27.584 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T17:43:27.584 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T17:43:27.584 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T17:43:27.585 INFO:teuthology.orchestra.run.smithi097.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T17:43:27.585 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T17:43:27.585 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T17:43:27.585 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T17:43:27.585 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T17:43:27.585 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T17:43:27.585 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T17:43:27.585 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T17:43:27.585 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.585 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.585 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.585 INFO:teuthology.orchestra.run.smithi097.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T17:43:27.585 INFO:teuthology.orchestra.run.smithi097.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T17:43:27.586 INFO:teuthology.orchestra.run.smithi097.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T17:43:27.586 INFO:teuthology.orchestra.run.smithi097.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T17:43:27.586 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T17:43:27.586 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T17:43:27.586 INFO:teuthology.orchestra.run.smithi097.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T17:43:27.586 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-26T17:43:27.586 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-26T17:43:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T17:43:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:43:27.751 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2024-04-26T17:43:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T17:43:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T17:43:27.752 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-26T17:43:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T17:43:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T17:43:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T17:43:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T17:43:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-shell-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top-2:19.0.0-3267.gbc94a2a9.el9.noarch 2024-04-26T17:43:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T17:43:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T17:43:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T17:43:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T17:43:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T17:43:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T17:43:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T17:43:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T17:43:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T17:43:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T17:43:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T17:43:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T17:43:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T17:43:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T17:43:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T17:43:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T17:43:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T17:43:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T17:43:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T17:43:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T17:43:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T17:43:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T17:43:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T17:43:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T17:43:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T17:43:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T17:43:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T17:43:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T17:43:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T17:43:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T17:43:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T17:43:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T17:43:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T17:43:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T17:43:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T17:43:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T17:43:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T17:43:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T17:43:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T17:43:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T17:43:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T17:43:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T17:43:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T17:43:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T17:43:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T17:43:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T17:43:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T17:43:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T17:43:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T17:43:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T17:43:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T17:43:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T17:43:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T17:43:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T17:43:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T17:43:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T17:43:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T17:43:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T17:43:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T17:43:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T17:43:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T17:43:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T17:43:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T17:43:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T17:43:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T17:43:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T17:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T17:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T17:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T17:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T17:43:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T17:43:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T17:43:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T17:43:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T17:43:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T17:43:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T17:43:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T17:43:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T17:43:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T17:43:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T17:43:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T17:43:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T17:43:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T17:43:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T17:43:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T17:43:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T17:43:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T17:43:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T17:43:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T17:43:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T17:43:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T17:43:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T17:43:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T17:43:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T17:43:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T17:43:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T17:43:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T17:43:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T17:43:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:19.0.0-3267.gbc94a2a9.el9.x86_64 2024-04-26T17:43:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T17:43:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T17:43:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T17:43:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T17:43:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T17:43:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T17:43:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T17:43:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:43:27.765 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-26T17:43:27.870 DEBUG:teuthology.parallel:result is None 2024-04-26T17:43:27.994 DEBUG:teuthology.parallel:result is None 2024-04-26T17:43:28.117 DEBUG:teuthology.parallel:result is None 2024-04-26T17:43:28.117 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T17:43:28.354 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T17:43:28.378 INFO:teuthology.orchestra.run.smithi031.stdout:19.0.0-3267.gbc94a2a9.el9 2024-04-26T17:43:28.379 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3267.gbc94a2a9.el9 2024-04-26T17:43:28.379 INFO:teuthology.task.install:The correct ceph version 19.0.0-3267.gbc94a2a9 is installed. 2024-04-26T17:43:28.381 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T17:43:28.520 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T17:43:28.541 INFO:teuthology.orchestra.run.smithi097.stdout:19.0.0-3267.gbc94a2a9.el9 2024-04-26T17:43:28.542 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3267.gbc94a2a9.el9 2024-04-26T17:43:28.542 INFO:teuthology.task.install:The correct ceph version 19.0.0-3267.gbc94a2a9 is installed. 2024-04-26T17:43:28.544 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T17:43:28.781 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T17:43:28.802 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-3267.gbc94a2a9.el9 2024-04-26T17:43:28.803 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3267.gbc94a2a9.el9 2024-04-26T17:43:28.803 INFO:teuthology.task.install:The correct ceph version 19.0.0-3267.gbc94a2a9 is installed. 2024-04-26T17:43:28.804 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T17:43:28.805 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:43:28.805 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T17:43:28.831 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-26T17:43:28.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T17:43:28.858 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T17:43:28.858 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T17:43:28.884 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T17:43:28.884 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:43:28.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T17:43:28.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T17:43:28.973 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-26T17:43:28.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T17:43:28.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T17:43:29.061 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T17:43:29.061 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T17:43:29.085 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T17:43:29.149 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T17:43:29.150 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:43:29.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T17:43:29.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T17:43:29.242 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-26T17:43:29.243 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T17:43:29.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T17:43:29.331 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T17:43:29.331 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T17:43:29.355 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T17:43:29.420 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T17:43:29.420 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:43:29.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T17:43:29.446 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T17:43:29.510 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-26T17:43:29.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T17:43:29.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T17:43:29.598 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T17:43:29.598 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T17:43:29.621 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T17:43:29.685 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-26T17:43:29.809 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-26T17:43:29.810 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T17:43:29.810 INFO:tasks.cephadm:Cluster fsid is 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:43:29.810 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-26T17:43:29.810 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.31', 'mon.b': '172.21.15.97', 'mon.c': '172.21.15.186'} 2024-04-26T17:43:29.811 INFO:tasks.cephadm:First mon is mon.a on smithi031 2024-04-26T17:43:29.811 INFO:tasks.cephadm:First mgr is x 2024-04-26T17:43:29.811 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-26T17:43:29.811 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname $(hostname -s) 2024-04-26T17:43:29.836 DEBUG:teuthology.orchestra.run.smithi097:> sudo hostname $(hostname -s) 2024-04-26T17:43:29.860 DEBUG:teuthology.orchestra.run.smithi186:> sudo hostname $(hostname -s) 2024-04-26T17:43:29.884 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-26T17:43:29.885 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T17:43:30.102 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78913/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3267-gbc94a2a9', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3267.gbc94a2a9'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-26 08:22:46.917786', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer-fixes', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-26T17:43:30.317 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-lusov-quiescer-fixes, sha1 bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T17:43:30.319 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/x86_64/flavors/default/cephadm 2024-04-26T17:43:30.319 DEBUG:teuthology.orchestra.run.smithi031:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T17:43:30.762 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r--. 1 ubuntu ubuntu 781590 Apr 26 17:43 /home/ubuntu/cephtest/cephadm 2024-04-26T17:43:30.762 DEBUG:teuthology.orchestra.run.smithi097:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T17:43:31.292 INFO:teuthology.orchestra.run.smithi097.stdout:-rw-r--r--. 1 ubuntu ubuntu 781590 Apr 26 17:43 /home/ubuntu/cephtest/cephadm 2024-04-26T17:43:31.293 DEBUG:teuthology.orchestra.run.smithi186:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer-fixes/bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T17:43:31.719 INFO:teuthology.orchestra.run.smithi186.stdout:-rw-r--r--. 1 ubuntu ubuntu 781590 Apr 26 17:43 /home/ubuntu/cephtest/cephadm 2024-04-26T17:43:31.719 DEBUG:teuthology.orchestra.run.smithi031:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T17:43:31.739 DEBUG:teuthology.orchestra.run.smithi097:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T17:43:31.756 DEBUG:teuthology.orchestra.run.smithi186:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T17:43:31.792 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 on all hosts... 2024-04-26T17:43:31.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 pull 2024-04-26T17:43:31.795 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 pull 2024-04-26T17:43:31.799 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 pull 2024-04-26T17:43:33.332 INFO:teuthology.orchestra.run.smithi186.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1... 2024-04-26T17:43:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1... 2024-04-26T17:43:33.458 INFO:teuthology.orchestra.run.smithi097.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1... 2024-04-26T17:44:36.031 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T17:44:36.031 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev)", 2024-04-26T17:44:36.031 INFO:teuthology.orchestra.run.smithi031.stdout: "image_id": "3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555", 2024-04-26T17:44:36.032 INFO:teuthology.orchestra.run.smithi031.stdout: "repo_digests": [ 2024-04-26T17:44:36.032 INFO:teuthology.orchestra.run.smithi031.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc" 2024-04-26T17:44:36.032 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-26T17:44:36.032 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T17:44:37.246 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-26T17:44:37.246 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_version": "ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev)", 2024-04-26T17:44:37.246 INFO:teuthology.orchestra.run.smithi186.stdout: "image_id": "3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555", 2024-04-26T17:44:37.246 INFO:teuthology.orchestra.run.smithi186.stdout: "repo_digests": [ 2024-04-26T17:44:37.247 INFO:teuthology.orchestra.run.smithi186.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc" 2024-04-26T17:44:37.247 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-26T17:44:37.247 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-26T17:44:39.547 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-26T17:44:39.547 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_version": "ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev)", 2024-04-26T17:44:39.548 INFO:teuthology.orchestra.run.smithi097.stdout: "image_id": "3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555", 2024-04-26T17:44:39.548 INFO:teuthology.orchestra.run.smithi097.stdout: "repo_digests": [ 2024-04-26T17:44:39.548 INFO:teuthology.orchestra.run.smithi097.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc" 2024-04-26T17:44:39.548 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-26T17:44:39.548 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-26T17:44:39.572 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph 2024-04-26T17:44:39.605 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph 2024-04-26T17:44:39.631 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph 2024-04-26T17:44:39.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /etc/ceph 2024-04-26T17:44:39.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /etc/ceph 2024-04-26T17:44:39.715 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /etc/ceph 2024-04-26T17:44:39.741 INFO:tasks.cephadm:Writing seed config... 2024-04-26T17:44:39.743 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-26T17:44:39.743 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-26T17:44:39.743 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-26T17:44:39.743 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-26T17:44:39.743 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-26T17:44:39.743 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-26T17:44:39.743 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-26T17:44:39.743 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-26T17:44:39.743 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-26T17:44:39.743 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-26T17:44:39.743 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-26T17:44:39.744 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-26T17:44:39.745 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-26T17:44:39.746 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-26T17:44:39.746 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-26T17:44:39.746 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-26T17:44:39.746 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-26T17:44:39.746 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-26T17:44:39.747 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:44:39.747 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-26T17:44:39.765 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7e53e050-03f4-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-26T17:44:39.766 DEBUG:teuthology.orchestra.run.smithi031:mon.a> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.a.service 2024-04-26T17:44:39.808 DEBUG:teuthology.orchestra.run.smithi031:mgr.x> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mgr.x.service 2024-04-26T17:44:39.851 INFO:tasks.cephadm:Bootstrapping... 2024-04-26T17:44:39.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 -v bootstrap --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.31 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-26T17:44:40.054 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-26T17:44:40.054 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1', '-v', 'bootstrap', '--fsid', '7e53e050-03f4-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.31', '--skip-admin-label'] 2024-04-26T17:44:40.054 INFO:teuthology.orchestra.run.smithi031.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-26T17:44:40.054 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying podman|docker is present... 2024-04-26T17:44:40.081 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T17:44:40.081 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying lvm2 is present... 2024-04-26T17:44:40.082 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying time synchronization is in place... 2024-04-26T17:44:40.089 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T17:44:40.089 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T17:44:40.095 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T17:44:40.095 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-26T17:44:40.102 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout enabled 2024-04-26T17:44:40.108 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout active 2024-04-26T17:44:40.108 INFO:teuthology.orchestra.run.smithi031.stdout:Unit chronyd.service is enabled and running 2024-04-26T17:44:40.108 INFO:teuthology.orchestra.run.smithi031.stdout:Repeating the final host check... 2024-04-26T17:44:40.133 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T17:44:40.133 INFO:teuthology.orchestra.run.smithi031.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-26T17:44:40.133 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl is present 2024-04-26T17:44:40.133 INFO:teuthology.orchestra.run.smithi031.stdout:lvcreate is present 2024-04-26T17:44:40.140 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T17:44:40.140 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T17:44:40.146 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T17:44:40.146 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-26T17:44:40.152 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout enabled 2024-04-26T17:44:40.159 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout active 2024-04-26T17:44:40.159 INFO:teuthology.orchestra.run.smithi031.stdout:Unit chronyd.service is enabled and running 2024-04-26T17:44:40.159 INFO:teuthology.orchestra.run.smithi031.stdout:Host looks OK 2024-04-26T17:44:40.159 INFO:teuthology.orchestra.run.smithi031.stdout:Cluster fsid: 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:44:40.159 INFO:teuthology.orchestra.run.smithi031.stdout:Acquiring lock 139669160262816 on /run/cephadm/7e53e050-03f4-11ef-bc93-c7b262605968.lock 2024-04-26T17:44:40.159 INFO:teuthology.orchestra.run.smithi031.stdout:Lock 139669160262816 acquired on /run/cephadm/7e53e050-03f4-11ef-bc93-c7b262605968.lock 2024-04-26T17:44:40.160 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying IP 172.21.15.31 port 3300 ... 2024-04-26T17:44:40.160 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying IP 172.21.15.31 port 6789 ... 2024-04-26T17:44:40.160 INFO:teuthology.orchestra.run.smithi031.stdout:Base mon IP(s) is [172.21.15.31:3300, 172.21.15.31:6789], mon addrv is [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2024-04-26T17:44:40.163 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.31 metric 100 2024-04-26T17:44:40.163 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.31 metric 100 2024-04-26T17:44:40.166 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-26T17:44:40.166 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-26T17:44:40.166 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-26T17:44:40.168 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-26T17:44:40.168 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-26T17:44:40.169 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T17:44:40.169 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 3: ens1f0: mtu 1500 state UP qlen 1000 2024-04-26T17:44:40.169 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:caba/64 scope link noprefixroute 2024-04-26T17:44:40.169 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T17:44:40.169 INFO:teuthology.orchestra.run.smithi031.stdout:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2024-04-26T17:44:40.169 INFO:teuthology.orchestra.run.smithi031.stdout:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2024-04-26T17:44:40.170 INFO:teuthology.orchestra.run.smithi031.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-26T17:44:40.170 INFO:teuthology.orchestra.run.smithi031.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-26T17:44:40.171 INFO:teuthology.orchestra.run.smithi031.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1... 2024-04-26T17:44:40.648 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 2024-04-26T17:44:40.648 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1... 2024-04-26T17:44:40.648 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Getting image source signatures 2024-04-26T17:44:40.648 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-26T17:44:40.648 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying blob sha256:449452a284afe5840255d087488a224329a4679c5f5d200bb867a639c75ddfd2 2024-04-26T17:44:40.648 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying config sha256:3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 2024-04-26T17:44:40.648 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-26T17:44:41.534 INFO:teuthology.orchestra.run.smithi031.stdout:ceph: stdout ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T17:44:41.534 INFO:teuthology.orchestra.run.smithi031.stdout:Ceph version: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T17:44:41.535 INFO:teuthology.orchestra.run.smithi031.stdout:Extracting ceph user uid/gid from container image... 2024-04-26T17:44:42.544 INFO:teuthology.orchestra.run.smithi031.stdout:stat: stdout 167 167 2024-04-26T17:44:42.544 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial keys... 2024-04-26T17:44:43.501 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQAK6CtmC/aINBAA2v34f4obJ5UPI39c3UrHvw== 2024-04-26T17:44:44.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQAL6Ctm0onUMRAAlZ0qKxXWY4/Cw/dWU2yFsQ== 2024-04-26T17:44:45.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQAM6CtmBM/TKxAAdsl71g50sVYXFBRo5cxWoQ== 2024-04-26T17:44:45.273 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial monmap... 2024-04-26T17:44:46.279 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T17:44:46.279 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-26T17:44:46.279 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:44:46.279 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T17:44:46.280 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool for a [v2:172.21.15.31:3300,v1:172.21.15.31:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T17:44:46.280 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = quincy 2024-04-26T17:44:46.280 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: set fsid to 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:44:46.280 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T17:44:46.280 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:44:46.280 INFO:teuthology.orchestra.run.smithi031.stdout:Creating mon... 2024-04-26T17:44:47.153 INFO:teuthology.orchestra.run.smithi031.stdout:create mon.a on 2024-04-26T17:44:47.358 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-26T17:44:47.506 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-26T17:44:47.677 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7e53e050-03f4-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-7e53e050-03f4-11ef-bc93-c7b262605968.target. 2024-04-26T17:44:47.678 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-7e53e050-03f4-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-7e53e050-03f4-11ef-bc93-c7b262605968.target. 2024-04-26T17:44:47.948 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.a 2024-04-26T17:44:47.948 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-26T17:44:48.117 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7e53e050-03f4-11ef-bc93-c7b262605968.target.wants/ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-7e53e050-03f4-11ef-bc93-c7b262605968@.service. 2024-04-26T17:44:48.670 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T17:44:48.670 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-04-26T17:44:48.677 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T17:44:48.678 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-26T17:44:48.678 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-04-26T17:44:48.678 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T17:44:48.678 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mon to start... 2024-04-26T17:44:48.678 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mon... 2024-04-26T17:44:49.708 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout cluster: 2024-04-26T17:44:49.708 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout id: 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:44:49.708 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-26T17:44:49.708 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-26T17:44:49.708 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout services: 2024-04-26T17:44:49.708 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.543864s) 2024-04-26T17:44:49.708 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-26T17:44:49.708 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-26T17:44:49.708 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-26T17:44:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout data: 2024-04-26T17:44:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-26T17:44:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-26T17:44:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-26T17:44:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout pgs: 2024-04-26T17:44:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-26T17:44:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.078+0000 7f6416e24700 1 Processor -- start 2024-04-26T17:44:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.078+0000 7f6416e24700 1 -- start start 2024-04-26T17:44:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.078+0000 7f6416e24700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64101097e0 0x7f641010bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:44:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.078+0000 7f6416e24700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f641007ceb0 con 0x7f64101097e0 2024-04-26T17:44:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.079+0000 7f6414bc0700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64101097e0 0x7f641010bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:44:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.079+0000 7f6414bc0700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64101097e0 0x7f641010bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43186/0 (socket says 172.21.15.31:43186) 2024-04-26T17:44:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.079+0000 7f6414bc0700 1 -- 172.21.15.31:0/2130248665 learned_addr learned my addr 172.21.15.31:0/2130248665 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:44:49.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.079+0000 7f6414bc0700 1 -- 172.21.15.31:0/2130248665 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641010c110 con 0x7f64101097e0 2024-04-26T17:44:49.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.079+0000 7f6414bc0700 1 --2- 172.21.15.31:0/2130248665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64101097e0 0x7f641010bbd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f64040098a0 tx=0x7f6404009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3bb66a739b6dd01 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:44:49.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.079+0000 7f640f7fe700 1 -- 172.21.15.31:0/2130248665 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6404004160 con 0x7f64101097e0 2024-04-26T17:44:49.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.079+0000 7f640f7fe700 1 -- 172.21.15.31:0/2130248665 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6404004300 con 0x7f64101097e0 2024-04-26T17:44:49.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.079+0000 7f640f7fe700 1 -- 172.21.15.31:0/2130248665 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64040036a0 con 0x7f64101097e0 2024-04-26T17:44:49.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.080+0000 7f6416e24700 1 -- 172.21.15.31:0/2130248665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64101097e0 msgr2=0x7f641010bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:44:49.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.080+0000 7f6416e24700 1 --2- 172.21.15.31:0/2130248665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64101097e0 0x7f641010bbd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f64040098a0 tx=0x7f6404009bb0 comp rx=0 tx=0).stop 2024-04-26T17:44:49.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.080+0000 7f6416e24700 1 -- 172.21.15.31:0/2130248665 shutdown_connections 2024-04-26T17:44:49.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.080+0000 7f6416e24700 1 --2- 172.21.15.31:0/2130248665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64101097e0 0x7f641010bbd0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:44:49.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.080+0000 7f6416e24700 1 -- 172.21.15.31:0/2130248665 >> 172.21.15.31:0/2130248665 conn(0x7f6410103210 msgr2=0x7f6410105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:44:49.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.080+0000 7f6416e24700 1 -- 172.21.15.31:0/2130248665 shutdown_connections 2024-04-26T17:44:49.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.080+0000 7f6416e24700 1 -- 172.21.15.31:0/2130248665 wait complete. 2024-04-26T17:44:49.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.080+0000 7f6416e24700 1 Processor -- start 2024-04-26T17:44:49.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.080+0000 7f6416e24700 1 -- start start 2024-04-26T17:44:49.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.081+0000 7f6416e24700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64101097e0 0x7f641019c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:44:49.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.081+0000 7f6416e24700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f641010c580 con 0x7f64101097e0 2024-04-26T17:44:49.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.081+0000 7f6414bc0700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64101097e0 0x7f641019c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:44:49.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.081+0000 7f6414bc0700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64101097e0 0x7f641019c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43192/0 (socket says 172.21.15.31:43192) 2024-04-26T17:44:49.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.081+0000 7f6414bc0700 1 -- 172.21.15.31:0/1301479118 learned_addr learned my addr 172.21.15.31:0/1301479118 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:44:49.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.081+0000 7f6414bc0700 1 -- 172.21.15.31:0/1301479118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641019c940 con 0x7f64101097e0 2024-04-26T17:44:49.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.081+0000 7f6414bc0700 1 --2- 172.21.15.31:0/1301479118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64101097e0 0x7f641019c400 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6404000c00 tx=0x7f6404004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:44:49.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.081+0000 7f640dffb700 1 -- 172.21.15.31:0/1301479118 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6404003db0 con 0x7f64101097e0 2024-04-26T17:44:49.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.081+0000 7f640dffb700 1 -- 172.21.15.31:0/1301479118 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6404011630 con 0x7f64101097e0 2024-04-26T17:44:49.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.081+0000 7f6416e24700 1 -- 172.21.15.31:0/1301479118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f641019cc30 con 0x7f64101097e0 2024-04-26T17:44:49.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.081+0000 7f640dffb700 1 -- 172.21.15.31:0/1301479118 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6404011950 con 0x7f64101097e0 2024-04-26T17:44:49.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.082+0000 7f6416e24700 1 -- 172.21.15.31:0/1301479118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f641019f8c0 con 0x7f64101097e0 2024-04-26T17:44:49.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.082+0000 7f640dffb700 1 -- 172.21.15.31:0/1301479118 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6404011af0 con 0x7f64101097e0 2024-04-26T17:44:49.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.082+0000 7f640dffb700 1 -- 172.21.15.31:0/1301479118 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f640401c450 con 0x7f64101097e0 2024-04-26T17:44:49.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.083+0000 7f6416e24700 1 -- 172.21.15.31:0/1301479118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6410064a80 con 0x7f64101097e0 2024-04-26T17:44:49.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.085+0000 7f640dffb700 1 -- 172.21.15.31:0/1301479118 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f6404004260 con 0x7f64101097e0 2024-04-26T17:44:49.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.159+0000 7f6416e24700 1 -- 172.21.15.31:0/1301479118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f641002e6c0 con 0x7f64101097e0 2024-04-26T17:44:49.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.160+0000 7f640dffb700 1 -- 172.21.15.31:0/1301479118 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f640401b450 con 0x7f64101097e0 2024-04-26T17:44:49.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.161+0000 7f6416e24700 1 -- 172.21.15.31:0/1301479118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64101097e0 msgr2=0x7f641019c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:44:49.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.161+0000 7f6416e24700 1 --2- 172.21.15.31:0/1301479118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64101097e0 0x7f641019c400 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6404000c00 tx=0x7f6404004750 comp rx=0 tx=0).stop 2024-04-26T17:44:49.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.161+0000 7f6416e24700 1 -- 172.21.15.31:0/1301479118 shutdown_connections 2024-04-26T17:44:49.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.161+0000 7f6416e24700 1 --2- 172.21.15.31:0/1301479118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64101097e0 0x7f641019c400 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:44:49.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.161+0000 7f6416e24700 1 -- 172.21.15.31:0/1301479118 >> 172.21.15.31:0/1301479118 conn(0x7f6410103210 msgr2=0x7f6410105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:44:49.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.161+0000 7f6416e24700 1 -- 172.21.15.31:0/1301479118 shutdown_connections 2024-04-26T17:44:49.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:49.161+0000 7f6416e24700 1 -- 172.21.15.31:0/1301479118 wait complete. 2024-04-26T17:44:49.713 INFO:teuthology.orchestra.run.smithi031.stdout:mon is available 2024-04-26T17:44:49.713 INFO:teuthology.orchestra.run.smithi031.stdout:Assimilating anything we can from ceph.conf... 2024-04-26T17:44:50.797 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:50 smithi031 ceph-mon[28794]: from='client.? 172.21.15.31:0/3377715057' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T17:44:50.807 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-26T17:44:50.807 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T17:44:50.807 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout fsid = 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:44:50.807 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T17:44:50.807 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2024-04-26T17:44:50.807 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T17:44:50.807 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T17:44:50.807 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T17:44:50.808 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T17:44:50.808 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-26T17:44:50.808 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T17:44:50.808 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T17:44:50.808 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-26T17:44:50.808 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T17:44:50.808 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T17:44:50.808 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T17:44:50.808 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T17:44:50.808 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T17:44:50.808 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T17:44:50.808 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.120+0000 7efdc7c1b700 1 Processor -- start 2024-04-26T17:44:50.808 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.121+0000 7efdc7c1b700 1 -- start start 2024-04-26T17:44:50.809 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.121+0000 7efdc7c1b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc0108d50 0x7efdc0109170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:44:50.809 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.121+0000 7efdc7c1b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efdc0109740 con 0x7efdc0108d50 2024-04-26T17:44:50.809 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.121+0000 7efdc59b7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc0108d50 0x7efdc0109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:44:50.809 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.121+0000 7efdc59b7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc0108d50 0x7efdc0109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43206/0 (socket says 172.21.15.31:43206) 2024-04-26T17:44:50.809 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.121+0000 7efdc59b7700 1 -- 172.21.15.31:0/3592427855 learned_addr learned my addr 172.21.15.31:0/3592427855 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:44:50.809 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.121+0000 7efdc59b7700 1 -- 172.21.15.31:0/3592427855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc0109f00 con 0x7efdc0108d50 2024-04-26T17:44:50.809 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.122+0000 7efdc59b7700 1 --2- 172.21.15.31:0/3592427855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc0108d50 0x7efdc0109170 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7efdb400f960 tx=0x7efdb400fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21bb7dc0b18c810b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:44:50.809 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.122+0000 7efdc49b5700 1 -- 172.21.15.31:0/3592427855 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdb401c070 con 0x7efdc0108d50 2024-04-26T17:44:50.809 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.122+0000 7efdc49b5700 1 -- 172.21.15.31:0/3592427855 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdb4009b30 con 0x7efdc0108d50 2024-04-26T17:44:50.809 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.122+0000 7efdc49b5700 1 -- 172.21.15.31:0/3592427855 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdb4009e10 con 0x7efdc0108d50 2024-04-26T17:44:50.809 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.122+0000 7efdc7c1b700 1 -- 172.21.15.31:0/3592427855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc0108d50 msgr2=0x7efdc0109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:44:50.809 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.122+0000 7efdc7c1b700 1 --2- 172.21.15.31:0/3592427855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc0108d50 0x7efdc0109170 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7efdb400f960 tx=0x7efdb400fc70 comp rx=0 tx=0).stop 2024-04-26T17:44:50.809 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.122+0000 7efdc7c1b700 1 -- 172.21.15.31:0/3592427855 shutdown_connections 2024-04-26T17:44:50.810 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.122+0000 7efdc7c1b700 1 --2- 172.21.15.31:0/3592427855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc0108d50 0x7efdc0109170 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:44:50.810 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.122+0000 7efdc7c1b700 1 -- 172.21.15.31:0/3592427855 >> 172.21.15.31:0/3592427855 conn(0x7efdc0104310 msgr2=0x7efdc0106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:44:50.810 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.123+0000 7efdc7c1b700 1 -- 172.21.15.31:0/3592427855 shutdown_connections 2024-04-26T17:44:50.810 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.123+0000 7efdc7c1b700 1 -- 172.21.15.31:0/3592427855 wait complete. 2024-04-26T17:44:50.810 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.123+0000 7efdc7c1b700 1 Processor -- start 2024-04-26T17:44:50.810 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.123+0000 7efdc7c1b700 1 -- start start 2024-04-26T17:44:50.810 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.123+0000 7efdc7c1b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc0108d50 0x7efdc019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:44:50.810 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.123+0000 7efdc7c1b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efdc010a370 con 0x7efdc0108d50 2024-04-26T17:44:50.810 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.123+0000 7efdc59b7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc0108d50 0x7efdc019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:44:50.810 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.123+0000 7efdc59b7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc0108d50 0x7efdc019a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43222/0 (socket says 172.21.15.31:43222) 2024-04-26T17:44:50.810 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.123+0000 7efdc59b7700 1 -- 172.21.15.31:0/3377715057 learned_addr learned my addr 172.21.15.31:0/3377715057 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:44:50.810 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.124+0000 7efdc59b7700 1 -- 172.21.15.31:0/3377715057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc019a7a0 con 0x7efdc0108d50 2024-04-26T17:44:50.810 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.124+0000 7efdc59b7700 1 --2- 172.21.15.31:0/3377715057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc0108d50 0x7efdc019a260 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7efdb401a040 tx=0x7efdb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:44:50.810 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.124+0000 7efdb2ffd700 1 -- 172.21.15.31:0/3377715057 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdb4027020 con 0x7efdc0108d50 2024-04-26T17:44:50.811 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.124+0000 7efdb2ffd700 1 -- 172.21.15.31:0/3377715057 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdb4003850 con 0x7efdc0108d50 2024-04-26T17:44:50.811 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.124+0000 7efdc7c1b700 1 -- 172.21.15.31:0/3377715057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdc019aa30 con 0x7efdc0108d50 2024-04-26T17:44:50.811 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.124+0000 7efdb2ffd700 1 -- 172.21.15.31:0/3377715057 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdb401c040 con 0x7efdc0108d50 2024-04-26T17:44:50.811 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.124+0000 7efdc7c1b700 1 -- 172.21.15.31:0/3377715057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdc019d720 con 0x7efdc0108d50 2024-04-26T17:44:50.811 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.124+0000 7efdb2ffd700 1 -- 172.21.15.31:0/3377715057 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efdb402a020 con 0x7efdc0108d50 2024-04-26T17:44:50.811 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.124+0000 7efdb2ffd700 1 -- 172.21.15.31:0/3377715057 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efdb4014070 con 0x7efdc0108d50 2024-04-26T17:44:50.811 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.125+0000 7efdc7c1b700 1 -- 172.21.15.31:0/3377715057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdc0064a80 con 0x7efdc0108d50 2024-04-26T17:44:50.811 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.127+0000 7efdb2ffd700 1 -- 172.21.15.31:0/3377715057 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7efdb4031030 con 0x7efdc0108d50 2024-04-26T17:44:50.811 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.197+0000 7efdc7c1b700 1 -- 172.21.15.31:0/3377715057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7efdc019d8f0 con 0x7efdc0108d50 2024-04-26T17:44:50.811 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.200+0000 7efdb2ffd700 1 -- 172.21.15.31:0/3377715057 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7efdb4022470 con 0x7efdc0108d50 2024-04-26T17:44:50.811 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.201+0000 7efdb2ffd700 1 -- 172.21.15.31:0/3377715057 <== mon.0 v2:172.21.15.31:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efdb4017450 con 0x7efdc0108d50 2024-04-26T17:44:50.811 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.202+0000 7efdc7c1b700 1 -- 172.21.15.31:0/3377715057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc0108d50 msgr2=0x7efdc019a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:44:50.812 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.202+0000 7efdc7c1b700 1 --2- 172.21.15.31:0/3377715057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc0108d50 0x7efdc019a260 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7efdb401a040 tx=0x7efdb4004750 comp rx=0 tx=0).stop 2024-04-26T17:44:50.812 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.202+0000 7efdc7c1b700 1 -- 172.21.15.31:0/3377715057 shutdown_connections 2024-04-26T17:44:50.812 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.202+0000 7efdc7c1b700 1 --2- 172.21.15.31:0/3377715057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdc0108d50 0x7efdc019a260 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:44:50.812 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.202+0000 7efdc7c1b700 1 -- 172.21.15.31:0/3377715057 >> 172.21.15.31:0/3377715057 conn(0x7efdc0104310 msgr2=0x7efdc01049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:44:50.812 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.202+0000 7efdc7c1b700 1 -- 172.21.15.31:0/3377715057 shutdown_connections 2024-04-26T17:44:50.812 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:50.202+0000 7efdc7c1b700 1 -- 172.21.15.31:0/3377715057 wait complete. 2024-04-26T17:44:50.812 INFO:teuthology.orchestra.run.smithi031.stdout:Generating new minimal ceph.conf... 2024-04-26T17:44:51.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:51 smithi031 ceph-mon[28794]: from='client.? 172.21.15.31:0/4235082162' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:44:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.229+0000 7ff93a34c700 1 Processor -- start 2024-04-26T17:44:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.229+0000 7ff93a34c700 1 -- start start 2024-04-26T17:44:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.229+0000 7ff93a34c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff934107610 0x7ff934107a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:44:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.229+0000 7ff93a34c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff93407ce90 con 0x7ff934107610 2024-04-26T17:44:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.229+0000 7ff933fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff934107610 0x7ff934107a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:44:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.229+0000 7ff933fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff934107610 0x7ff934107a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43230/0 (socket says 172.21.15.31:43230) 2024-04-26T17:44:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.229+0000 7ff933fff700 1 -- 172.21.15.31:0/634783011 learned_addr learned my addr 172.21.15.31:0/634783011 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:44:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.229+0000 7ff933fff700 1 -- 172.21.15.31:0/634783011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff934107f70 con 0x7ff934107610 2024-04-26T17:44:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.229+0000 7ff933fff700 1 --2- 172.21.15.31:0/634783011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff934107610 0x7ff934107a30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff9280098a0 tx=0x7ff928009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58dbcd5e29d93d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:44:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.230+0000 7ff932ffd700 1 -- 172.21.15.31:0/634783011 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff928004160 con 0x7ff934107610 2024-04-26T17:44:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.230+0000 7ff932ffd700 1 -- 172.21.15.31:0/634783011 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff928004300 con 0x7ff934107610 2024-04-26T17:44:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.230+0000 7ff932ffd700 1 -- 172.21.15.31:0/634783011 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff928011af0 con 0x7ff934107610 2024-04-26T17:44:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.230+0000 7ff93a34c700 1 -- 172.21.15.31:0/634783011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff934107610 msgr2=0x7ff934107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:44:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.230+0000 7ff93a34c700 1 --2- 172.21.15.31:0/634783011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff934107610 0x7ff934107a30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff9280098a0 tx=0x7ff928009bb0 comp rx=0 tx=0).stop 2024-04-26T17:44:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.230+0000 7ff93a34c700 1 -- 172.21.15.31:0/634783011 shutdown_connections 2024-04-26T17:44:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.230+0000 7ff93a34c700 1 --2- 172.21.15.31:0/634783011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff934107610 0x7ff934107a30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:44:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.230+0000 7ff93a34c700 1 -- 172.21.15.31:0/634783011 >> 172.21.15.31:0/634783011 conn(0x7ff934103230 msgr2=0x7ff934105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:44:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.230+0000 7ff93a34c700 1 -- 172.21.15.31:0/634783011 shutdown_connections 2024-04-26T17:44:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.230+0000 7ff93a34c700 1 -- 172.21.15.31:0/634783011 wait complete. 2024-04-26T17:44:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.231+0000 7ff93a34c700 1 Processor -- start 2024-04-26T17:44:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.231+0000 7ff93a34c700 1 -- start start 2024-04-26T17:44:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.231+0000 7ff93a34c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff934107610 0x7ff93419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:44:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.231+0000 7ff93a34c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff934108fa0 con 0x7ff934107610 2024-04-26T17:44:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.231+0000 7ff933fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff934107610 0x7ff93419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:44:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.231+0000 7ff933fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff934107610 0x7ff93419ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43236/0 (socket says 172.21.15.31:43236) 2024-04-26T17:44:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.231+0000 7ff933fff700 1 -- 172.21.15.31:0/4235082162 learned_addr learned my addr 172.21.15.31:0/4235082162 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:44:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.232+0000 7ff933fff700 1 -- 172.21.15.31:0/4235082162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff93419f440 con 0x7ff934107610 2024-04-26T17:44:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.232+0000 7ff933fff700 1 --2- 172.21.15.31:0/4235082162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff934107610 0x7ff93419ef00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff928009fb0 tx=0x7ff928011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:44:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.232+0000 7ff9317fa700 1 -- 172.21.15.31:0/4235082162 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9280036a0 con 0x7ff934107610 2024-04-26T17:44:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.232+0000 7ff9317fa700 1 -- 172.21.15.31:0/4235082162 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff92801a430 con 0x7ff934107610 2024-04-26T17:44:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.232+0000 7ff93a34c700 1 -- 172.21.15.31:0/4235082162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff93419f6d0 con 0x7ff934107610 2024-04-26T17:44:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.232+0000 7ff9317fa700 1 -- 172.21.15.31:0/4235082162 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff928019550 con 0x7ff934107610 2024-04-26T17:44:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.232+0000 7ff93a34c700 1 -- 172.21.15.31:0/4235082162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9341a23c0 con 0x7ff934107610 2024-04-26T17:44:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.232+0000 7ff9317fa700 1 -- 172.21.15.31:0/4235082162 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff928019be0 con 0x7ff934107610 2024-04-26T17:44:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.233+0000 7ff9317fa700 1 -- 172.21.15.31:0/4235082162 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff928024730 con 0x7ff934107610 2024-04-26T17:44:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.233+0000 7ff93a34c700 1 -- 172.21.15.31:0/4235082162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff934064a80 con 0x7ff934107610 2024-04-26T17:44:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.235+0000 7ff9317fa700 1 -- 172.21.15.31:0/4235082162 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ff928024930 con 0x7ff934107610 2024-04-26T17:44:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.306+0000 7ff93a34c700 1 -- 172.21.15.31:0/4235082162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7ff93402e700 con 0x7ff934107610 2024-04-26T17:44:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.306+0000 7ff9317fa700 1 -- 172.21.15.31:0/4235082162 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7ff92801a5e0 con 0x7ff934107610 2024-04-26T17:44:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.307+0000 7ff93a34c700 1 -- 172.21.15.31:0/4235082162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff934107610 msgr2=0x7ff93419ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:44:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.307+0000 7ff93a34c700 1 --2- 172.21.15.31:0/4235082162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff934107610 0x7ff93419ef00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff928009fb0 tx=0x7ff928011940 comp rx=0 tx=0).stop 2024-04-26T17:44:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.307+0000 7ff93a34c700 1 -- 172.21.15.31:0/4235082162 shutdown_connections 2024-04-26T17:44:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.307+0000 7ff93a34c700 1 --2- 172.21.15.31:0/4235082162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff934107610 0x7ff93419ef00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:44:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.307+0000 7ff93a34c700 1 -- 172.21.15.31:0/4235082162 >> 172.21.15.31:0/4235082162 conn(0x7ff934103230 msgr2=0x7ff934195b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:44:51.881 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.307+0000 7ff93a34c700 1 -- 172.21.15.31:0/4235082162 shutdown_connections 2024-04-26T17:44:51.881 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:51.307+0000 7ff93a34c700 1 -- 172.21.15.31:0/4235082162 wait complete. 2024-04-26T17:44:51.881 INFO:teuthology.orchestra.run.smithi031.stdout:Restarting the monitor... 2024-04-26T17:44:52.149 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:51 smithi031 systemd[1]: Stopping Ceph mon.a for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:44:52.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:52 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a[28790]: 2024-04-26T17:44:52.148+0000 7f2ec212b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T17:44:52.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:52 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a[28790]: 2024-04-26T17:44:52.148+0000 7f2ec212b700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T17:44:52.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:52 smithi031 podman[29037]: 2024-04-26 17:44:52.395290429 +0000 UTC m=+0.345887864 container died 84e45b09922ea1bb93623f35a74b9ce246d2afdae852ca4151eeb921692ccc8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-26T17:44:53.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:52 smithi031 podman[29037]: 2024-04-26 17:44:52.706796634 +0000 UTC m=+0.657394049 container cleanup 84e45b09922ea1bb93623f35a74b9ce246d2afdae852ca4151eeb921692ccc8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-26T17:44:53.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:52 smithi031 bash[29037]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a 2024-04-26T17:44:53.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:52 smithi031 podman[29051]: 2024-04-26 17:44:52.882563801 +0000 UTC m=+0.481533524 container remove 84e45b09922ea1bb93623f35a74b9ce246d2afdae852ca4151eeb921692ccc8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T17:44:53.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T17:44:53.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 systemd[1]: Stopped Ceph mon.a for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:44:53.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 systemd[1]: Starting Ceph mon.a for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:44:53.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 podman[29145]: 2024-04-26 17:44:53.225247602 +0000 UTC m=+0.097057420 container create bd136c7b0b4e18ff5e53e302625de5bf7259e4adcbaf5b1565680493ea4d6b14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-26T17:44:53.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 podman[29145]: 2024-04-26 17:44:53.146564238 +0000 UTC m=+0.018374075 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 2024-04-26T17:44:53.538 INFO:teuthology.orchestra.run.smithi031.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-26T17:44:53.664 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 podman[29145]: 2024-04-26 17:44:53.419014136 +0000 UTC m=+0.290823954 container init bd136c7b0b4e18ff5e53e302625de5bf7259e4adcbaf5b1565680493ea4d6b14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T17:44:53.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 podman[29145]: 2024-04-26 17:44:53.42218572 +0000 UTC m=+0.293995537 container start bd136c7b0b4e18ff5e53e302625de5bf7259e4adcbaf5b1565680493ea4d6b14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-04-26T17:44:53.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T17:44:53.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev), process ceph-mon, pid 2 2024-04-26T17:44:53.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: pidfile_write: ignore empty --pid-file 2024-04-26T17:44:53.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: load: jerasure load: lrc 2024-04-26T17:44:53.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: RocksDB version: 7.9.2 2024-04-26T17:44:53.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Git sha 0 2024-04-26T17:44:53.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Compile date 2024-04-26 07:20:26 2024-04-26T17:44:53.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: DB SUMMARY 2024-04-26T17:44:53.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: DB Session ID: V7KV5AG9N08NYEV0931O 2024-04-26T17:44:53.666 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: CURRENT file: CURRENT 2024-04-26T17:44:53.666 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T17:44:53.666 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-26T17:44:53.666 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-26T17:44:53.666 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104534 ; 2024-04-26T17:44:53.666 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.error_if_exists: 0 2024-04-26T17:44:53.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.create_if_missing: 0 2024-04-26T17:44:53.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.paranoid_checks: 1 2024-04-26T17:44:53.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T17:44:53.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T17:44:53.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T17:44:53.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.env: 0x562f9e1f9cc0 2024-04-26T17:44:53.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T17:44:53.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.info_log: 0x562f9f935d80 2024-04-26T17:44:53.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T17:44:53.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.statistics: (nil) 2024-04-26T17:44:53.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.use_fsync: 0 2024-04-26T17:44:53.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_log_file_size: 0 2024-04-26T17:44:53.668 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T17:44:53.668 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T17:44:53.668 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T17:44:53.668 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T17:44:53.668 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.allow_fallocate: 1 2024-04-26T17:44:53.668 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T17:44:53.668 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T17:44:53.668 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.use_direct_reads: 0 2024-04-26T17:44:53.668 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T17:44:53.668 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T17:44:53.668 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.db_log_dir: 2024-04-26T17:44:53.668 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.wal_dir: 2024-04-26T17:44:53.668 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T17:44:53.668 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T17:44:53.669 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T17:44:53.669 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T17:44:53.669 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T17:44:53.669 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T17:44:53.669 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T17:44:53.669 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T17:44:53.669 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.write_buffer_manager: 0x562f9f9070e0 2024-04-26T17:44:53.669 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T17:44:53.669 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T17:44:53.669 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T17:44:53.669 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T17:44:53.669 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T17:44:53.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T17:44:53.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T17:44:53.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T17:44:53.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.unordered_write: 0 2024-04-26T17:44:53.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T17:44:53.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T17:44:53.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T17:44:53.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T17:44:53.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.row_cache: None 2024-04-26T17:44:53.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.wal_filter: None 2024-04-26T17:44:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T17:44:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T17:44:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.two_write_queues: 0 2024-04-26T17:44:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T17:44:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.wal_compression: 0 2024-04-26T17:44:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.atomic_flush: 0 2024-04-26T17:44:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T17:44:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T17:44:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T17:44:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.log_readahead_size: 0 2024-04-26T17:44:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T17:44:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T17:44:53.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T17:44:53.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T17:44:53.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T17:44:53.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T17:44:53.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T17:44:53.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_background_jobs: 2 2024-04-26T17:44:53.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_background_compactions: -1 2024-04-26T17:44:53.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_subcompactions: 1 2024-04-26T17:44:53.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T17:44:53.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T17:44:53.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T17:44:53.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T17:44:53.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T17:44:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T17:44:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T17:44:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T17:44:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_open_files: -1 2024-04-26T17:44:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T17:44:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T17:44:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T17:44:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T17:44:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_background_flushes: -1 2024-04-26T17:44:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Compression algorithms supported: 2024-04-26T17:44:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T17:44:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: kZSTD supported: 0 2024-04-26T17:44:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: kXpressCompression supported: 0 2024-04-26T17:44:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T17:44:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: kZlibCompression supported: 1 2024-04-26T17:44:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: kSnappyCompression supported: 1 2024-04-26T17:44:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: kLZ4Compression supported: 1 2024-04-26T17:44:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: kBZip2Compression supported: 0 2024-04-26T17:44:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T17:44:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T17:44:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-26T17:44:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T17:44:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T17:44:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.merge_operator: 2024-04-26T17:44:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compaction_filter: None 2024-04-26T17:44:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T17:44:53.676 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T17:44:53.676 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T17:44:53.676 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T17:44:53.676 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562f9f935f40) 2024-04-26T17:44:53.676 INFO:journalctl@ceph.mon.a.smithi031.stdout: cache_index_and_filter_blocks: 1 2024-04-26T17:44:53.676 INFO:journalctl@ceph.mon.a.smithi031.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T17:44:53.676 INFO:journalctl@ceph.mon.a.smithi031.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T17:44:53.676 INFO:journalctl@ceph.mon.a.smithi031.stdout: pin_top_level_index_and_filter: 1 2024-04-26T17:44:53.676 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_type: 0 2024-04-26T17:44:53.676 INFO:journalctl@ceph.mon.a.smithi031.stdout: data_block_index_type: 0 2024-04-26T17:44:53.676 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_shortening: 1 2024-04-26T17:44:53.676 INFO:journalctl@ceph.mon.a.smithi031.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T17:44:53.676 INFO:journalctl@ceph.mon.a.smithi031.stdout: checksum: 4 2024-04-26T17:44:53.677 INFO:journalctl@ceph.mon.a.smithi031.stdout: no_block_cache: 0 2024-04-26T17:44:53.677 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache: 0x562f9f9391f0 2024-04-26T17:44:53.677 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_name: BinnedLRUCache 2024-04-26T17:44:53.677 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_options: 2024-04-26T17:44:53.677 INFO:journalctl@ceph.mon.a.smithi031.stdout: capacity : 536870912 2024-04-26T17:44:53.677 INFO:journalctl@ceph.mon.a.smithi031.stdout: num_shard_bits : 4 2024-04-26T17:44:53.677 INFO:journalctl@ceph.mon.a.smithi031.stdout: strict_capacity_limit : 0 2024-04-26T17:44:53.677 INFO:journalctl@ceph.mon.a.smithi031.stdout: high_pri_pool_ratio: 0.000 2024-04-26T17:44:53.677 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_compressed: (nil) 2024-04-26T17:44:53.677 INFO:journalctl@ceph.mon.a.smithi031.stdout: persistent_cache: (nil) 2024-04-26T17:44:53.678 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_size: 4096 2024-04-26T17:44:53.678 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_size_deviation: 10 2024-04-26T17:44:53.678 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_restart_interval: 16 2024-04-26T17:44:53.678 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_block_restart_interval: 1 2024-04-26T17:44:53.678 INFO:journalctl@ceph.mon.a.smithi031.stdout: metadata_block_size: 4096 2024-04-26T17:44:53.678 INFO:journalctl@ceph.mon.a.smithi031.stdout: partition_filters: 0 2024-04-26T17:44:53.678 INFO:journalctl@ceph.mon.a.smithi031.stdout: use_delta_encoding: 1 2024-04-26T17:44:53.678 INFO:journalctl@ceph.mon.a.smithi031.stdout: filter_policy: bloomfilter 2024-04-26T17:44:53.678 INFO:journalctl@ceph.mon.a.smithi031.stdout: whole_key_filtering: 1 2024-04-26T17:44:53.679 INFO:journalctl@ceph.mon.a.smithi031.stdout: verify_compression: 0 2024-04-26T17:44:53.679 INFO:journalctl@ceph.mon.a.smithi031.stdout: read_amp_bytes_per_bit: 0 2024-04-26T17:44:53.679 INFO:journalctl@ceph.mon.a.smithi031.stdout: format_version: 5 2024-04-26T17:44:53.679 INFO:journalctl@ceph.mon.a.smithi031.stdout: enable_index_compression: 1 2024-04-26T17:44:53.679 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_align: 0 2024-04-26T17:44:53.679 INFO:journalctl@ceph.mon.a.smithi031.stdout: max_auto_readahead_size: 262144 2024-04-26T17:44:53.679 INFO:journalctl@ceph.mon.a.smithi031.stdout: prepopulate_block_cache: 0 2024-04-26T17:44:53.679 INFO:journalctl@ceph.mon.a.smithi031.stdout: initial_auto_readahead_size: 8192 2024-04-26T17:44:53.679 INFO:journalctl@ceph.mon.a.smithi031.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T17:44:53.679 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T17:44:53.679 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T17:44:53.679 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compression: NoCompression 2024-04-26T17:44:53.679 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T17:44:53.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T17:44:53.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T17:44:53.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.num_levels: 7 2024-04-26T17:44:53.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T17:44:53.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T17:44:53.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T17:44:53.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T17:44:53.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T17:44:53.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T17:44:53.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T17:44:53.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T17:44:53.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T17:44:53.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T17:44:53.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T17:44:53.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T17:44:53.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T17:44:53.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T17:44:53.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T17:44:53.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T17:44:53.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T17:44:53.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T17:44:53.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T17:44:53.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T17:44:53.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T17:44:53.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T17:44:53.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T17:44:53.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T17:44:53.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T17:44:53.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T17:44:53.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T17:44:53.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T17:44:53.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T17:44:53.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T17:44:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T17:44:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T17:44:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T17:44:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T17:44:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T17:44:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T17:44:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T17:44:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T17:44:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T17:44:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T17:44:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T17:44:53.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T17:44:53.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T17:44:53.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T17:44:53.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T17:44:53.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T17:44:53.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T17:44:53.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T17:44:53.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T17:44:53.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T17:44:53.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T17:44:53.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T17:44:53.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T17:44:53.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T17:44:53.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.inplace_update_support: 0 2024-04-26T17:44:53.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T17:44:53.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T17:44:53.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T17:44:53.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T17:44:53.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.bloom_locality: 0 2024-04-26T17:44:53.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.max_successive_merges: 0 2024-04-26T17:44:53.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T17:44:53.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T17:44:53.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T17:44:53.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T17:44:53.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.ttl: 2592000 2024-04-26T17:44:53.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T17:44:53.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T17:44:53.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T17:44:53.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.enable_blob_files: false 2024-04-26T17:44:53.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.min_blob_size: 0 2024-04-26T17:44:53.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T17:44:53.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T17:44:53.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T17:44:53.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T17:44:53.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T17:44:53.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T17:44:53.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T17:44:53.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T17:44:53.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-26T17:44:53.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-26T17:44:53.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7fbdf264-bbb2-4933-9bbd-ded2d729b7ac 2024-04-26T17:44:53.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714153493451448, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-26T17:44:53.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-26T17:44:53.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714153493452561, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99761, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97643, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14028, "raw_average_key_size": 49, "raw_value_size": 90001, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714153493, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7fbdf264-bbb2-4933-9bbd-ded2d729b7ac", "db_session_id": "V7KV5AG9N08NYEV0931O", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-26T17:44:53.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714153493452650, "job": 1, "event": "recovery_finished"} 2024-04-26T17:44:53.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-26T17:44:53.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T17:44:53.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562f9fa56000 2024-04-26T17:44:53.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: DB pointer 0x562f9fa42000 2024-04-26T17:44:53.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T17:44:53.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T17:44:53.688 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** DB Stats ** 2024-04-26T17:44:53.688 INFO:journalctl@ceph.mon.a.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T17:44:53.688 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T17:44:53.688 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T17:44:53.688 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T17:44:53.688 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T17:44:53.688 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T17:44:53.688 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T17:44:53.688 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-26T17:44:53.689 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** Compaction Stats [default] ** 2024-04-26T17:44:53.689 INFO:journalctl@ceph.mon.a.smithi031.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T17:44:53.689 INFO:journalctl@ceph.mon.a.smithi031.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T17:44:53.689 INFO:journalctl@ceph.mon.a.smithi031.stdout: L0 2/0 99.28 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T17:44:53.689 INFO:journalctl@ceph.mon.a.smithi031.stdout: Sum 2/0 99.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T17:44:53.689 INFO:journalctl@ceph.mon.a.smithi031.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T17:44:53.689 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-26T17:44:53.689 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** Compaction Stats [default] ** 2024-04-26T17:44:53.689 INFO:journalctl@ceph.mon.a.smithi031.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T17:44:53.689 INFO:journalctl@ceph.mon.a.smithi031.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T17:44:53.689 INFO:journalctl@ceph.mon.a.smithi031.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 136.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T17:44:53.690 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-26T17:44:53.690 INFO:journalctl@ceph.mon.a.smithi031.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T17:44:53.690 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-26T17:44:53.690 INFO:journalctl@ceph.mon.a.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T17:44:53.690 INFO:journalctl@ceph.mon.a.smithi031.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T17:44:53.690 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T17:44:53.690 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T17:44:53.690 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T17:44:53.690 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T17:44:53.690 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative compaction: 0.00 GB write, 29.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T17:44:53.690 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval compaction: 0.00 GB write, 29.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T17:44:53.691 INFO:journalctl@ceph.mon.a.smithi031.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T17:44:53.691 INFO:journalctl@ceph.mon.a.smithi031.stdout: Block cache BinnedLRUCache@0x562f9f9391f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-26T17:44:53.691 INFO:journalctl@ceph.mon.a.smithi031.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-26T17:44:53.691 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-26T17:44:53.691 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T17:44:53.691 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: starting mon.a rank 0 at public addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] at bind addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:44:53.691 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: mon.a@-1(???) e1 preinit fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:44:53.691 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: mon.a@-1(???).mds e1 new map 2024-04-26T17:44:53.691 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: mon.a@-1(???).mds e1 print_map 2024-04-26T17:44:53.691 INFO:journalctl@ceph.mon.a.smithi031.stdout: e1 2024-04-26T17:44:53.691 INFO:journalctl@ceph.mon.a.smithi031.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T17:44:53.691 INFO:journalctl@ceph.mon.a.smithi031.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T17:44:53.691 INFO:journalctl@ceph.mon.a.smithi031.stdout: legacy client fscid: -1 2024-04-26T17:44:53.691 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout: No filesystems configured 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: monmap epoch 1 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: last_changed 2024-04-26T17:44:45.630672+0000 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: created 2024-04-26T17:44:45.630672+0000 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: min_mon_release 19 (squid) 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: election_strategy: 1 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-26T17:44:53.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: fsmap 2024-04-26T17:44:53.693 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T17:44:53.693 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 ceph-mon[29164]: mgrmap e1: no daemons active 2024-04-26T17:44:53.693 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 bash[29145]: bd136c7b0b4e18ff5e53e302625de5bf7259e4adcbaf5b1565680493ea4d6b14 2024-04-26T17:44:53.693 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:53 smithi031 systemd[1]: Started Ceph mon.a for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:44:54.640 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.052+0000 7f2a683fd700 1 Processor -- start 2024-04-26T17:44:54.640 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.052+0000 7f2a683fd700 1 -- start start 2024-04-26T17:44:54.640 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.052+0000 7f2a683fd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a60108e20 0x7f2a60109240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:44:54.640 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.052+0000 7f2a683fd700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2a60109810 con 0x7f2a60108e20 2024-04-26T17:44:54.640 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.053+0000 7f2a66199700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a60108e20 0x7f2a60109240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:44:54.640 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.053+0000 7f2a66199700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a60108e20 0x7f2a60109240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43238/0 (socket says 172.21.15.31:43238) 2024-04-26T17:44:54.640 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.053+0000 7f2a66199700 1 -- 172.21.15.31:0/1124725621 learned_addr learned my addr 172.21.15.31:0/1124725621 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:44:54.640 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.053+0000 7f2a66199700 1 -- 172.21.15.31:0/1124725621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a60109f80 con 0x7f2a60108e20 2024-04-26T17:44:54.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.053+0000 7f2a66199700 1 --2- 172.21.15.31:0/1124725621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a60108e20 0x7f2a60109240 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2a580098a0 tx=0x7f2a58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c5042ffe1a8a70 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:44:54.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.054+0000 7f2a65197700 1 -- 172.21.15.31:0/1124725621 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a58004160 con 0x7f2a60108e20 2024-04-26T17:44:54.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.054+0000 7f2a65197700 1 -- 172.21.15.31:0/1124725621 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2a58004300 con 0x7f2a60108e20 2024-04-26T17:44:54.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.054+0000 7f2a65197700 1 -- 172.21.15.31:0/1124725621 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a58011af0 con 0x7f2a60108e20 2024-04-26T17:44:54.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.054+0000 7f2a683fd700 1 -- 172.21.15.31:0/1124725621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a60108e20 msgr2=0x7f2a60109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:44:54.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.054+0000 7f2a683fd700 1 --2- 172.21.15.31:0/1124725621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a60108e20 0x7f2a60109240 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2a580098a0 tx=0x7f2a58009bb0 comp rx=0 tx=0).stop 2024-04-26T17:44:54.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.054+0000 7f2a683fd700 1 -- 172.21.15.31:0/1124725621 shutdown_connections 2024-04-26T17:44:54.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.054+0000 7f2a683fd700 1 --2- 172.21.15.31:0/1124725621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a60108e20 0x7f2a60109240 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:44:54.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.054+0000 7f2a683fd700 1 -- 172.21.15.31:0/1124725621 >> 172.21.15.31:0/1124725621 conn(0x7f2a60104400 msgr2=0x7f2a60106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:44:54.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.054+0000 7f2a683fd700 1 -- 172.21.15.31:0/1124725621 shutdown_connections 2024-04-26T17:44:54.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.054+0000 7f2a683fd700 1 -- 172.21.15.31:0/1124725621 wait complete. 2024-04-26T17:44:54.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.055+0000 7f2a683fd700 1 Processor -- start 2024-04-26T17:44:54.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.055+0000 7f2a683fd700 1 -- start start 2024-04-26T17:44:54.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.055+0000 7f2a683fd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a60108e20 0x7f2a6019cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:44:54.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.055+0000 7f2a683fd700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6010d330 con 0x7f2a60108e20 2024-04-26T17:44:54.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.055+0000 7f2a66199700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a60108e20 0x7f2a6019cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:44:54.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.055+0000 7f2a66199700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a60108e20 0x7f2a6019cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43244/0 (socket says 172.21.15.31:43244) 2024-04-26T17:44:54.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.055+0000 7f2a66199700 1 -- 172.21.15.31:0/1986541108 learned_addr learned my addr 172.21.15.31:0/1986541108 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:44:54.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.056+0000 7f2a66199700 1 -- 172.21.15.31:0/1986541108 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6019d250 con 0x7f2a60108e20 2024-04-26T17:44:54.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.056+0000 7f2a66199700 1 --2- 172.21.15.31:0/1986541108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a60108e20 0x7f2a6019cd10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2a580117b0 tx=0x7f2a58011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:44:54.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.056+0000 7f2a5f7fe700 1 -- 172.21.15.31:0/1986541108 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a58003af0 con 0x7f2a60108e20 2024-04-26T17:44:54.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.056+0000 7f2a5f7fe700 1 -- 172.21.15.31:0/1986541108 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2a58020070 con 0x7f2a60108e20 2024-04-26T17:44:54.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.056+0000 7f2a683fd700 1 -- 172.21.15.31:0/1986541108 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a6019d4e0 con 0x7f2a60108e20 2024-04-26T17:44:54.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.056+0000 7f2a5f7fe700 1 -- 172.21.15.31:0/1986541108 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a58019630 con 0x7f2a60108e20 2024-04-26T17:44:54.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.056+0000 7f2a683fd700 1 -- 172.21.15.31:0/1986541108 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a601a01c0 con 0x7f2a60108e20 2024-04-26T17:44:54.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.056+0000 7f2a5f7fe700 1 -- 172.21.15.31:0/1986541108 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2a58019c80 con 0x7f2a60108e20 2024-04-26T17:44:54.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.056+0000 7f2a5f7fe700 1 -- 172.21.15.31:0/1986541108 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2a5800b720 con 0x7f2a60108e20 2024-04-26T17:44:54.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.057+0000 7f2a683fd700 1 -- 172.21.15.31:0/1986541108 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a60064a80 con 0x7f2a60108e20 2024-04-26T17:44:54.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.059+0000 7f2a5f7fe700 1 -- 172.21.15.31:0/1986541108 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2a580042a0 con 0x7f2a60108e20 2024-04-26T17:44:54.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.129+0000 7f2a683fd700 1 -- 172.21.15.31:0/1986541108 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f2a601a0390 con 0x7f2a60108e20 2024-04-26T17:44:54.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.131+0000 7f2a5f7fe700 1 -- 172.21.15.31:0/1986541108 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f2a5801a9c0 con 0x7f2a60108e20 2024-04-26T17:44:54.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.132+0000 7f2a683fd700 1 -- 172.21.15.31:0/1986541108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a60108e20 msgr2=0x7f2a6019cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:44:54.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.132+0000 7f2a683fd700 1 --2- 172.21.15.31:0/1986541108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a60108e20 0x7f2a6019cd10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2a580117b0 tx=0x7f2a58011b70 comp rx=0 tx=0).stop 2024-04-26T17:44:54.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.132+0000 7f2a683fd700 1 -- 172.21.15.31:0/1986541108 shutdown_connections 2024-04-26T17:44:54.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.132+0000 7f2a683fd700 1 --2- 172.21.15.31:0/1986541108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a60108e20 0x7f2a6019cd10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:44:54.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.132+0000 7f2a683fd700 1 -- 172.21.15.31:0/1986541108 >> 172.21.15.31:0/1986541108 conn(0x7f2a60104400 msgr2=0x7f2a60104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:44:54.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.132+0000 7f2a683fd700 1 -- 172.21.15.31:0/1986541108 shutdown_connections 2024-04-26T17:44:54.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:54.132+0000 7f2a683fd700 1 -- 172.21.15.31:0/1986541108 wait complete. 2024-04-26T17:44:54.643 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-26T17:44:54.665 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-26T17:44:54.666 INFO:teuthology.orchestra.run.smithi031.stdout:Creating mgr... 2024-04-26T17:44:54.666 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-26T17:44:54.667 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-26T17:44:54.930 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mgr.x 2024-04-26T17:44:54.930 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-26T17:44:55.085 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7e53e050-03f4-11ef-bc93-c7b262605968.target.wants/ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-7e53e050-03f4-11ef-bc93-c7b262605968@.service. 2024-04-26T17:44:55.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:55 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1986541108' entity='client.admin' 2024-04-26T17:44:55.605 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:55 smithi031 podman[29380]: 2024-04-26 17:44:55.526486609 +0000 UTC m=+0.315498678 container start 7f2efebd461ef656468903d0ae1c18ccf025bd0917ef162313d42efa2a7fd1b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-26T17:44:55.620 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T17:44:55.620 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-04-26T17:44:55.628 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T17:44:55.628 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-26T17:44:55.628 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-04-26T17:44:55.628 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T17:44:55.636 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T17:44:55.636 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-04-26T17:44:55.644 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T17:44:55.644 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-26T17:44:55.644 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-04-26T17:44:55.644 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-26T17:44:55.645 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr to start... 2024-04-26T17:44:55.645 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr... 2024-04-26T17:44:55.878 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:55 smithi031 bash[29380]: 7f2efebd461ef656468903d0ae1c18ccf025bd0917ef162313d42efa2a7fd1b7 2024-04-26T17:44:55.879 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:55 smithi031 systemd[1]: Started Ceph mgr.x for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:44:55.879 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:55 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:44:55.684+0000 7f1f91936200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T17:44:55.879 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:55 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:44:55.878+0000 7f1f91936200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:44:56.178 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:55 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:44:55.986+0000 7f1f91936200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T17:44:56.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:44:56 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3086769320' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T17:44:56.559 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:56 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:44:56.322+0000 7f1f91936200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T17:44:56.559 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:56 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:44:56.410+0000 7f1f91936200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T17:44:58.032 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:57 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:44:57.676+0000 7f1f91936200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T17:44:58.032 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:57 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:44:57.854+0000 7f1f91936200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T17:44:58.032 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:58 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:44:58.032+0000 7f1f91936200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T17:44:58.109 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-26T17:44:58.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-26T17:44:58.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "7e53e050-03f4-11ef-bc93-c7b262605968", 2024-04-26T17:44:58.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T17:44:58.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T17:44:58.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T17:44:58.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T17:44:58.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:44:58.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T17:44:58.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T17:44:58.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-04-26T17:44:58.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-26T17:44:58.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T17:44:58.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T17:44:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-26T17:44:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-26T17:44:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T17:44:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T17:44:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T17:44:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T17:44:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:44:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T17:44:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T17:44:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T17:44:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T17:44:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T17:44:58.119 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T17:44:58.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T17:44:58.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T17:44:58.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:44:58.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T17:44:58.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T17:44:58.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T17:44:58.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T17:44:58.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T17:44:58.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T17:44:58.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T17:44:58.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T17:44:58.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T17:44:58.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:44:58.121 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T17:44:58.121 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T17:44:58.121 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T17:44:58.121 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T17:44:58.121 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:44:58.121 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T17:44:58.121 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T17:44:58.121 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T17:44:58.121 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T17:44:58.121 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T17:44:58.121 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T17:44:58.121 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T17:44:58.121 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-26T17:44:58.122 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T17:44:58.122 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:44:58.122 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T17:44:58.122 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T17:44:58.122 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T17:44:48.618737+0000", 2024-04-26T17:44:58.122 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T17:44:58.122 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:44:58.122 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T17:44:58.122 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-26T17:44:58.122 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.095+0000 7f92e3873700 1 Processor -- start 2024-04-26T17:44:58.122 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.096+0000 7f92e3873700 1 -- start start 2024-04-26T17:44:58.122 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.096+0000 7f92e3873700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92dc1097c0 0x7f92dc10bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:44:58.122 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.096+0000 7f92e3873700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc07ce90 con 0x7f92dc1097c0 2024-04-26T17:44:58.123 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.096+0000 7f92e160f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92dc1097c0 0x7f92dc10bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:44:58.123 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.096+0000 7f92e160f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92dc1097c0 0x7f92dc10bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56766/0 (socket says 172.21.15.31:56766) 2024-04-26T17:44:58.123 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.096+0000 7f92e160f700 1 -- 172.21.15.31:0/1170636368 learned_addr learned my addr 172.21.15.31:0/1170636368 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:44:58.123 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.097+0000 7f92e160f700 1 -- 172.21.15.31:0/1170636368 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92dc10c0f0 con 0x7f92dc1097c0 2024-04-26T17:44:58.123 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.097+0000 7f92e160f700 1 --2- 172.21.15.31:0/1170636368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92dc1097c0 0x7f92dc10bbb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f92d80098a0 tx=0x7f92d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8576813db86628b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:44:58.123 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.097+0000 7f92d3fff700 1 -- 172.21.15.31:0/1170636368 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92d8004160 con 0x7f92dc1097c0 2024-04-26T17:44:58.123 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.097+0000 7f92d3fff700 1 -- 172.21.15.31:0/1170636368 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f92d8004300 con 0x7f92dc1097c0 2024-04-26T17:44:58.123 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.097+0000 7f92d3fff700 1 -- 172.21.15.31:0/1170636368 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92d8011af0 con 0x7f92dc1097c0 2024-04-26T17:44:58.123 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.098+0000 7f92e3873700 1 -- 172.21.15.31:0/1170636368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92dc1097c0 msgr2=0x7f92dc10bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:44:58.123 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.098+0000 7f92e3873700 1 --2- 172.21.15.31:0/1170636368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92dc1097c0 0x7f92dc10bbb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f92d80098a0 tx=0x7f92d8009bb0 comp rx=0 tx=0).stop 2024-04-26T17:44:58.123 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.098+0000 7f92e3873700 1 -- 172.21.15.31:0/1170636368 shutdown_connections 2024-04-26T17:44:58.123 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.098+0000 7f92e3873700 1 --2- 172.21.15.31:0/1170636368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92dc1097c0 0x7f92dc10bbb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:44:58.123 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.098+0000 7f92e3873700 1 -- 172.21.15.31:0/1170636368 >> 172.21.15.31:0/1170636368 conn(0x7f92dc103210 msgr2=0x7f92dc105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:44:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.098+0000 7f92e3873700 1 -- 172.21.15.31:0/1170636368 shutdown_connections 2024-04-26T17:44:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.098+0000 7f92e3873700 1 -- 172.21.15.31:0/1170636368 wait complete. 2024-04-26T17:44:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.099+0000 7f92e3873700 1 Processor -- start 2024-04-26T17:44:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.099+0000 7f92e3873700 1 -- start start 2024-04-26T17:44:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.099+0000 7f92e3873700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92dc1097c0 0x7f92dc19ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:44:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.099+0000 7f92e3873700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc10d0b0 con 0x7f92dc1097c0 2024-04-26T17:44:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.099+0000 7f92e160f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92dc1097c0 0x7f92dc19ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:44:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.099+0000 7f92e160f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92dc1097c0 0x7f92dc19ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56782/0 (socket says 172.21.15.31:56782) 2024-04-26T17:44:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.099+0000 7f92e160f700 1 -- 172.21.15.31:0/3086769320 learned_addr learned my addr 172.21.15.31:0/3086769320 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:44:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.099+0000 7f92e160f700 1 -- 172.21.15.31:0/3086769320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92dc19f3c0 con 0x7f92dc1097c0 2024-04-26T17:44:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.099+0000 7f92e160f700 1 --2- 172.21.15.31:0/3086769320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92dc1097c0 0x7f92dc19ee80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f92d8003f40 tx=0x7f92d8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:44:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.100+0000 7f92d27fc700 1 -- 172.21.15.31:0/3086769320 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92d80038b0 con 0x7f92dc1097c0 2024-04-26T17:44:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.100+0000 7f92d27fc700 1 -- 172.21.15.31:0/3086769320 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f92d8003a50 con 0x7f92dc1097c0 2024-04-26T17:44:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.100+0000 7f92e3873700 1 -- 172.21.15.31:0/3086769320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92dc19f650 con 0x7f92dc1097c0 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.100+0000 7f92e3873700 1 -- 172.21.15.31:0/3086769320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92dc1a2340 con 0x7f92dc1097c0 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.100+0000 7f92d27fc700 1 -- 172.21.15.31:0/3086769320 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92d801a5c0 con 0x7f92dc1097c0 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.100+0000 7f92d27fc700 1 -- 172.21.15.31:0/3086769320 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f92d801ac50 con 0x7f92dc1097c0 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.100+0000 7f92d27fc700 1 -- 172.21.15.31:0/3086769320 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f92d8019700 con 0x7f92dc1097c0 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.101+0000 7f92e3873700 1 -- 172.21.15.31:0/3086769320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92dc064a80 con 0x7f92dc1097c0 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.103+0000 7f92d27fc700 1 -- 172.21.15.31:0/3086769320 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f92d8019950 con 0x7f92dc1097c0 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.179+0000 7f92e3873700 1 -- 172.21.15.31:0/3086769320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f92dc1a2510 con 0x7f92dc1097c0 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.179+0000 7f92d27fc700 1 -- 172.21.15.31:0/3086769320 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f92d8019b30 con 0x7f92dc1097c0 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.180+0000 7f92e3873700 1 -- 172.21.15.31:0/3086769320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92dc1097c0 msgr2=0x7f92dc19ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.180+0000 7f92e3873700 1 --2- 172.21.15.31:0/3086769320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92dc1097c0 0x7f92dc19ee80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f92d8003f40 tx=0x7f92d8011890 comp rx=0 tx=0).stop 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.180+0000 7f92e3873700 1 -- 172.21.15.31:0/3086769320 shutdown_connections 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.180+0000 7f92e3873700 1 --2- 172.21.15.31:0/3086769320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92dc1097c0 0x7f92dc19ee80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.180+0000 7f92e3873700 1 -- 172.21.15.31:0/3086769320 >> 172.21.15.31:0/3086769320 conn(0x7f92dc103210 msgr2=0x7f92dc103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.180+0000 7f92e3873700 1 -- 172.21.15.31:0/3086769320 shutdown_connections 2024-04-26T17:44:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:44:56.181+0000 7f92e3873700 1 -- 172.21.15.31:0/3086769320 wait complete. 2024-04-26T17:44:58.126 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (1/15)... 2024-04-26T17:44:58.646 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:58 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:44:58.335+0000 7f1f91936200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T17:44:58.646 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:58 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:44:58.433+0000 7f1f91936200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T17:44:58.646 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:58 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:44:58.646+0000 7f1f91936200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T17:44:59.059 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:58 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:44:58.847+0000 7f1f91936200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:44:59.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:59 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:44:59.475+0000 7f1f91936200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T17:45:00.233 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:44:59 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:44:59.913+0000 7f1f91936200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T17:45:00.233 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:00 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:00.014+0000 7f1f91936200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T17:45:00.233 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:00 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:00.233+0000 7f1f91936200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T17:45:00.794 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:00 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1480354461' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T17:45:01.059 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:00 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:00.794+0000 7f1f91936200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T17:45:01.059 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:00 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:00.893+0000 7f1f91936200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T17:45:01.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-26T17:45:01.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-26T17:45:01.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "7e53e050-03f4-11ef-bc93-c7b262605968", 2024-04-26T17:45:01.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T17:45:01.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T17:45:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T17:45:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T17:45:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:45:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T17:45:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T17:45:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-04-26T17:45:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-26T17:45:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T17:45:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T17:45:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-26T17:45:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-26T17:45:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T17:45:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T17:45:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T17:45:01.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T17:45:01.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:45:01.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T17:45:01.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T17:45:01.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T17:45:01.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T17:45:01.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T17:45:01.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T17:45:01.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T17:45:01.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T17:45:01.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:45:01.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T17:45:01.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T17:45:01.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T17:45:01.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T17:45:01.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T17:45:01.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T17:45:01.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T17:45:01.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T17:45:01.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T17:45:01.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:45:01.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:45:01.259 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T17:45:01.260 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T17:45:01.260 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T17:44:48.618737+0000", 2024-04-26T17:45:01.260 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T17:45:01.260 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:45:01.260 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T17:45:01.260 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-26T17:45:01.260 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.550+0000 7f76a6aeb700 1 Processor -- start 2024-04-26T17:45:01.260 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.550+0000 7f76a6aeb700 1 -- start start 2024-04-26T17:45:01.260 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.550+0000 7f76a6aeb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76a0109260 0x7f76a0109680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:01.260 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.550+0000 7f76a6aeb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f76a0109c50 con 0x7f76a0109260 2024-04-26T17:45:01.260 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.550+0000 7f76a4887700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76a0109260 0x7f76a0109680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:01.260 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.550+0000 7f76a4887700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76a0109260 0x7f76a0109680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56796/0 (socket says 172.21.15.31:56796) 2024-04-26T17:45:01.260 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.550+0000 7f76a4887700 1 -- 172.21.15.31:0/4271719260 learned_addr learned my addr 172.21.15.31:0/4271719260 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:01.261 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.550+0000 7f76a4887700 1 -- 172.21.15.31:0/4271719260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a0109dd0 con 0x7f76a0109260 2024-04-26T17:45:01.261 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.551+0000 7f76a4887700 1 --2- 172.21.15.31:0/4271719260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76a0109260 0x7f76a0109680 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f76940098a0 tx=0x7f7694009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9cde31330b9bbfb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:01.261 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.551+0000 7f769f7fe700 1 -- 172.21.15.31:0/4271719260 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7694004160 con 0x7f76a0109260 2024-04-26T17:45:01.261 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.551+0000 7f769f7fe700 1 -- 172.21.15.31:0/4271719260 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7694004300 con 0x7f76a0109260 2024-04-26T17:45:01.261 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.551+0000 7f769f7fe700 1 -- 172.21.15.31:0/4271719260 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7694011af0 con 0x7f76a0109260 2024-04-26T17:45:01.261 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.551+0000 7f76a6aeb700 1 -- 172.21.15.31:0/4271719260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76a0109260 msgr2=0x7f76a0109680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:01.261 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.551+0000 7f76a6aeb700 1 --2- 172.21.15.31:0/4271719260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76a0109260 0x7f76a0109680 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f76940098a0 tx=0x7f7694009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:01.261 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.551+0000 7f76a6aeb700 1 -- 172.21.15.31:0/4271719260 shutdown_connections 2024-04-26T17:45:01.261 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.552+0000 7f76a6aeb700 1 --2- 172.21.15.31:0/4271719260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76a0109260 0x7f76a0109680 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:01.261 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.552+0000 7f76a6aeb700 1 -- 172.21.15.31:0/4271719260 >> 172.21.15.31:0/4271719260 conn(0x7f76a0102a30 msgr2=0x7f76a0104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:01.261 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.552+0000 7f76a6aeb700 1 -- 172.21.15.31:0/4271719260 shutdown_connections 2024-04-26T17:45:01.261 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.552+0000 7f76a6aeb700 1 -- 172.21.15.31:0/4271719260 wait complete. 2024-04-26T17:45:01.262 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.552+0000 7f76a6aeb700 1 Processor -- start 2024-04-26T17:45:01.262 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.552+0000 7f76a6aeb700 1 -- start start 2024-04-26T17:45:01.262 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.553+0000 7f76a6aeb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76a0109260 0x7f76a019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:01.262 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.553+0000 7f76a6aeb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f76a0107b00 con 0x7f76a0109260 2024-04-26T17:45:01.262 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.553+0000 7f76a4887700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76a0109260 0x7f76a019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:01.262 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.553+0000 7f76a4887700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76a0109260 0x7f76a019ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56812/0 (socket says 172.21.15.31:56812) 2024-04-26T17:45:01.262 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.553+0000 7f76a4887700 1 -- 172.21.15.31:0/1480354461 learned_addr learned my addr 172.21.15.31:0/1480354461 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:01.262 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.553+0000 7f76a4887700 1 -- 172.21.15.31:0/1480354461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a019f360 con 0x7f76a0109260 2024-04-26T17:45:01.262 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.553+0000 7f76a4887700 1 --2- 172.21.15.31:0/1480354461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76a0109260 0x7f76a019ee20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7694003f40 tx=0x7f7694011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:01.262 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.553+0000 7f769dffb700 1 -- 172.21.15.31:0/1480354461 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76940038b0 con 0x7f76a0109260 2024-04-26T17:45:01.262 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.553+0000 7f769dffb700 1 -- 172.21.15.31:0/1480354461 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7694003a50 con 0x7f76a0109260 2024-04-26T17:45:01.262 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.553+0000 7f76a6aeb700 1 -- 172.21.15.31:0/1480354461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76a019f650 con 0x7f76a0109260 2024-04-26T17:45:01.262 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.553+0000 7f769dffb700 1 -- 172.21.15.31:0/1480354461 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f769401a5c0 con 0x7f76a0109260 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.554+0000 7f76a6aeb700 1 -- 172.21.15.31:0/1480354461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76a01a2290 con 0x7f76a0109260 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.554+0000 7f769dffb700 1 -- 172.21.15.31:0/1480354461 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f769401ac00 con 0x7f76a0109260 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.554+0000 7f769dffb700 1 -- 172.21.15.31:0/1480354461 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f769400b720 con 0x7f76a0109260 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.555+0000 7f76a6aeb700 1 -- 172.21.15.31:0/1480354461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76a0064a80 con 0x7f76a0109260 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.557+0000 7f769dffb700 1 -- 172.21.15.31:0/1480354461 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7694019e10 con 0x7f76a0109260 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.632+0000 7f76a6aeb700 1 -- 172.21.15.31:0/1480354461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f76a0198380 con 0x7f76a0109260 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.633+0000 7f769dffb700 1 -- 172.21.15.31:0/1480354461 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f769400b9c0 con 0x7f76a0109260 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.634+0000 7f76a6aeb700 1 -- 172.21.15.31:0/1480354461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76a0109260 msgr2=0x7f76a019ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.634+0000 7f76a6aeb700 1 --2- 172.21.15.31:0/1480354461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76a0109260 0x7f76a019ee20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7694003f40 tx=0x7f7694011890 comp rx=0 tx=0).stop 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.634+0000 7f76a6aeb700 1 -- 172.21.15.31:0/1480354461 shutdown_connections 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.634+0000 7f76a6aeb700 1 --2- 172.21.15.31:0/1480354461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76a0109260 0x7f76a019ee20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.634+0000 7f76a6aeb700 1 -- 172.21.15.31:0/1480354461 >> 172.21.15.31:0/1480354461 conn(0x7f76a0102a30 msgr2=0x7f76a0195440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.635+0000 7f76a6aeb700 1 -- 172.21.15.31:0/1480354461 shutdown_connections 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:00.635+0000 7f76a6aeb700 1 -- 172.21.15.31:0/1480354461 wait complete. 2024-04-26T17:45:01.263 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (2/15)... 2024-04-26T17:45:01.505 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:01 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:01.146+0000 7f1f91936200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T17:45:01.505 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:01 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:01.286+0000 7f1f91936200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T17:45:01.505 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:01 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:01.385+0000 7f1f91936200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T17:45:01.505 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:01 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:01.505+0000 7f1f91936200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T17:45:02.309 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:01 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:01.817+0000 7f1f91936200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T17:45:02.723 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:02.447+0000 7f1f91936200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T17:45:02.724 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:02.671+0000 7f1f91936200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T17:45:03.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-mon[29164]: Activating manager daemon x 2024-04-26T17:45:03.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-mon[29164]: mgrmap e2: x(active, starting, since 0.0030093s) 2024-04-26T17:45:03.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-mon[29164]: from='mgr.14100 172.21.15.31:0/187837705' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T17:45:03.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-mon[29164]: from='mgr.14100 172.21.15.31:0/187837705' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T17:45:03.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-mon[29164]: from='mgr.14100 172.21.15.31:0/187837705' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T17:45:03.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-mon[29164]: from='mgr.14100 172.21.15.31:0/187837705' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:45:03.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-mon[29164]: from='mgr.14100 172.21.15.31:0/187837705' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T17:45:03.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-mon[29164]: Manager daemon x is now available 2024-04-26T17:45:03.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-mon[29164]: from='mgr.14100 172.21.15.31:0/187837705' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T17:45:03.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-mon[29164]: from='mgr.14100 172.21.15.31:0/187837705' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T17:45:03.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-mon[29164]: from='mgr.14100 172.21.15.31:0/187837705' entity='mgr.x' 2024-04-26T17:45:03.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-mon[29164]: from='mgr.14100 172.21.15.31:0/187837705' entity='mgr.x' 2024-04-26T17:45:03.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:02 smithi031 ceph-mon[29164]: from='mgr.14100 172.21.15.31:0/187837705' entity='mgr.x' 2024-04-26T17:45:05.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:04 smithi031 ceph-mon[29164]: mgrmap e3: x(active, since 1.00653s) 2024-04-26T17:45:05.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:04 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3568964688' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T17:45:06.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:05 smithi031 ceph-mon[29164]: mgrmap e4: x(active, since 2s) 2024-04-26T17:45:06.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-26T17:45:06.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-26T17:45:06.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "7e53e050-03f4-11ef-bc93-c7b262605968", 2024-04-26T17:45:06.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T17:45:06.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T17:45:06.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T17:45:06.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T17:45:06.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:45:06.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T17:45:06.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T17:45:06.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-04-26T17:45:06.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-26T17:45:06.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T17:45:06.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T17:45:06.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-26T17:45:06.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-26T17:45:06.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T17:45:06.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T17:45:06.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T17:45:06.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T17:45:06.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:45:06.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T17:45:06.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T17:45:06.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T17:45:06.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T17:45:06.234 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T17:45:06.234 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T17:45:06.234 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T17:45:06.234 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T17:45:06.234 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:45:06.234 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T17:45:06.234 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T17:45:06.234 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T17:45:06.234 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T17:45:06.234 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T17:45:06.234 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T17:45:06.234 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T17:45:06.234 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T17:45:06.234 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T17:45:06.235 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:45:06.235 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T17:45:06.235 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T17:45:06.235 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T17:45:06.235 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T17:45:06.235 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:45:06.235 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T17:45:06.235 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T17:45:06.235 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T17:45:06.235 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T17:45:06.235 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T17:45:06.235 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T17:45:06.235 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T17:45:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-26T17:45:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T17:45:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:45:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T17:45:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T17:45:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T17:44:48.618737+0000", 2024-04-26T17:45:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T17:45:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-26T17:45:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T17:45:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-26T17:45:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.674+0000 7f701e229700 1 Processor -- start 2024-04-26T17:45:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.674+0000 7f701e229700 1 -- start start 2024-04-26T17:45:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.674+0000 7f701e229700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7018108e20 0x7f7018109240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.674+0000 7f701e229700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7018109810 con 0x7f7018108e20 2024-04-26T17:45:06.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.674+0000 7f70177fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7018108e20 0x7f7018109240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:06.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.674+0000 7f70177fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7018108e20 0x7f7018109240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56874/0 (socket says 172.21.15.31:56874) 2024-04-26T17:45:06.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.674+0000 7f70177fe700 1 -- 172.21.15.31:0/552974096 learned_addr learned my addr 172.21.15.31:0/552974096 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:06.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.674+0000 7f70177fe700 1 -- 172.21.15.31:0/552974096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7018109f80 con 0x7f7018108e20 2024-04-26T17:45:06.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.675+0000 7f70177fe700 1 --2- 172.21.15.31:0/552974096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7018108e20 0x7f7018109240 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f70080098a0 tx=0x7f7008009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b8d2eb335b6bee5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:06.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.679+0000 7f70167fc700 1 -- 172.21.15.31:0/552974096 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7008004160 con 0x7f7018108e20 2024-04-26T17:45:06.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.679+0000 7f70167fc700 1 -- 172.21.15.31:0/552974096 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7008004300 con 0x7f7018108e20 2024-04-26T17:45:06.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.679+0000 7f70167fc700 1 -- 172.21.15.31:0/552974096 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7008011af0 con 0x7f7018108e20 2024-04-26T17:45:06.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.679+0000 7f701e229700 1 -- 172.21.15.31:0/552974096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7018108e20 msgr2=0x7f7018109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:06.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.679+0000 7f701e229700 1 --2- 172.21.15.31:0/552974096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7018108e20 0x7f7018109240 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f70080098a0 tx=0x7f7008009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:06.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.680+0000 7f701e229700 1 -- 172.21.15.31:0/552974096 shutdown_connections 2024-04-26T17:45:06.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.680+0000 7f701e229700 1 --2- 172.21.15.31:0/552974096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7018108e20 0x7f7018109240 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:06.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.680+0000 7f701e229700 1 -- 172.21.15.31:0/552974096 >> 172.21.15.31:0/552974096 conn(0x7f7018104400 msgr2=0x7f7018106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:06.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.680+0000 7f701e229700 1 -- 172.21.15.31:0/552974096 shutdown_connections 2024-04-26T17:45:06.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.680+0000 7f701e229700 1 -- 172.21.15.31:0/552974096 wait complete. 2024-04-26T17:45:06.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.680+0000 7f701e229700 1 Processor -- start 2024-04-26T17:45:06.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.681+0000 7f701e229700 1 -- start start 2024-04-26T17:45:06.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.681+0000 7f701e229700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7018108e20 0x7f701819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:06.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.681+0000 7f701e229700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f701810d330 con 0x7f7018108e20 2024-04-26T17:45:06.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.681+0000 7f70177fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7018108e20 0x7f701819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:06.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.681+0000 7f70177fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7018108e20 0x7f701819cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56880/0 (socket says 172.21.15.31:56880) 2024-04-26T17:45:06.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.681+0000 7f70177fe700 1 -- 172.21.15.31:0/3568964688 learned_addr learned my addr 172.21.15.31:0/3568964688 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:06.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.681+0000 7f70177fe700 1 -- 172.21.15.31:0/3568964688 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701819d250 con 0x7f7018108e20 2024-04-26T17:45:06.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.681+0000 7f70177fe700 1 --2- 172.21.15.31:0/3568964688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7018108e20 0x7f701819cd10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7008009fb0 tx=0x7f7008011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:06.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.681+0000 7f7014ff9700 1 -- 172.21.15.31:0/3568964688 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70080036a0 con 0x7f7018108e20 2024-04-26T17:45:06.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.681+0000 7f7014ff9700 1 -- 172.21.15.31:0/3568964688 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f700801a430 con 0x7f7018108e20 2024-04-26T17:45:06.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.682+0000 7f701e229700 1 -- 172.21.15.31:0/3568964688 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f701819d4e0 con 0x7f7018108e20 2024-04-26T17:45:06.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.682+0000 7f7014ff9700 1 -- 172.21.15.31:0/3568964688 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7008019550 con 0x7f7018108e20 2024-04-26T17:45:06.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.682+0000 7f701e229700 1 -- 172.21.15.31:0/3568964688 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70181a01c0 con 0x7f7018108e20 2024-04-26T17:45:06.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.683+0000 7f7014ff9700 1 -- 172.21.15.31:0/3568964688 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 3) ==== 46441+0+0 (secure 0 0 0) 0x7f7008019850 con 0x7f7018108e20 2024-04-26T17:45:06.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.683+0000 7f7014ff9700 1 --2- 172.21.15.31:0/3568964688 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f70000393e0 0x7f700003b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:06.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.683+0000 7f7014ff9700 1 -- 172.21.15.31:0/3568964688 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f700804d6c0 con 0x7f7018108e20 2024-04-26T17:45:06.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.683+0000 7f701e229700 1 -- 172.21.15.31:0/3568964688 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7018064a80 con 0x7f7018108e20 2024-04-26T17:45:06.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.683+0000 7f7016ffd700 1 --2- 172.21.15.31:0/3568964688 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f70000393e0 0x7f700003b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:06.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.683+0000 7f7016ffd700 1 --2- 172.21.15.31:0/3568964688 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f70000393e0 0x7f700003b8a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f700c006fd0 tx=0x7f700c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:06.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.689+0000 7f7014ff9700 1 -- 172.21.15.31:0/3568964688 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f700800fda0 con 0x7f7018108e20 2024-04-26T17:45:06.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.991+0000 7f701e229700 1 -- 172.21.15.31:0/3568964688 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f70181a0430 con 0x7f7018108e20 2024-04-26T17:45:06.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.991+0000 7f7014ff9700 1 -- 172.21.15.31:0/3568964688 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f700801a5e0 con 0x7f7018108e20 2024-04-26T17:45:06.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.993+0000 7f701e229700 1 -- 172.21.15.31:0/3568964688 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f70000393e0 msgr2=0x7f700003b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:06.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.994+0000 7f701e229700 1 --2- 172.21.15.31:0/3568964688 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f70000393e0 0x7f700003b8a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f700c006fd0 tx=0x7f700c006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:06.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.994+0000 7f701e229700 1 -- 172.21.15.31:0/3568964688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7018108e20 msgr2=0x7f701819cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:06.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.994+0000 7f701e229700 1 --2- 172.21.15.31:0/3568964688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7018108e20 0x7f701819cd10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7008009fb0 tx=0x7f7008011940 comp rx=0 tx=0).stop 2024-04-26T17:45:06.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.994+0000 7f701e229700 1 -- 172.21.15.31:0/3568964688 shutdown_connections 2024-04-26T17:45:06.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.994+0000 7f701e229700 1 --2- 172.21.15.31:0/3568964688 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f70000393e0 0x7f700003b8a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:06.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.994+0000 7f701e229700 1 --2- 172.21.15.31:0/3568964688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7018108e20 0x7f701819cd10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:06.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.994+0000 7f701e229700 1 -- 172.21.15.31:0/3568964688 >> 172.21.15.31:0/3568964688 conn(0x7f7018104400 msgr2=0x7f7018104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:06.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.994+0000 7f701e229700 1 -- 172.21.15.31:0/3568964688 shutdown_connections 2024-04-26T17:45:06.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:03.994+0000 7f701e229700 1 -- 172.21.15.31:0/3568964688 wait complete. 2024-04-26T17:45:06.240 INFO:teuthology.orchestra.run.smithi031.stdout:mgr is available 2024-04-26T17:45:09.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:09 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2296409518' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T17:45:13.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-26T17:45:13.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T17:45:13.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout fsid = 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:45:13.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T17:45:13.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T17:45:13.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T17:45:13.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T17:45:13.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T17:45:13.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-26T17:45:13.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T17:45:13.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T17:45:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-26T17:45:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T17:45:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T17:45:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T17:45:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T17:45:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T17:45:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T17:45:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.230+0000 7f54fe57c700 1 Processor -- start 2024-04-26T17:45:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.230+0000 7f54fe57c700 1 -- start start 2024-04-26T17:45:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.230+0000 7f54fe57c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54f8108e00 0x7f54f8109220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.230+0000 7f54fe57c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f54f81097f0 con 0x7f54f8108e00 2024-04-26T17:45:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.231+0000 7f54f7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54f8108e00 0x7f54f8109220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.231+0000 7f54f7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54f8108e00 0x7f54f8109220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35862/0 (socket says 172.21.15.31:35862) 2024-04-26T17:45:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.231+0000 7f54f7fff700 1 -- 172.21.15.31:0/1897039729 learned_addr learned my addr 172.21.15.31:0/1897039729 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:13.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.231+0000 7f54f7fff700 1 -- 172.21.15.31:0/1897039729 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f8109f60 con 0x7f54f8108e00 2024-04-26T17:45:13.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.231+0000 7f54f7fff700 1 --2- 172.21.15.31:0/1897039729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54f8108e00 0x7f54f8109220 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f54e80098a0 tx=0x7f54e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e61f92972d2762d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:13.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.231+0000 7f54f6ffd700 1 -- 172.21.15.31:0/1897039729 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54e8004160 con 0x7f54f8108e00 2024-04-26T17:45:13.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.231+0000 7f54f6ffd700 1 -- 172.21.15.31:0/1897039729 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f54e8004300 con 0x7f54f8108e00 2024-04-26T17:45:13.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.231+0000 7f54f6ffd700 1 -- 172.21.15.31:0/1897039729 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54e8011af0 con 0x7f54f8108e00 2024-04-26T17:45:13.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.232+0000 7f54fe57c700 1 -- 172.21.15.31:0/1897039729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54f8108e00 msgr2=0x7f54f8109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:13.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.232+0000 7f54fe57c700 1 --2- 172.21.15.31:0/1897039729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54f8108e00 0x7f54f8109220 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f54e80098a0 tx=0x7f54e8009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:13.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.232+0000 7f54fe57c700 1 -- 172.21.15.31:0/1897039729 shutdown_connections 2024-04-26T17:45:13.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.232+0000 7f54fe57c700 1 --2- 172.21.15.31:0/1897039729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54f8108e00 0x7f54f8109220 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:13.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.232+0000 7f54fe57c700 1 -- 172.21.15.31:0/1897039729 >> 172.21.15.31:0/1897039729 conn(0x7f54f81043c0 msgr2=0x7f54f81067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:13.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.232+0000 7f54fe57c700 1 -- 172.21.15.31:0/1897039729 shutdown_connections 2024-04-26T17:45:13.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.232+0000 7f54fe57c700 1 -- 172.21.15.31:0/1897039729 wait complete. 2024-04-26T17:45:13.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.233+0000 7f54fe57c700 1 Processor -- start 2024-04-26T17:45:13.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.233+0000 7f54fe57c700 1 -- start start 2024-04-26T17:45:13.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.233+0000 7f54fe57c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54f8108e00 0x7f54f819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:13.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.233+0000 7f54fe57c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f54f810d310 con 0x7f54f8108e00 2024-04-26T17:45:13.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.233+0000 7f54f7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54f8108e00 0x7f54f819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:13.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.233+0000 7f54f7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54f8108e00 0x7f54f819cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35878/0 (socket says 172.21.15.31:35878) 2024-04-26T17:45:13.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.233+0000 7f54f7fff700 1 -- 172.21.15.31:0/2296409518 learned_addr learned my addr 172.21.15.31:0/2296409518 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:13.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.233+0000 7f54f7fff700 1 -- 172.21.15.31:0/2296409518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f819d250 con 0x7f54f8108e00 2024-04-26T17:45:13.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.234+0000 7f54f7fff700 1 --2- 172.21.15.31:0/2296409518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54f8108e00 0x7f54f819cd10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f54e8009fb0 tx=0x7f54e8011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:13.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.234+0000 7f54f57fa700 1 -- 172.21.15.31:0/2296409518 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54e80036a0 con 0x7f54f8108e00 2024-04-26T17:45:13.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.234+0000 7f54f57fa700 1 -- 172.21.15.31:0/2296409518 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f54e801a430 con 0x7f54f8108e00 2024-04-26T17:45:13.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.234+0000 7f54fe57c700 1 -- 172.21.15.31:0/2296409518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54f819d4e0 con 0x7f54f8108e00 2024-04-26T17:45:13.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.234+0000 7f54f57fa700 1 -- 172.21.15.31:0/2296409518 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54e8019550 con 0x7f54f8108e00 2024-04-26T17:45:13.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.234+0000 7f54fe57c700 1 -- 172.21.15.31:0/2296409518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54f81a01c0 con 0x7f54f8108e00 2024-04-26T17:45:13.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.235+0000 7f54f57fa700 1 -- 172.21.15.31:0/2296409518 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f54e8019800 con 0x7f54f8108e00 2024-04-26T17:45:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.235+0000 7f54f57fa700 1 --2- 172.21.15.31:0/2296409518 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f54e4039560 0x7f54e403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.235+0000 7f54f57fa700 1 -- 172.21.15.31:0/2296409518 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f54e804e820 con 0x7f54f8108e00 2024-04-26T17:45:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.235+0000 7f54f77fe700 1 --2- 172.21.15.31:0/2296409518 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f54e4039560 0x7f54e403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.235+0000 7f54fe57c700 1 -- 172.21.15.31:0/2296409518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54f8064a80 con 0x7f54f8108e00 2024-04-26T17:45:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.235+0000 7f54f77fe700 1 --2- 172.21.15.31:0/2296409518 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f54e4039560 0x7f54e403ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f54e0006fd0 tx=0x7f54e0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.240+0000 7f54f57fa700 1 -- 172.21.15.31:0/2296409518 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f54e8023c90 con 0x7f54f8108e00 2024-04-26T17:45:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.452+0000 7f54fe57c700 1 -- 172.21.15.31:0/2296409518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f54f81a0390 con 0x7f54f8108e00 2024-04-26T17:45:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.452+0000 7f54f57fa700 1 -- 172.21.15.31:0/2296409518 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f54e804e240 con 0x7f54f8108e00 2024-04-26T17:45:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.454+0000 7f54fe57c700 1 -- 172.21.15.31:0/2296409518 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f54e4039560 msgr2=0x7f54e403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.454+0000 7f54fe57c700 1 --2- 172.21.15.31:0/2296409518 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f54e4039560 0x7f54e403ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f54e0006fd0 tx=0x7f54e0006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.454+0000 7f54fe57c700 1 -- 172.21.15.31:0/2296409518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54f8108e00 msgr2=0x7f54f819cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.454+0000 7f54fe57c700 1 --2- 172.21.15.31:0/2296409518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54f8108e00 0x7f54f819cd10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f54e8009fb0 tx=0x7f54e8011940 comp rx=0 tx=0).stop 2024-04-26T17:45:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.455+0000 7f54fe57c700 1 -- 172.21.15.31:0/2296409518 shutdown_connections 2024-04-26T17:45:13.241 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.455+0000 7f54fe57c700 1 --2- 172.21.15.31:0/2296409518 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f54e4039560 0x7f54e403ba20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:13.241 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.455+0000 7f54fe57c700 1 --2- 172.21.15.31:0/2296409518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54f8108e00 0x7f54f819cd10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:13.241 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.455+0000 7f54fe57c700 1 -- 172.21.15.31:0/2296409518 >> 172.21.15.31:0/2296409518 conn(0x7f54f81043c0 msgr2=0x7f54f8104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:13.241 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.455+0000 7f54fe57c700 1 -- 172.21.15.31:0/2296409518 shutdown_connections 2024-04-26T17:45:13.241 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:09.455+0000 7f54fe57c700 1 -- 172.21.15.31:0/2296409518 wait complete. 2024-04-26T17:45:13.241 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling cephadm module... 2024-04-26T17:45:14.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:13 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3201900392' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T17:45:14.309 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:14 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: ignoring --setuser ceph since I am not root 2024-04-26T17:45:14.309 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:14 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: ignoring --setgroup ceph since I am not root 2024-04-26T17:45:14.310 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:14 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:14.147+0000 7f474478f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T17:45:14.540 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.640+0000 7f2aecb45700 1 Processor -- start 2024-04-26T17:45:14.541 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.640+0000 7f2aecb45700 1 -- start start 2024-04-26T17:45:14.541 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.641+0000 7f2aecb45700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ae8108e00 0x7f2ae8109220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:14.541 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.641+0000 7f2aecb45700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae81097f0 con 0x7f2ae8108e00 2024-04-26T17:45:14.541 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.641+0000 7f2ae659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ae8108e00 0x7f2ae8109220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:14.541 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.641+0000 7f2ae659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ae8108e00 0x7f2ae8109220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35884/0 (socket says 172.21.15.31:35884) 2024-04-26T17:45:14.541 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.641+0000 7f2ae659c700 1 -- 172.21.15.31:0/1328808551 learned_addr learned my addr 172.21.15.31:0/1328808551 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:14.541 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.641+0000 7f2ae659c700 1 -- 172.21.15.31:0/1328808551 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae8109f60 con 0x7f2ae8108e00 2024-04-26T17:45:14.541 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.641+0000 7f2ae659c700 1 --2- 172.21.15.31:0/1328808551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ae8108e00 0x7f2ae8109220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2ad80098a0 tx=0x7f2ad8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b7a4aa165c53e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:14.541 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.641+0000 7f2ae559a700 1 -- 172.21.15.31:0/1328808551 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ad8004160 con 0x7f2ae8108e00 2024-04-26T17:45:14.541 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.641+0000 7f2ae559a700 1 -- 172.21.15.31:0/1328808551 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2ad8004300 con 0x7f2ae8108e00 2024-04-26T17:45:14.541 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.641+0000 7f2ae559a700 1 -- 172.21.15.31:0/1328808551 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ad8011af0 con 0x7f2ae8108e00 2024-04-26T17:45:14.542 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.642+0000 7f2aecb45700 1 -- 172.21.15.31:0/1328808551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ae8108e00 msgr2=0x7f2ae8109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:14.542 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.642+0000 7f2aecb45700 1 --2- 172.21.15.31:0/1328808551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ae8108e00 0x7f2ae8109220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2ad80098a0 tx=0x7f2ad8009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:14.542 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.642+0000 7f2aecb45700 1 -- 172.21.15.31:0/1328808551 shutdown_connections 2024-04-26T17:45:14.542 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.642+0000 7f2aecb45700 1 --2- 172.21.15.31:0/1328808551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ae8108e00 0x7f2ae8109220 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:14.542 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.642+0000 7f2aecb45700 1 -- 172.21.15.31:0/1328808551 >> 172.21.15.31:0/1328808551 conn(0x7f2ae81043c0 msgr2=0x7f2ae81067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:14.542 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.642+0000 7f2aecb45700 1 -- 172.21.15.31:0/1328808551 shutdown_connections 2024-04-26T17:45:14.542 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.642+0000 7f2aecb45700 1 -- 172.21.15.31:0/1328808551 wait complete. 2024-04-26T17:45:14.542 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.643+0000 7f2aecb45700 1 Processor -- start 2024-04-26T17:45:14.542 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.643+0000 7f2aecb45700 1 -- start start 2024-04-26T17:45:14.542 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.643+0000 7f2aecb45700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ae8108e00 0x7f2ae819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:14.542 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.643+0000 7f2aecb45700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae810d310 con 0x7f2ae8108e00 2024-04-26T17:45:14.542 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.643+0000 7f2ae659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ae8108e00 0x7f2ae819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:14.542 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.643+0000 7f2ae659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ae8108e00 0x7f2ae819cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35900/0 (socket says 172.21.15.31:35900) 2024-04-26T17:45:14.543 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.643+0000 7f2ae659c700 1 -- 172.21.15.31:0/3201900392 learned_addr learned my addr 172.21.15.31:0/3201900392 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:14.543 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.643+0000 7f2ae659c700 1 -- 172.21.15.31:0/3201900392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae819d250 con 0x7f2ae8108e00 2024-04-26T17:45:14.543 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.644+0000 7f2ae659c700 1 --2- 172.21.15.31:0/3201900392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ae8108e00 0x7f2ae819cd10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2ad80117b0 tx=0x7f2ad8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:14.543 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.644+0000 7f2ad77fe700 1 -- 172.21.15.31:0/3201900392 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ad8003af0 con 0x7f2ae8108e00 2024-04-26T17:45:14.543 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.644+0000 7f2aecb45700 1 -- 172.21.15.31:0/3201900392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae819d4e0 con 0x7f2ae8108e00 2024-04-26T17:45:14.543 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.644+0000 7f2ad77fe700 1 -- 172.21.15.31:0/3201900392 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2ad8020070 con 0x7f2ae8108e00 2024-04-26T17:45:14.543 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.644+0000 7f2ad77fe700 1 -- 172.21.15.31:0/3201900392 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ad8019630 con 0x7f2ae8108e00 2024-04-26T17:45:14.543 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.644+0000 7f2aecb45700 1 -- 172.21.15.31:0/3201900392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae81a01c0 con 0x7f2ae8108e00 2024-04-26T17:45:14.543 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.645+0000 7f2ad77fe700 1 -- 172.21.15.31:0/3201900392 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f2ad80198f0 con 0x7f2ae8108e00 2024-04-26T17:45:14.543 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.645+0000 7f2ad77fe700 1 --2- 172.21.15.31:0/3201900392 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f2ad0039560 0x7f2ad003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:14.543 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.645+0000 7f2ad77fe700 1 -- 172.21.15.31:0/3201900392 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2ad804db30 con 0x7f2ae8108e00 2024-04-26T17:45:14.543 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.645+0000 7f2aecb45700 1 -- 172.21.15.31:0/3201900392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ae8064a80 con 0x7f2ae8108e00 2024-04-26T17:45:14.543 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.645+0000 7f2ae5d9b700 1 --2- 172.21.15.31:0/3201900392 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f2ad0039560 0x7f2ad003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:14.544 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.645+0000 7f2ae5d9b700 1 --2- 172.21.15.31:0/3201900392 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f2ad0039560 0x7f2ad003ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2adc006fd0 tx=0x7f2adc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:14.544 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.650+0000 7f2ad77fe700 1 -- 172.21.15.31:0/3201900392 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ad80042a0 con 0x7f2ae8108e00 2024-04-26T17:45:14.544 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.921+0000 7f2aecb45700 1 -- 172.21.15.31:0/3201900392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f2ae81a0430 con 0x7f2ae8108e00 2024-04-26T17:45:14.544 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.972+0000 7f2ad77fe700 1 -- 172.21.15.31:0/3201900392 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f2ad804d550 con 0x7f2ae8108e00 2024-04-26T17:45:14.544 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.974+0000 7f2ad77fe700 1 -- 172.21.15.31:0/3201900392 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f2ad800b430 con 0x7f2ae8108e00 2024-04-26T17:45:14.544 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.975+0000 7f2aecb45700 1 -- 172.21.15.31:0/3201900392 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f2ad0039560 msgr2=0x7f2ad003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:14.544 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.975+0000 7f2aecb45700 1 --2- 172.21.15.31:0/3201900392 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f2ad0039560 0x7f2ad003ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2adc006fd0 tx=0x7f2adc006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:14.544 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.975+0000 7f2aecb45700 1 -- 172.21.15.31:0/3201900392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ae8108e00 msgr2=0x7f2ae819cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:14.544 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.975+0000 7f2aecb45700 1 --2- 172.21.15.31:0/3201900392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ae8108e00 0x7f2ae819cd10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2ad80117b0 tx=0x7f2ad8011b70 comp rx=0 tx=0).stop 2024-04-26T17:45:14.544 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.975+0000 7f2aecb45700 1 -- 172.21.15.31:0/3201900392 shutdown_connections 2024-04-26T17:45:14.544 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.975+0000 7f2aecb45700 1 --2- 172.21.15.31:0/3201900392 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f2ad0039560 0x7f2ad003ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:14.545 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.975+0000 7f2aecb45700 1 --2- 172.21.15.31:0/3201900392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ae8108e00 0x7f2ae819cd10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:14.545 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.975+0000 7f2aecb45700 1 -- 172.21.15.31:0/3201900392 >> 172.21.15.31:0/3201900392 conn(0x7f2ae81043c0 msgr2=0x7f2ae8104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:14.545 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.975+0000 7f2aecb45700 1 -- 172.21.15.31:0/3201900392 shutdown_connections 2024-04-26T17:45:14.545 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:13.976+0000 7f2aecb45700 1 -- 172.21.15.31:0/3201900392 wait complete. 2024-04-26T17:45:14.660 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:14 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:14.337+0000 7f474478f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:45:14.660 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:14 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:14.445+0000 7f474478f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T17:45:14.976 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:14 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:14.785+0000 7f474478f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T17:45:14.976 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:14 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:14.876+0000 7f474478f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T17:45:15.280 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:14 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3201900392' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T17:45:15.280 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:14 smithi031 ceph-mon[29164]: mgrmap e5: x(active, since 11s) 2024-04-26T17:45:15.815 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-26T17:45:15.815 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-26T17:45:15.815 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T17:45:15.815 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T17:45:15.816 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T17:45:15.816 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-26T17:45:15.816 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.995+0000 7f659f4c0700 1 Processor -- start 2024-04-26T17:45:15.816 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.995+0000 7f659f4c0700 1 -- start start 2024-04-26T17:45:15.816 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.995+0000 7f659f4c0700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6598107c10 0x7f6598108030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:15.816 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.995+0000 7f659f4c0700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6598108600 con 0x7f6598107c10 2024-04-26T17:45:15.817 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.995+0000 7f659d25c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6598107c10 0x7f6598108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:15.817 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.995+0000 7f659d25c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6598107c10 0x7f6598108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54884/0 (socket says 172.21.15.31:54884) 2024-04-26T17:45:15.817 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.995+0000 7f659d25c700 1 -- 172.21.15.31:0/1715047501 learned_addr learned my addr 172.21.15.31:0/1715047501 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:15.817 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.996+0000 7f659d25c700 1 -- 172.21.15.31:0/1715047501 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6598108e10 con 0x7f6598107c10 2024-04-26T17:45:15.817 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.996+0000 7f659d25c700 1 --2- 172.21.15.31:0/1715047501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6598107c10 0x7f6598108030 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f65940098a0 tx=0x7f6594009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4f13af19f1a1b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:15.818 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.996+0000 7f658ffff700 1 -- 172.21.15.31:0/1715047501 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6594004160 con 0x7f6598107c10 2024-04-26T17:45:15.818 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.996+0000 7f658ffff700 1 -- 172.21.15.31:0/1715047501 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6594004300 con 0x7f6598107c10 2024-04-26T17:45:15.818 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.996+0000 7f658ffff700 1 -- 172.21.15.31:0/1715047501 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6594011af0 con 0x7f6598107c10 2024-04-26T17:45:15.818 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.997+0000 7f659f4c0700 1 -- 172.21.15.31:0/1715047501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6598107c10 msgr2=0x7f6598108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:15.818 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.997+0000 7f659f4c0700 1 --2- 172.21.15.31:0/1715047501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6598107c10 0x7f6598108030 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f65940098a0 tx=0x7f6594009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:15.818 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.997+0000 7f659f4c0700 1 -- 172.21.15.31:0/1715047501 shutdown_connections 2024-04-26T17:45:15.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.997+0000 7f659f4c0700 1 --2- 172.21.15.31:0/1715047501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6598107c10 0x7f6598108030 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:15.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.997+0000 7f659f4c0700 1 -- 172.21.15.31:0/1715047501 >> 172.21.15.31:0/1715047501 conn(0x7f6598103210 msgr2=0x7f6598105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:15.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.997+0000 7f659f4c0700 1 -- 172.21.15.31:0/1715047501 shutdown_connections 2024-04-26T17:45:15.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.997+0000 7f659f4c0700 1 -- 172.21.15.31:0/1715047501 wait complete. 2024-04-26T17:45:15.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.998+0000 7f659f4c0700 1 Processor -- start 2024-04-26T17:45:15.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.998+0000 7f659f4c0700 1 -- start start 2024-04-26T17:45:15.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.998+0000 7f659f4c0700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6598107c10 0x7f659807ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:15.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.998+0000 7f659f4c0700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f659810c1c0 con 0x7f6598107c10 2024-04-26T17:45:15.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.998+0000 7f659d25c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6598107c10 0x7f659807ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:15.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.998+0000 7f659d25c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6598107c10 0x7f659807ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54896/0 (socket says 172.21.15.31:54896) 2024-04-26T17:45:15.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.998+0000 7f659d25c700 1 -- 172.21.15.31:0/2135494377 learned_addr learned my addr 172.21.15.31:0/2135494377 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:15.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.999+0000 7f659d25c700 1 -- 172.21.15.31:0/2135494377 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f659819fa70 con 0x7f6598107c10 2024-04-26T17:45:15.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.999+0000 7f659d25c700 1 --2- 172.21.15.31:0/2135494377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6598107c10 0x7f659807ca70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6594003f40 tx=0x7f6594011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:15.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.999+0000 7f658e7fc700 1 -- 172.21.15.31:0/2135494377 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65940038b0 con 0x7f6598107c10 2024-04-26T17:45:15.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.999+0000 7f658e7fc700 1 -- 172.21.15.31:0/2135494377 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6594003a50 con 0x7f6598107c10 2024-04-26T17:45:15.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.999+0000 7f659f4c0700 1 -- 172.21.15.31:0/2135494377 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f659819fc40 con 0x7f6598107c10 2024-04-26T17:45:15.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.999+0000 7f658e7fc700 1 -- 172.21.15.31:0/2135494377 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f659401a5c0 con 0x7f6598107c10 2024-04-26T17:45:15.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:14.999+0000 7f659f4c0700 1 -- 172.21.15.31:0/2135494377 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f659819ffe0 con 0x7f6598107c10 2024-04-26T17:45:15.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.000+0000 7f659f4c0700 1 -- 172.21.15.31:0/2135494377 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6598064a80 con 0x7f6598107c10 2024-04-26T17:45:15.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.000+0000 7f658e7fc700 1 -- 172.21.15.31:0/2135494377 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f659401a870 con 0x7f6598107c10 2024-04-26T17:45:15.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.000+0000 7f658e7fc700 1 --2- 172.21.15.31:0/2135494377 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f65840395b0 0x7f658403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:15.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.000+0000 7f658e7fc700 1 -- 172.21.15.31:0/2135494377 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f659404da80 con 0x7f6598107c10 2024-04-26T17:45:15.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.000+0000 7f659ca5b700 1 -- 172.21.15.31:0/2135494377 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f65840395b0 msgr2=0x7f658403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/545710545 2024-04-26T17:45:15.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.000+0000 7f659ca5b700 1 --2- 172.21.15.31:0/2135494377 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f65840395b0 0x7f658403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T17:45:15.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.005+0000 7f658e7fc700 1 -- 172.21.15.31:0/2135494377 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6594019c70 con 0x7f6598107c10 2024-04-26T17:45:15.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.201+0000 7f659ca5b700 1 -- 172.21.15.31:0/2135494377 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f65840395b0 msgr2=0x7f658403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/545710545 2024-04-26T17:45:15.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.201+0000 7f659ca5b700 1 --2- 172.21.15.31:0/2135494377 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f65840395b0 0x7f658403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T17:45:15.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.280+0000 7f659f4c0700 1 -- 172.21.15.31:0/2135494377 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f65981a0400 con 0x7f6598107c10 2024-04-26T17:45:15.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.281+0000 7f658e7fc700 1 -- 172.21.15.31:0/2135494377 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f65981a0400 con 0x7f6598107c10 2024-04-26T17:45:15.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.285+0000 7f659f4c0700 1 -- 172.21.15.31:0/2135494377 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f65840395b0 msgr2=0x7f658403ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T17:45:15.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.285+0000 7f659f4c0700 1 --2- 172.21.15.31:0/2135494377 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f65840395b0 0x7f658403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:15.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.285+0000 7f659f4c0700 1 -- 172.21.15.31:0/2135494377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6598107c10 msgr2=0x7f659807ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:15.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.285+0000 7f659f4c0700 1 --2- 172.21.15.31:0/2135494377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6598107c10 0x7f659807ca70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6594003f40 tx=0x7f6594011890 comp rx=0 tx=0).stop 2024-04-26T17:45:15.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.285+0000 7f659f4c0700 1 -- 172.21.15.31:0/2135494377 shutdown_connections 2024-04-26T17:45:15.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.285+0000 7f659f4c0700 1 --2- 172.21.15.31:0/2135494377 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7f65840395b0 0x7f658403ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:15.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.285+0000 7f659f4c0700 1 --2- 172.21.15.31:0/2135494377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6598107c10 0x7f659807ca70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:15.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.285+0000 7f659f4c0700 1 -- 172.21.15.31:0/2135494377 >> 172.21.15.31:0/2135494377 conn(0x7f6598103210 msgr2=0x7f6598103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:15.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.285+0000 7f659f4c0700 1 -- 172.21.15.31:0/2135494377 shutdown_connections 2024-04-26T17:45:15.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:15.285+0000 7f659f4c0700 1 -- 172.21.15.31:0/2135494377 wait complete. 2024-04-26T17:45:15.825 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for the mgr to restart... 2024-04-26T17:45:15.826 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr epoch 5... 2024-04-26T17:45:16.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:15 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2135494377' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T17:45:16.182 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:16 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:16.166+0000 7f474478f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T17:45:16.523 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:16 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:16.346+0000 7f474478f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T17:45:16.523 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:16 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:16.523+0000 7f474478f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T17:45:17.059 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:16 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:16.826+0000 7f474478f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T17:45:17.059 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:16 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:16.922+0000 7f474478f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T17:45:17.559 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:17 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:17.134+0000 7f474478f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T17:45:17.559 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:17 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:17.334+0000 7f474478f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:45:18.309 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:17 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:17.964+0000 7f474478f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T17:45:18.711 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:18 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:18.401+0000 7f474478f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T17:45:18.712 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:18 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:18.502+0000 7f474478f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T17:45:18.712 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:18 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:18.711+0000 7f474478f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T17:45:19.559 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:19 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:19.256+0000 7f474478f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T17:45:19.559 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:19 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:19.352+0000 7f474478f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T17:45:19.962 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:19 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:19.605+0000 7f474478f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T17:45:19.962 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:19 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:19.743+0000 7f474478f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T17:45:19.962 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:19 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:19.842+0000 7f474478f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T17:45:19.962 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:19 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:19.962+0000 7f474478f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T17:45:20.559 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:20 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:20.273+0000 7f474478f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T17:45:21.179 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:20 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:20.901+0000 7f474478f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T17:45:21.179 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:21.126+0000 7f474478f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T17:45:21.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: Active manager daemon x restarted 2024-04-26T17:45:21.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: Activating manager daemon x 2024-04-26T17:45:21.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: osdmap e2: 0 total, 0 up, 0 in 2024-04-26T17:45:21.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: mgrmap e6: x(active, starting, since 0.00351377s) 2024-04-26T17:45:21.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:45:21.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T17:45:21.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T17:45:21.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T17:45:21.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T17:45:21.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: Manager daemon x is now available 2024-04-26T17:45:21.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' 2024-04-26T17:45:21.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' 2024-04-26T17:45:21.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:45:21.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:45:21.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:21 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T17:45:22.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:22 smithi031 ceph-mon[29164]: Found migration_current of "None". Setting to last migration. 2024-04-26T17:45:22.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:22 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T17:45:22.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:22 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' 2024-04-26T17:45:22.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:22 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' 2024-04-26T17:45:22.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:22 smithi031 ceph-mon[29164]: mgrmap e7: x(active, since 1.00554s) 2024-04-26T17:45:22.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-26T17:45:22.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-26T17:45:22.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T17:45:22.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-26T17:45:22.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.202+0000 7fc55b389700 1 Processor -- start 2024-04-26T17:45:22.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.202+0000 7fc55b389700 1 -- start start 2024-04-26T17:45:22.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.202+0000 7fc55b389700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc554108e20 0x7fc554109240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:22.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.202+0000 7fc55b389700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc554109810 con 0x7fc554108e20 2024-04-26T17:45:22.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.202+0000 7fc559125700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc554108e20 0x7fc554109240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:22.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.202+0000 7fc559125700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc554108e20 0x7fc554109240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54898/0 (socket says 172.21.15.31:54898) 2024-04-26T17:45:22.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.202+0000 7fc559125700 1 -- 172.21.15.31:0/3861661155 learned_addr learned my addr 172.21.15.31:0/3861661155 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:22.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.203+0000 7fc559125700 1 -- 172.21.15.31:0/3861661155 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc554109f80 con 0x7fc554108e20 2024-04-26T17:45:22.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.203+0000 7fc559125700 1 --2- 172.21.15.31:0/3861661155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc554108e20 0x7fc554109240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc5500098a0 tx=0x7fc550009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d6149afe70ae8f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:22.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.203+0000 7fc54bfff700 1 -- 172.21.15.31:0/3861661155 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc550004160 con 0x7fc554108e20 2024-04-26T17:45:22.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.203+0000 7fc54bfff700 1 -- 172.21.15.31:0/3861661155 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc550004300 con 0x7fc554108e20 2024-04-26T17:45:22.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.203+0000 7fc54bfff700 1 -- 172.21.15.31:0/3861661155 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc550011af0 con 0x7fc554108e20 2024-04-26T17:45:22.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.204+0000 7fc55b389700 1 -- 172.21.15.31:0/3861661155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc554108e20 msgr2=0x7fc554109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:22.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.204+0000 7fc55b389700 1 --2- 172.21.15.31:0/3861661155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc554108e20 0x7fc554109240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc5500098a0 tx=0x7fc550009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:22.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.204+0000 7fc55b389700 1 -- 172.21.15.31:0/3861661155 shutdown_connections 2024-04-26T17:45:22.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.204+0000 7fc55b389700 1 --2- 172.21.15.31:0/3861661155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc554108e20 0x7fc554109240 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:22.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.204+0000 7fc55b389700 1 -- 172.21.15.31:0/3861661155 >> 172.21.15.31:0/3861661155 conn(0x7fc554104400 msgr2=0x7fc554106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:22.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.204+0000 7fc55b389700 1 -- 172.21.15.31:0/3861661155 shutdown_connections 2024-04-26T17:45:22.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.204+0000 7fc55b389700 1 -- 172.21.15.31:0/3861661155 wait complete. 2024-04-26T17:45:22.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.204+0000 7fc55b389700 1 Processor -- start 2024-04-26T17:45:22.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.204+0000 7fc55b389700 1 -- start start 2024-04-26T17:45:22.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.205+0000 7fc55b389700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc554108e20 0x7fc55419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:22.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.205+0000 7fc55b389700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc55410d330 con 0x7fc554108e20 2024-04-26T17:45:22.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.205+0000 7fc559125700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc554108e20 0x7fc55419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:22.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.205+0000 7fc559125700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc554108e20 0x7fc55419cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54904/0 (socket says 172.21.15.31:54904) 2024-04-26T17:45:22.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.205+0000 7fc559125700 1 -- 172.21.15.31:0/815459771 learned_addr learned my addr 172.21.15.31:0/815459771 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:22.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.205+0000 7fc559125700 1 -- 172.21.15.31:0/815459771 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc55419d250 con 0x7fc554108e20 2024-04-26T17:45:22.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.205+0000 7fc559125700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc554108e20 0x7fc55419cd10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc550009fb0 tx=0x7fc550011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:22.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.205+0000 7fc54a7fc700 1 -- 172.21.15.31:0/815459771 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5500036a0 con 0x7fc554108e20 2024-04-26T17:45:22.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.205+0000 7fc55b389700 1 -- 172.21.15.31:0/815459771 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc55419d4e0 con 0x7fc554108e20 2024-04-26T17:45:22.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.205+0000 7fc54a7fc700 1 -- 172.21.15.31:0/815459771 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc55001a430 con 0x7fc554108e20 2024-04-26T17:45:22.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.205+0000 7fc54a7fc700 1 -- 172.21.15.31:0/815459771 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc550019550 con 0x7fc554108e20 2024-04-26T17:45:22.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.206+0000 7fc55b389700 1 -- 172.21.15.31:0/815459771 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5541a01c0 con 0x7fc554108e20 2024-04-26T17:45:22.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.206+0000 7fc54a7fc700 1 -- 172.21.15.31:0/815459771 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fc550019800 con 0x7fc554108e20 2024-04-26T17:45:22.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.207+0000 7fc54a7fc700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7fc5400395b0 0x7fc54003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:22.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.207+0000 7fc54a7fc700 1 -- 172.21.15.31:0/815459771 --> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc54003c180 con 0x7fc5400395b0 2024-04-26T17:45:22.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.207+0000 7fc54a7fc700 1 -- 172.21.15.31:0/815459771 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc55004d8a0 con 0x7fc554108e20 2024-04-26T17:45:22.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.207+0000 7fc558924700 1 -- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7fc5400395b0 msgr2=0x7fc54003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/545710545 2024-04-26T17:45:22.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.207+0000 7fc558924700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7fc5400395b0 0x7fc54003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T17:45:22.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.408+0000 7fc558924700 1 -- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7fc5400395b0 msgr2=0x7fc54003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/545710545 2024-04-26T17:45:22.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.408+0000 7fc558924700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7fc5400395b0 0x7fc54003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T17:45:22.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.809+0000 7fc558924700 1 -- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7fc5400395b0 msgr2=0x7fc54003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/545710545 2024-04-26T17:45:22.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:16.809+0000 7fc558924700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7fc5400395b0 0x7fc54003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T17:45:22.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:17.610+0000 7fc558924700 1 -- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7fc5400395b0 msgr2=0x7fc54003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/545710545 2024-04-26T17:45:22.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:17.610+0000 7fc558924700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7fc5400395b0 0x7fc54003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T17:45:22.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:19.211+0000 7fc558924700 1 -- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7fc5400395b0 msgr2=0x7fc54003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/545710545 2024-04-26T17:45:22.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:19.211+0000 7fc558924700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7fc5400395b0 0x7fc54003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T17:45:22.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:21.130+0000 7fc54a7fc700 1 -- 172.21.15.31:0/815459771 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7fc550019b40 con 0x7fc554108e20 2024-04-26T17:45:22.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:21.130+0000 7fc54a7fc700 1 -- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7fc5400395b0 msgr2=0x7fc54003ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T17:45:22.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:21.130+0000 7fc54a7fc700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7fc5400395b0 0x7fc54003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:22.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.131+0000 7fc54a7fc700 1 -- 172.21.15.31:0/815459771 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7fc55004f680 con 0x7fc554108e20 2024-04-26T17:45:22.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.131+0000 7fc54a7fc700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7fc54003cea0 0x7fc54003f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:22.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.131+0000 7fc54a7fc700 1 -- 172.21.15.31:0/815459771 --> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc54003c180 con 0x7fc54003cea0 2024-04-26T17:45:22.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.131+0000 7fc558924700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7fc54003cea0 0x7fc54003f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:22.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.132+0000 7fc558924700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7fc54003cea0 0x7fc54003f290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc544003a10 tx=0x7fc5440092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:22.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.133+0000 7fc54a7fc700 1 -- 172.21.15.31:0/815459771 <== mgr.14118 v2:172.21.15.31:6800/153341213 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fc54003c180 con 0x7fc54003cea0 2024-04-26T17:45:22.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.147+0000 7fc55b389700 1 -- 172.21.15.31:0/815459771 --> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fc55406a530 con 0x7fc54003cea0 2024-04-26T17:45:22.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.148+0000 7fc54a7fc700 1 -- 172.21.15.31:0/815459771 <== mgr.14118 v2:172.21.15.31:6800/153341213 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fc55406a530 con 0x7fc54003cea0 2024-04-26T17:45:22.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.148+0000 7fc55b389700 1 -- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7fc54003cea0 msgr2=0x7fc54003f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:22.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.148+0000 7fc55b389700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7fc54003cea0 0x7fc54003f290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc544003a10 tx=0x7fc5440092b0 comp rx=0 tx=0).stop 2024-04-26T17:45:22.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.148+0000 7fc55b389700 1 -- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc554108e20 msgr2=0x7fc55419cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:22.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.148+0000 7fc55b389700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc554108e20 0x7fc55419cd10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc550009fb0 tx=0x7fc550011940 comp rx=0 tx=0).stop 2024-04-26T17:45:22.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.148+0000 7fc55b389700 1 -- 172.21.15.31:0/815459771 shutdown_connections 2024-04-26T17:45:22.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.148+0000 7fc55b389700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7fc54003cea0 0x7fc54003f290 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:22.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.148+0000 7fc55b389700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc554108e20 0x7fc55419cd10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:22.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.148+0000 7fc55b389700 1 --2- 172.21.15.31:0/815459771 >> [v2:172.21.15.31:6800/545710545,v1:172.21.15.31:6801/545710545] conn(0x7fc5400395b0 0x7fc54003ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:22.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.148+0000 7fc55b389700 1 -- 172.21.15.31:0/815459771 >> 172.21.15.31:0/815459771 conn(0x7fc554104400 msgr2=0x7fc554104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:22.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.148+0000 7fc55b389700 1 -- 172.21.15.31:0/815459771 shutdown_connections 2024-04-26T17:45:22.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:22.148+0000 7fc55b389700 1 -- 172.21.15.31:0/815459771 wait complete. 2024-04-26T17:45:22.734 INFO:teuthology.orchestra.run.smithi031.stdout:mgr epoch 5 is available 2024-04-26T17:45:22.735 INFO:teuthology.orchestra.run.smithi031.stdout:Setting orchestrator backend to cephadm... 2024-04-26T17:45:23.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:23 smithi031 ceph-mon[29164]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T17:45:23.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:23 smithi031 ceph-mon[29164]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T17:45:23.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:23 smithi031 ceph-mon[29164]: [26/Apr/2024:17:45:22] ENGINE Bus STARTING 2024-04-26T17:45:23.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:23 smithi031 ceph-mon[29164]: [26/Apr/2024:17:45:22] ENGINE Serving on http://172.21.15.31:8765 2024-04-26T17:45:23.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:23 smithi031 ceph-mon[29164]: [26/Apr/2024:17:45:22] ENGINE Serving on https://172.21.15.31:7150 2024-04-26T17:45:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:23 smithi031 ceph-mon[29164]: [26/Apr/2024:17:45:22] ENGINE Bus STARTED 2024-04-26T17:45:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:23 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:45:23.992 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.083+0000 7f73befac700 1 Processor -- start 2024-04-26T17:45:23.992 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.083+0000 7f73befac700 1 -- start start 2024-04-26T17:45:23.992 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.083+0000 7f73befac700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73b807ee00 0x7f73b807d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:23.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.083+0000 7f73befac700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f73b807d840 con 0x7f73b807ee00 2024-04-26T17:45:23.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.083+0000 7f73bcd48700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73b807ee00 0x7f73b807d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:23.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.083+0000 7f73bcd48700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73b807ee00 0x7f73b807d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54976/0 (socket says 172.21.15.31:54976) 2024-04-26T17:45:23.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.083+0000 7f73bcd48700 1 -- 172.21.15.31:0/3394618320 learned_addr learned my addr 172.21.15.31:0/3394618320 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:23.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.084+0000 7f73bcd48700 1 -- 172.21.15.31:0/3394618320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73b807d9c0 con 0x7f73b807ee00 2024-04-26T17:45:23.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.084+0000 7f73bcd48700 1 --2- 172.21.15.31:0/3394618320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73b807ee00 0x7f73b807d300 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0098a0 tx=0x7f73ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e505cdd9ac078a6d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:23.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.084+0000 7f73b77fe700 1 -- 172.21.15.31:0/3394618320 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73ac004160 con 0x7f73b807ee00 2024-04-26T17:45:23.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.084+0000 7f73b77fe700 1 -- 172.21.15.31:0/3394618320 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f73ac004300 con 0x7f73b807ee00 2024-04-26T17:45:23.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.084+0000 7f73b77fe700 1 -- 172.21.15.31:0/3394618320 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73ac011af0 con 0x7f73b807ee00 2024-04-26T17:45:23.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.084+0000 7f73befac700 1 -- 172.21.15.31:0/3394618320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73b807ee00 msgr2=0x7f73b807d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:23.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.084+0000 7f73befac700 1 --2- 172.21.15.31:0/3394618320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73b807ee00 0x7f73b807d300 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0098a0 tx=0x7f73ac009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:23.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.085+0000 7f73befac700 1 -- 172.21.15.31:0/3394618320 shutdown_connections 2024-04-26T17:45:23.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.085+0000 7f73befac700 1 --2- 172.21.15.31:0/3394618320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73b807ee00 0x7f73b807d300 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:23.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.085+0000 7f73befac700 1 -- 172.21.15.31:0/3394618320 >> 172.21.15.31:0/3394618320 conn(0x7f73b81043c0 msgr2=0x7f73b81067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:23.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.085+0000 7f73befac700 1 -- 172.21.15.31:0/3394618320 shutdown_connections 2024-04-26T17:45:23.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.085+0000 7f73befac700 1 -- 172.21.15.31:0/3394618320 wait complete. 2024-04-26T17:45:23.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.085+0000 7f73befac700 1 Processor -- start 2024-04-26T17:45:23.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.085+0000 7f73befac700 1 -- start start 2024-04-26T17:45:23.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.085+0000 7f73befac700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73b807ee00 0x7f73b819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:23.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.085+0000 7f73befac700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f73b810f310 con 0x7f73b807ee00 2024-04-26T17:45:23.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.085+0000 7f73bcd48700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73b807ee00 0x7f73b819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:23.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.086+0000 7f73bcd48700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73b807ee00 0x7f73b819eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54984/0 (socket says 172.21.15.31:54984) 2024-04-26T17:45:23.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.086+0000 7f73bcd48700 1 -- 172.21.15.31:0/2937585477 learned_addr learned my addr 172.21.15.31:0/2937585477 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:23.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.086+0000 7f73bcd48700 1 -- 172.21.15.31:0/2937585477 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73b819f420 con 0x7f73b807ee00 2024-04-26T17:45:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.086+0000 7f73bcd48700 1 --2- 172.21.15.31:0/2937585477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73b807ee00 0x7f73b819eee0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009fb0 tx=0x7f73ac011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.086+0000 7f73b5ffb700 1 -- 172.21.15.31:0/2937585477 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73ac0036a0 con 0x7f73b807ee00 2024-04-26T17:45:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.086+0000 7f73b5ffb700 1 -- 172.21.15.31:0/2937585477 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f73ac01a430 con 0x7f73b807ee00 2024-04-26T17:45:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.086+0000 7f73befac700 1 -- 172.21.15.31:0/2937585477 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73b819f6b0 con 0x7f73b807ee00 2024-04-26T17:45:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.086+0000 7f73b5ffb700 1 -- 172.21.15.31:0/2937585477 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73ac019550 con 0x7f73b807ee00 2024-04-26T17:45:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.086+0000 7f73befac700 1 -- 172.21.15.31:0/2937585477 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73b81a23a0 con 0x7f73b807ee00 2024-04-26T17:45:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.087+0000 7f73b5ffb700 1 -- 172.21.15.31:0/2937585477 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f73ac019850 con 0x7f73b807ee00 2024-04-26T17:45:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.087+0000 7f73befac700 1 -- 172.21.15.31:0/2937585477 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73b8064a80 con 0x7f73b807ee00 2024-04-26T17:45:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.087+0000 7f73b5ffb700 1 --2- 172.21.15.31:0/2937585477 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f73a8039430 0x7f73a803b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.087+0000 7f73b5ffb700 1 -- 172.21.15.31:0/2937585477 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f73ac04d8d0 con 0x7f73b807ee00 2024-04-26T17:45:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.087+0000 7f73b7fff700 1 --2- 172.21.15.31:0/2937585477 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f73a8039430 0x7f73a803b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.088+0000 7f73b7fff700 1 --2- 172.21.15.31:0/2937585477 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f73a8039430 0x7f73a803b8f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f73a4006fd0 tx=0x7f73a4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.093+0000 7f73b5ffb700 1 -- 172.21.15.31:0/2937585477 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f73ac00bc40 con 0x7f73b807ee00 2024-04-26T17:45:23.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.190+0000 7f73b5ffb700 1 -- 172.21.15.31:0/2937585477 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f73ac02d960 con 0x7f73b807ee00 2024-04-26T17:45:23.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.326+0000 7f73befac700 1 -- 172.21.15.31:0/2937585477 --> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f73b8105660 con 0x7f73a8039430 2024-04-26T17:45:23.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.331+0000 7f73b5ffb700 1 -- 172.21.15.31:0/2937585477 <== mgr.14118 v2:172.21.15.31:6800/153341213 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f73b8105660 con 0x7f73a8039430 2024-04-26T17:45:23.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.333+0000 7f73befac700 1 -- 172.21.15.31:0/2937585477 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f73a8039430 msgr2=0x7f73a803b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:23.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.333+0000 7f73befac700 1 --2- 172.21.15.31:0/2937585477 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f73a8039430 0x7f73a803b8f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f73a4006fd0 tx=0x7f73a4006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:23.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.333+0000 7f73befac700 1 -- 172.21.15.31:0/2937585477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73b807ee00 msgr2=0x7f73b819eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:23.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.333+0000 7f73befac700 1 --2- 172.21.15.31:0/2937585477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73b807ee00 0x7f73b819eee0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009fb0 tx=0x7f73ac011940 comp rx=0 tx=0).stop 2024-04-26T17:45:23.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.333+0000 7f73befac700 1 -- 172.21.15.31:0/2937585477 shutdown_connections 2024-04-26T17:45:23.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.333+0000 7f73befac700 1 --2- 172.21.15.31:0/2937585477 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f73a8039430 0x7f73a803b8f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:23.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.333+0000 7f73befac700 1 --2- 172.21.15.31:0/2937585477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73b807ee00 0x7f73b819eee0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:23.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.333+0000 7f73befac700 1 -- 172.21.15.31:0/2937585477 >> 172.21.15.31:0/2937585477 conn(0x7f73b81043c0 msgr2=0x7f73b8104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:23.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.333+0000 7f73befac700 1 -- 172.21.15.31:0/2937585477 shutdown_connections 2024-04-26T17:45:23.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:23.334+0000 7f73befac700 1 -- 172.21.15.31:0/2937585477 wait complete. 2024-04-26T17:45:24.278 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:24 smithi031 ceph-mon[29164]: mgrmap e8: x(active, since 2s) 2024-04-26T17:45:24.278 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:24 smithi031 ceph-mon[29164]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:45:24.278 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:24 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' 2024-04-26T17:45:24.278 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:24 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:45:25.214 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:25 smithi031 ceph-mon[29164]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:45:25.222 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-26T17:45:25.222 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.386+0000 7f9893e38700 1 Processor -- start 2024-04-26T17:45:25.222 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.386+0000 7f9893e38700 1 -- start start 2024-04-26T17:45:25.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.386+0000 7f9893e38700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f988c07ee00 0x7f988c07d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:25.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.386+0000 7f9893e38700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f988c07d840 con 0x7f988c07ee00 2024-04-26T17:45:25.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.386+0000 7f9891bd4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f988c07ee00 0x7f988c07d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:25.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.386+0000 7f9891bd4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f988c07ee00 0x7f988c07d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54996/0 (socket says 172.21.15.31:54996) 2024-04-26T17:45:25.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.386+0000 7f9891bd4700 1 -- 172.21.15.31:0/1143800075 learned_addr learned my addr 172.21.15.31:0/1143800075 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:25.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.386+0000 7f9891bd4700 1 -- 172.21.15.31:0/1143800075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c07d9c0 con 0x7f988c07ee00 2024-04-26T17:45:25.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.387+0000 7f9891bd4700 1 --2- 172.21.15.31:0/1143800075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f988c07ee00 0x7f988c07d300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f98880098a0 tx=0x7f9888009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6764064c125148c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:25.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.387+0000 7f9890bd2700 1 -- 172.21.15.31:0/1143800075 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9888004160 con 0x7f988c07ee00 2024-04-26T17:45:25.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.387+0000 7f9890bd2700 1 -- 172.21.15.31:0/1143800075 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9888004300 con 0x7f988c07ee00 2024-04-26T17:45:25.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.387+0000 7f9890bd2700 1 -- 172.21.15.31:0/1143800075 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9888011af0 con 0x7f988c07ee00 2024-04-26T17:45:25.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.387+0000 7f9893e38700 1 -- 172.21.15.31:0/1143800075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f988c07ee00 msgr2=0x7f988c07d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:25.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.387+0000 7f9893e38700 1 --2- 172.21.15.31:0/1143800075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f988c07ee00 0x7f988c07d300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f98880098a0 tx=0x7f9888009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:25.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.388+0000 7f9893e38700 1 -- 172.21.15.31:0/1143800075 shutdown_connections 2024-04-26T17:45:25.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.388+0000 7f9893e38700 1 --2- 172.21.15.31:0/1143800075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f988c07ee00 0x7f988c07d300 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:25.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.388+0000 7f9893e38700 1 -- 172.21.15.31:0/1143800075 >> 172.21.15.31:0/1143800075 conn(0x7f988c104310 msgr2=0x7f988c106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:25.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.388+0000 7f9893e38700 1 -- 172.21.15.31:0/1143800075 shutdown_connections 2024-04-26T17:45:25.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.388+0000 7f9893e38700 1 -- 172.21.15.31:0/1143800075 wait complete. 2024-04-26T17:45:25.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.388+0000 7f9893e38700 1 Processor -- start 2024-04-26T17:45:25.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.388+0000 7f9893e38700 1 -- start start 2024-04-26T17:45:25.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.388+0000 7f9893e38700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f988c07ee00 0x7f988c19ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:25.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.388+0000 7f9893e38700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f988c10f280 con 0x7f988c07ee00 2024-04-26T17:45:25.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.389+0000 7f9891bd4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f988c07ee00 0x7f988c19ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:25.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.389+0000 7f9891bd4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f988c07ee00 0x7f988c19ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55004/0 (socket says 172.21.15.31:55004) 2024-04-26T17:45:25.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.389+0000 7f9891bd4700 1 -- 172.21.15.31:0/535029122 learned_addr learned my addr 172.21.15.31:0/535029122 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:25.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.389+0000 7f9891bd4700 1 -- 172.21.15.31:0/535029122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c19f370 con 0x7f988c07ee00 2024-04-26T17:45:25.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.389+0000 7f9891bd4700 1 --2- 172.21.15.31:0/535029122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f988c07ee00 0x7f988c19ee30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9888011ac0 tx=0x7f9888011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:25.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.389+0000 7f9882ffd700 1 -- 172.21.15.31:0/535029122 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9888003b20 con 0x7f988c07ee00 2024-04-26T17:45:25.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.389+0000 7f9882ffd700 1 -- 172.21.15.31:0/535029122 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f988801a8c0 con 0x7f988c07ee00 2024-04-26T17:45:25.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.389+0000 7f9893e38700 1 -- 172.21.15.31:0/535029122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f988c19f600 con 0x7f988c07ee00 2024-04-26T17:45:25.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.389+0000 7f9882ffd700 1 -- 172.21.15.31:0/535029122 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98880248d0 con 0x7f988c07ee00 2024-04-26T17:45:25.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.389+0000 7f9893e38700 1 -- 172.21.15.31:0/535029122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f988c1a22f0 con 0x7f988c07ee00 2024-04-26T17:45:25.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.390+0000 7f9882ffd700 1 -- 172.21.15.31:0/535029122 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f9888022020 con 0x7f988c07ee00 2024-04-26T17:45:25.226 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.390+0000 7f9893e38700 1 -- 172.21.15.31:0/535029122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f988c06a530 con 0x7f988c07ee00 2024-04-26T17:45:25.226 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.390+0000 7f9882ffd700 1 --2- 172.21.15.31:0/535029122 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f98780395b0 0x7f987803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:25.226 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.390+0000 7f9882ffd700 1 -- 172.21.15.31:0/535029122 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f988804d850 con 0x7f988c07ee00 2024-04-26T17:45:25.226 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.390+0000 7f98913d3700 1 --2- 172.21.15.31:0/535029122 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f98780395b0 0x7f987803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:25.226 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.391+0000 7f98913d3700 1 --2- 172.21.15.31:0/535029122 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f98780395b0 0x7f987803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f987c006fd0 tx=0x7f987c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:25.226 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.396+0000 7f9882ffd700 1 -- 172.21.15.31:0/535029122 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f988800f2e0 con 0x7f988c07ee00 2024-04-26T17:45:25.226 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.605+0000 7f9893e38700 1 -- 172.21.15.31:0/535029122 --> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f988c105600 con 0x7f98780395b0 2024-04-26T17:45:25.226 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.606+0000 7f9882ffd700 1 -- 172.21.15.31:0/535029122 <== mgr.14118 v2:172.21.15.31:6800/153341213 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f988c105600 con 0x7f98780395b0 2024-04-26T17:45:25.226 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.608+0000 7f9893e38700 1 -- 172.21.15.31:0/535029122 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f98780395b0 msgr2=0x7f987803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:25.226 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.608+0000 7f9893e38700 1 --2- 172.21.15.31:0/535029122 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f98780395b0 0x7f987803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f987c006fd0 tx=0x7f987c006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:25.226 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.608+0000 7f9893e38700 1 -- 172.21.15.31:0/535029122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f988c07ee00 msgr2=0x7f988c19ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:25.226 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.608+0000 7f9893e38700 1 --2- 172.21.15.31:0/535029122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f988c07ee00 0x7f988c19ee30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9888011ac0 tx=0x7f9888011ba0 comp rx=0 tx=0).stop 2024-04-26T17:45:25.226 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.608+0000 7f9893e38700 1 -- 172.21.15.31:0/535029122 shutdown_connections 2024-04-26T17:45:25.227 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.608+0000 7f9893e38700 1 --2- 172.21.15.31:0/535029122 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f98780395b0 0x7f987803ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:25.227 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.608+0000 7f9893e38700 1 --2- 172.21.15.31:0/535029122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f988c07ee00 0x7f988c19ee30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:25.227 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.608+0000 7f9893e38700 1 -- 172.21.15.31:0/535029122 >> 172.21.15.31:0/535029122 conn(0x7f988c104310 msgr2=0x7f988c104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:25.227 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.608+0000 7f9893e38700 1 -- 172.21.15.31:0/535029122 shutdown_connections 2024-04-26T17:45:25.227 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:24.608+0000 7f9893e38700 1 -- 172.21.15.31:0/535029122 wait complete. 2024-04-26T17:45:25.227 INFO:teuthology.orchestra.run.smithi031.stdout:Generating ssh key... 2024-04-26T17:45:26.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-mon[29164]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:45:26.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-mon[29164]: Generating ssh key... 2024-04-26T17:45:26.560 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: Generating public/private rsa key pair. 2024-04-26T17:45:26.560 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: Your identification has been saved in /tmp/tmpdql8gg26/key. 2024-04-26T17:45:26.560 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: Your public key has been saved in /tmp/tmpdql8gg26/key.pub. 2024-04-26T17:45:26.560 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: The key fingerprint is: 2024-04-26T17:45:26.560 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: SHA256:dVUg3DD1jHuaPBKDd3/QweNp2nmWpliA3W2UmsJ0C+4 ceph-7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:45:26.560 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: The key's randomart image is: 2024-04-26T17:45:26.560 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: +---[RSA 3072]----+ 2024-04-26T17:45:26.561 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: | .+++o.| 2024-04-26T17:45:26.561 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: | .+o+.| 2024-04-26T17:45:26.561 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: | .o...*o| 2024-04-26T17:45:26.561 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: | .*o+ Bo+| 2024-04-26T17:45:26.561 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: | S..*+=+*o| 2024-04-26T17:45:26.561 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: | ..o==*o| 2024-04-26T17:45:26.561 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: | E.o===| 2024-04-26T17:45:26.561 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: | o.ooo| 2024-04-26T17:45:26.561 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: | . . | 2024-04-26T17:45:26.561 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: +----[SHA256]-----+ 2024-04-26T17:45:26.837 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.652+0000 7f6c92377700 1 Processor -- start 2024-04-26T17:45:26.837 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.652+0000 7f6c92377700 1 -- start start 2024-04-26T17:45:26.837 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.652+0000 7f6c92377700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8c107c70 0x7f6c8c108090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:26.837 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.652+0000 7f6c92377700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c108660 con 0x7f6c8c107c70 2024-04-26T17:45:26.837 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.652+0000 7f6c8bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8c107c70 0x7f6c8c108090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:26.837 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.652+0000 7f6c8bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8c107c70 0x7f6c8c108090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53696/0 (socket says 172.21.15.31:53696) 2024-04-26T17:45:26.837 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.652+0000 7f6c8bfff700 1 -- 172.21.15.31:0/2121786639 learned_addr learned my addr 172.21.15.31:0/2121786639 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:26.838 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.652+0000 7f6c8bfff700 1 -- 172.21.15.31:0/2121786639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c108e70 con 0x7f6c8c107c70 2024-04-26T17:45:26.838 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.653+0000 7f6c8bfff700 1 --2- 172.21.15.31:0/2121786639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8c107c70 0x7f6c8c108090 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0098a0 tx=0x7f6c7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48b9b72a0e94cfb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:26.838 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.653+0000 7f6c8affd700 1 -- 172.21.15.31:0/2121786639 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c7c004160 con 0x7f6c8c107c70 2024-04-26T17:45:26.838 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.653+0000 7f6c8affd700 1 -- 172.21.15.31:0/2121786639 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6c7c004300 con 0x7f6c8c107c70 2024-04-26T17:45:26.838 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.653+0000 7f6c8affd700 1 -- 172.21.15.31:0/2121786639 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c7c011af0 con 0x7f6c8c107c70 2024-04-26T17:45:26.838 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.653+0000 7f6c92377700 1 -- 172.21.15.31:0/2121786639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8c107c70 msgr2=0x7f6c8c108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:26.838 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.653+0000 7f6c92377700 1 --2- 172.21.15.31:0/2121786639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8c107c70 0x7f6c8c108090 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0098a0 tx=0x7f6c7c009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:26.838 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.653+0000 7f6c92377700 1 -- 172.21.15.31:0/2121786639 shutdown_connections 2024-04-26T17:45:26.838 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.653+0000 7f6c92377700 1 --2- 172.21.15.31:0/2121786639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8c107c70 0x7f6c8c108090 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:26.838 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.653+0000 7f6c92377700 1 -- 172.21.15.31:0/2121786639 >> 172.21.15.31:0/2121786639 conn(0x7f6c8c103230 msgr2=0x7f6c8c105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:26.838 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.654+0000 7f6c92377700 1 -- 172.21.15.31:0/2121786639 shutdown_connections 2024-04-26T17:45:26.838 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.654+0000 7f6c92377700 1 -- 172.21.15.31:0/2121786639 wait complete. 2024-04-26T17:45:26.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.654+0000 7f6c92377700 1 Processor -- start 2024-04-26T17:45:26.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.654+0000 7f6c92377700 1 -- start start 2024-04-26T17:45:26.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.655+0000 7f6c92377700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8c107c70 0x7f6c8c192de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:26.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.655+0000 7f6c92377700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c10c220 con 0x7f6c8c107c70 2024-04-26T17:45:26.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.655+0000 7f6c8bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8c107c70 0x7f6c8c192de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:26.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.655+0000 7f6c8bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8c107c70 0x7f6c8c192de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53708/0 (socket says 172.21.15.31:53708) 2024-04-26T17:45:26.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.655+0000 7f6c8bfff700 1 -- 172.21.15.31:0/3954544253 learned_addr learned my addr 172.21.15.31:0/3954544253 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:26.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.655+0000 7f6c8bfff700 1 -- 172.21.15.31:0/3954544253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c193320 con 0x7f6c8c107c70 2024-04-26T17:45:26.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.655+0000 7f6c8bfff700 1 --2- 172.21.15.31:0/3954544253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8c107c70 0x7f6c8c192de0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0117b0 tx=0x7f6c7c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:26.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.655+0000 7f6c897fa700 1 -- 172.21.15.31:0/3954544253 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c7c003af0 con 0x7f6c8c107c70 2024-04-26T17:45:26.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.655+0000 7f6c92377700 1 -- 172.21.15.31:0/3954544253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c8c191510 con 0x7f6c8c107c70 2024-04-26T17:45:26.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.655+0000 7f6c897fa700 1 -- 172.21.15.31:0/3954544253 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6c7c020070 con 0x7f6c8c107c70 2024-04-26T17:45:26.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.655+0000 7f6c897fa700 1 -- 172.21.15.31:0/3954544253 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c7c019630 con 0x7f6c8c107c70 2024-04-26T17:45:26.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.656+0000 7f6c92377700 1 -- 172.21.15.31:0/3954544253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c8c191a50 con 0x7f6c8c107c70 2024-04-26T17:45:26.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.656+0000 7f6c897fa700 1 -- 172.21.15.31:0/3954544253 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f6c7c0198f0 con 0x7f6c8c107c70 2024-04-26T17:45:26.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.656+0000 7f6c92377700 1 -- 172.21.15.31:0/3954544253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c8c064a80 con 0x7f6c8c107c70 2024-04-26T17:45:26.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.657+0000 7f6c897fa700 1 --2- 172.21.15.31:0/3954544253 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f6c740395b0 0x7f6c7403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:26.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.657+0000 7f6c897fa700 1 -- 172.21.15.31:0/3954544253 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6c7c04dcc0 con 0x7f6c8c107c70 2024-04-26T17:45:26.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.657+0000 7f6c8b7fe700 1 --2- 172.21.15.31:0/3954544253 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f6c740395b0 0x7f6c7403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:26.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.657+0000 7f6c8b7fe700 1 --2- 172.21.15.31:0/3954544253 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f6c740395b0 0x7f6c7403ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6c80006fd0 tx=0x7f6c80006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:26.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.662+0000 7f6c897fa700 1 -- 172.21.15.31:0/3954544253 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c7c023370 con 0x7f6c8c107c70 2024-04-26T17:45:26.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:25.869+0000 7f6c92377700 1 -- 172.21.15.31:0/3954544253 --> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f6c8c196bd0 con 0x7f6c740395b0 2024-04-26T17:45:26.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:26.332+0000 7f6c897fa700 1 -- 172.21.15.31:0/3954544253 <== mgr.14118 v2:172.21.15.31:6800/153341213 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6c8c196bd0 con 0x7f6c740395b0 2024-04-26T17:45:26.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:26.334+0000 7f6c92377700 1 -- 172.21.15.31:0/3954544253 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f6c740395b0 msgr2=0x7f6c7403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:26.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:26.334+0000 7f6c92377700 1 --2- 172.21.15.31:0/3954544253 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f6c740395b0 0x7f6c7403ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6c80006fd0 tx=0x7f6c80006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:26.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:26.334+0000 7f6c92377700 1 -- 172.21.15.31:0/3954544253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8c107c70 msgr2=0x7f6c8c192de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:26.841 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:26.334+0000 7f6c92377700 1 --2- 172.21.15.31:0/3954544253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8c107c70 0x7f6c8c192de0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0117b0 tx=0x7f6c7c011b70 comp rx=0 tx=0).stop 2024-04-26T17:45:26.841 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:26.334+0000 7f6c92377700 1 -- 172.21.15.31:0/3954544253 shutdown_connections 2024-04-26T17:45:26.841 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:26.334+0000 7f6c92377700 1 --2- 172.21.15.31:0/3954544253 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f6c740395b0 0x7f6c7403ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:26.841 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:26.334+0000 7f6c92377700 1 --2- 172.21.15.31:0/3954544253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c8c107c70 0x7f6c8c192de0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:26.841 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:26.334+0000 7f6c92377700 1 -- 172.21.15.31:0/3954544253 >> 172.21.15.31:0/3954544253 conn(0x7f6c8c103230 msgr2=0x7f6c8c104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:26.841 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:26.334+0000 7f6c92377700 1 -- 172.21.15.31:0/3954544253 shutdown_connections 2024-04-26T17:45:26.841 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:26.334+0000 7f6c92377700 1 -- 172.21.15.31:0/3954544253 wait complete. 2024-04-26T17:45:27.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:27 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' 2024-04-26T17:45:27.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:27 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' 2024-04-26T17:45:28.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu+NEeVJmhwe7hcSYRp4pMsWBs7o6C9+QTkNP4fbnQExDeY/OTVmIni1bcxbzMUptWE3ZBNOgC93mqeAfSms6AaqGxFhdxQei7I6+IPNoKU2O9ry2g8Db4Qwz0OIUJUDLuISMJj8El8+EfPfAvhSdLIdJ7sj3eQwVpZDxp7X7t0P4e5OaxOON21wh2/RPltchW9JsjG0HZrcH0Yv7q/mkJz1jgzcp7imaBurMZ+bgrkEg/n1k+J/dzEHXJZRlBM/nyd9vbSBCTrmqhf6xtuRicjjXH+n2f6NfbFtfcM4bUMLz7D5qLzC2kVKIu1mIUtbiyXiHipgZ3AJwqL86b+k8mxrw30uLExiyWeEXQKnAmOAmwq7umn9lxkZk0oQ4ZEK74c+9mxWLke1c/MitDCGMGopF3/RL0AbwpFLcpV/ga1My7Dvs5eW+EHV9Ks+i4v82V+R3KCGquk1vxEFJ2fxlCAVI61gvw5JC2L3Cis1Rg6Cc/BodMs46Zrl4xnytlHds= ceph-7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:45:28.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.236+0000 7f9acc791700 1 Processor -- start 2024-04-26T17:45:28.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.236+0000 7f9acc791700 1 -- start start 2024-04-26T17:45:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.236+0000 7f9acc791700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac4108e00 0x7f9ac4109220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.236+0000 7f9acc791700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac41097f0 con 0x7f9ac4108e00 2024-04-26T17:45:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.237+0000 7f9aca52d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac4108e00 0x7f9ac4109220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.237+0000 7f9aca52d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac4108e00 0x7f9ac4109220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53712/0 (socket says 172.21.15.31:53712) 2024-04-26T17:45:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.237+0000 7f9aca52d700 1 -- 172.21.15.31:0/2036822417 learned_addr learned my addr 172.21.15.31:0/2036822417 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.237+0000 7f9aca52d700 1 -- 172.21.15.31:0/2036822417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac4109f60 con 0x7f9ac4108e00 2024-04-26T17:45:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.237+0000 7f9aca52d700 1 --2- 172.21.15.31:0/2036822417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac4108e00 0x7f9ac4109220 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4009e80 tx=0x7f9ab400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d329445f27fe34 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.237+0000 7f9ac952b700 1 -- 172.21.15.31:0/2036822417 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ab4017070 con 0x7f9ac4108e00 2024-04-26T17:45:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.237+0000 7f9ac952b700 1 -- 172.21.15.31:0/2036822417 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9ab400b8c0 con 0x7f9ac4108e00 2024-04-26T17:45:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.237+0000 7f9ac952b700 1 -- 172.21.15.31:0/2036822417 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ab4004490 con 0x7f9ac4108e00 2024-04-26T17:45:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.238+0000 7f9acc791700 1 -- 172.21.15.31:0/2036822417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac4108e00 msgr2=0x7f9ac4109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.238+0000 7f9acc791700 1 --2- 172.21.15.31:0/2036822417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac4108e00 0x7f9ac4109220 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4009e80 tx=0x7f9ab400b330 comp rx=0 tx=0).stop 2024-04-26T17:45:28.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.238+0000 7f9acc791700 1 -- 172.21.15.31:0/2036822417 shutdown_connections 2024-04-26T17:45:28.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.238+0000 7f9acc791700 1 --2- 172.21.15.31:0/2036822417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac4108e00 0x7f9ac4109220 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:28.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.238+0000 7f9acc791700 1 -- 172.21.15.31:0/2036822417 >> 172.21.15.31:0/2036822417 conn(0x7f9ac41043c0 msgr2=0x7f9ac41067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:28.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.238+0000 7f9acc791700 1 -- 172.21.15.31:0/2036822417 shutdown_connections 2024-04-26T17:45:28.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.238+0000 7f9acc791700 1 -- 172.21.15.31:0/2036822417 wait complete. 2024-04-26T17:45:28.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.238+0000 7f9acc791700 1 Processor -- start 2024-04-26T17:45:28.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.239+0000 7f9acc791700 1 -- start start 2024-04-26T17:45:28.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.239+0000 7f9acc791700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac4108e00 0x7f9ac419ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:28.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.239+0000 7f9acc791700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac410d310 con 0x7f9ac4108e00 2024-04-26T17:45:28.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.239+0000 7f9aca52d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac4108e00 0x7f9ac419ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:28.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.239+0000 7f9aca52d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac4108e00 0x7f9ac419ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53728/0 (socket says 172.21.15.31:53728) 2024-04-26T17:45:28.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.239+0000 7f9aca52d700 1 -- 172.21.15.31:0/2876968350 learned_addr learned my addr 172.21.15.31:0/2876968350 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:28.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.239+0000 7f9aca52d700 1 -- 172.21.15.31:0/2876968350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac419d1f0 con 0x7f9ac4108e00 2024-04-26T17:45:28.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.239+0000 7f9aca52d700 1 --2- 172.21.15.31:0/2876968350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac4108e00 0x7f9ac419ccb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4004a60 tx=0x7f9ab400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:28.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.239+0000 7f9ac37fe700 1 -- 172.21.15.31:0/2876968350 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ab4020020 con 0x7f9ac4108e00 2024-04-26T17:45:28.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.239+0000 7f9ac37fe700 1 -- 172.21.15.31:0/2876968350 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9ab4012430 con 0x7f9ac4108e00 2024-04-26T17:45:28.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.239+0000 7f9acc791700 1 -- 172.21.15.31:0/2876968350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac419d480 con 0x7f9ac4108e00 2024-04-26T17:45:28.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.239+0000 7f9ac37fe700 1 -- 172.21.15.31:0/2876968350 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ab4017040 con 0x7f9ac4108e00 2024-04-26T17:45:28.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.239+0000 7f9acc791700 1 -- 172.21.15.31:0/2876968350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac41a0170 con 0x7f9ac4108e00 2024-04-26T17:45:28.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.240+0000 7f9ac37fe700 1 -- 172.21.15.31:0/2876968350 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f9ab4023020 con 0x7f9ac4108e00 2024-04-26T17:45:28.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.240+0000 7f9acc791700 1 -- 172.21.15.31:0/2876968350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac4064a80 con 0x7f9ac4108e00 2024-04-26T17:45:28.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.241+0000 7f9ac37fe700 1 --2- 172.21.15.31:0/2876968350 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f9aac0395b0 0x7f9aac03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:28.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.241+0000 7f9ac37fe700 1 -- 172.21.15.31:0/2876968350 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9ab400f050 con 0x7f9ac4108e00 2024-04-26T17:45:28.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.241+0000 7f9ac9d2c700 1 --2- 172.21.15.31:0/2876968350 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f9aac0395b0 0x7f9aac03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:28.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.241+0000 7f9ac9d2c700 1 --2- 172.21.15.31:0/2876968350 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f9aac0395b0 0x7f9aac03ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8006fd0 tx=0x7f9ab8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:28.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.247+0000 7f9ac37fe700 1 -- 172.21.15.31:0/2876968350 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ab402a020 con 0x7f9ac4108e00 2024-04-26T17:45:28.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.464+0000 7f9acc791700 1 -- 172.21.15.31:0/2876968350 --> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f9ac41055e0 con 0x7f9aac0395b0 2024-04-26T17:45:28.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.464+0000 7f9ac37fe700 1 -- 172.21.15.31:0/2876968350 <== mgr.14118 v2:172.21.15.31:6800/153341213 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f9ac41055e0 con 0x7f9aac0395b0 2024-04-26T17:45:28.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.466+0000 7f9acc791700 1 -- 172.21.15.31:0/2876968350 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f9aac0395b0 msgr2=0x7f9aac03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:28.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.466+0000 7f9acc791700 1 --2- 172.21.15.31:0/2876968350 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f9aac0395b0 0x7f9aac03ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8006fd0 tx=0x7f9ab8006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:28.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.466+0000 7f9acc791700 1 -- 172.21.15.31:0/2876968350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac4108e00 msgr2=0x7f9ac419ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:28.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.466+0000 7f9acc791700 1 --2- 172.21.15.31:0/2876968350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac4108e00 0x7f9ac419ccb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4004a60 tx=0x7f9ab400bd50 comp rx=0 tx=0).stop 2024-04-26T17:45:28.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.466+0000 7f9acc791700 1 -- 172.21.15.31:0/2876968350 shutdown_connections 2024-04-26T17:45:28.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.466+0000 7f9acc791700 1 --2- 172.21.15.31:0/2876968350 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f9aac0395b0 0x7f9aac03ba70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:28.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.466+0000 7f9acc791700 1 --2- 172.21.15.31:0/2876968350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac4108e00 0x7f9ac419ccb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:28.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.466+0000 7f9acc791700 1 -- 172.21.15.31:0/2876968350 >> 172.21.15.31:0/2876968350 conn(0x7f9ac41043c0 msgr2=0x7f9ac4104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:28.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.467+0000 7f9acc791700 1 -- 172.21.15.31:0/2876968350 shutdown_connections 2024-04-26T17:45:28.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:27.467+0000 7f9acc791700 1 -- 172.21.15.31:0/2876968350 wait complete. 2024-04-26T17:45:28.033 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-26T17:45:28.033 INFO:teuthology.orchestra.run.smithi031.stdout:Adding key to root@localhost authorized_keys... 2024-04-26T17:45:28.033 INFO:teuthology.orchestra.run.smithi031.stdout:Adding host smithi031... 2024-04-26T17:45:28.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:28 smithi031 ceph-mon[29164]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:45:29.550 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:29 smithi031 ceph-mon[29164]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi031", "addr": "172.21.15.31", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:45:30.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:30 smithi031 ceph-mon[29164]: Deploying cephadm binary to smithi031 2024-04-26T17:45:33.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Added host 'smithi031' with addr '172.21.15.31' 2024-04-26T17:45:33.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.451+0000 7f54d37e7700 1 Processor -- start 2024-04-26T17:45:33.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.452+0000 7f54d37e7700 1 -- start start 2024-04-26T17:45:33.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.452+0000 7f54d37e7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54cc107060 0x7f54cc107480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:33.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.452+0000 7f54d37e7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc107a50 con 0x7f54cc107060 2024-04-26T17:45:33.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.452+0000 7f54d1583700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54cc107060 0x7f54cc107480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:33.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.452+0000 7f54d1583700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54cc107060 0x7f54cc107480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53738/0 (socket says 172.21.15.31:53738) 2024-04-26T17:45:33.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.452+0000 7f54d1583700 1 -- 172.21.15.31:0/4139276856 learned_addr learned my addr 172.21.15.31:0/4139276856 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:33.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.452+0000 7f54d1583700 1 -- 172.21.15.31:0/4139276856 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54cc10a000 con 0x7f54cc107060 2024-04-26T17:45:33.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.453+0000 7f54d1583700 1 --2- 172.21.15.31:0/4139276856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54cc107060 0x7f54cc107480 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f54c80098a0 tx=0x7f54c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d251ccf9278c1333 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:33.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.453+0000 7f54c3fff700 1 -- 172.21.15.31:0/4139276856 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54c8004160 con 0x7f54cc107060 2024-04-26T17:45:33.568 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.453+0000 7f54c3fff700 1 -- 172.21.15.31:0/4139276856 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f54c8004300 con 0x7f54cc107060 2024-04-26T17:45:33.568 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.453+0000 7f54c3fff700 1 -- 172.21.15.31:0/4139276856 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54c8011af0 con 0x7f54cc107060 2024-04-26T17:45:33.568 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.453+0000 7f54d37e7700 1 -- 172.21.15.31:0/4139276856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54cc107060 msgr2=0x7f54cc107480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:33.568 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.453+0000 7f54d37e7700 1 --2- 172.21.15.31:0/4139276856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54cc107060 0x7f54cc107480 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f54c80098a0 tx=0x7f54c8009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:33.568 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.453+0000 7f54d37e7700 1 -- 172.21.15.31:0/4139276856 shutdown_connections 2024-04-26T17:45:33.568 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.453+0000 7f54d37e7700 1 --2- 172.21.15.31:0/4139276856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54cc107060 0x7f54cc107480 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:33.568 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.454+0000 7f54d37e7700 1 -- 172.21.15.31:0/4139276856 >> 172.21.15.31:0/4139276856 conn(0x7f54cc102960 msgr2=0x7f54cc104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:33.568 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.454+0000 7f54d37e7700 1 -- 172.21.15.31:0/4139276856 shutdown_connections 2024-04-26T17:45:33.568 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.454+0000 7f54d37e7700 1 -- 172.21.15.31:0/4139276856 wait complete. 2024-04-26T17:45:33.568 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.454+0000 7f54d37e7700 1 Processor -- start 2024-04-26T17:45:33.568 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.454+0000 7f54d37e7700 1 -- start start 2024-04-26T17:45:33.568 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.454+0000 7f54d37e7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54cc107060 0x7f54cc19cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:33.569 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.454+0000 7f54d37e7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc10d3b0 con 0x7f54cc107060 2024-04-26T17:45:33.569 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.455+0000 7f54d1583700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54cc107060 0x7f54cc19cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:33.569 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.455+0000 7f54d1583700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54cc107060 0x7f54cc19cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53750/0 (socket says 172.21.15.31:53750) 2024-04-26T17:45:33.569 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.455+0000 7f54d1583700 1 -- 172.21.15.31:0/2931444662 learned_addr learned my addr 172.21.15.31:0/2931444662 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:33.569 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.455+0000 7f54d1583700 1 -- 172.21.15.31:0/2931444662 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54cc19d1c0 con 0x7f54cc107060 2024-04-26T17:45:33.569 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.455+0000 7f54d1583700 1 --2- 172.21.15.31:0/2931444662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54cc107060 0x7f54cc19cc80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f54c8003f40 tx=0x7f54c8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:33.569 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.455+0000 7f54c27fc700 1 -- 172.21.15.31:0/2931444662 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54c80038b0 con 0x7f54cc107060 2024-04-26T17:45:33.569 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.455+0000 7f54c27fc700 1 -- 172.21.15.31:0/2931444662 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f54c8003a50 con 0x7f54cc107060 2024-04-26T17:45:33.569 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.455+0000 7f54d37e7700 1 -- 172.21.15.31:0/2931444662 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54cc19d450 con 0x7f54cc107060 2024-04-26T17:45:33.569 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.455+0000 7f54c27fc700 1 -- 172.21.15.31:0/2931444662 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54c801a5c0 con 0x7f54cc107060 2024-04-26T17:45:33.569 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.455+0000 7f54d37e7700 1 -- 172.21.15.31:0/2931444662 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54cc1a0140 con 0x7f54cc107060 2024-04-26T17:45:33.569 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.456+0000 7f54c27fc700 1 -- 172.21.15.31:0/2931444662 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f54c801a8c0 con 0x7f54cc107060 2024-04-26T17:45:33.569 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.456+0000 7f54d37e7700 1 -- 172.21.15.31:0/2931444662 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54cc064a80 con 0x7f54cc107060 2024-04-26T17:45:33.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.457+0000 7f54c27fc700 1 --2- 172.21.15.31:0/2931444662 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f54b8039560 0x7f54b803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:33.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.457+0000 7f54c27fc700 1 -- 172.21.15.31:0/2931444662 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f54c804dc50 con 0x7f54cc107060 2024-04-26T17:45:33.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.457+0000 7f54d0d82700 1 --2- 172.21.15.31:0/2931444662 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f54b8039560 0x7f54b803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:33.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.457+0000 7f54d0d82700 1 --2- 172.21.15.31:0/2931444662 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f54b8039560 0x7f54b803ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f54bc006fd0 tx=0x7f54bc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:33.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.462+0000 7f54c27fc700 1 -- 172.21.15.31:0/2931444662 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f54c8053020 con 0x7f54cc107060 2024-04-26T17:45:33.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:28.676+0000 7f54d37e7700 1 -- 172.21.15.31:0/2931444662 --> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi031", "addr": "172.21.15.31", "target": ["mon-mgr", ""]}) -- 0x7f54cc103b60 con 0x7f54b8039560 2024-04-26T17:45:33.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:32.968+0000 7f54c27fc700 1 -- 172.21.15.31:0/2931444662 <== mgr.14118 v2:172.21.15.31:6800/153341213 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f54cc103b60 con 0x7f54b8039560 2024-04-26T17:45:33.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:32.971+0000 7f54d37e7700 1 -- 172.21.15.31:0/2931444662 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f54b8039560 msgr2=0x7f54b803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:33.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:32.971+0000 7f54d37e7700 1 --2- 172.21.15.31:0/2931444662 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f54b8039560 0x7f54b803ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f54bc006fd0 tx=0x7f54bc006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:33.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:32.971+0000 7f54d37e7700 1 -- 172.21.15.31:0/2931444662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54cc107060 msgr2=0x7f54cc19cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:33.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:32.971+0000 7f54d37e7700 1 --2- 172.21.15.31:0/2931444662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54cc107060 0x7f54cc19cc80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f54c8003f40 tx=0x7f54c8011890 comp rx=0 tx=0).stop 2024-04-26T17:45:33.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:32.971+0000 7f54d37e7700 1 -- 172.21.15.31:0/2931444662 shutdown_connections 2024-04-26T17:45:33.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:32.971+0000 7f54d37e7700 1 --2- 172.21.15.31:0/2931444662 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f54b8039560 0x7f54b803ba20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:33.571 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:32.971+0000 7f54d37e7700 1 --2- 172.21.15.31:0/2931444662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54cc107060 0x7f54cc19cc80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:33.571 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:32.971+0000 7f54d37e7700 1 -- 172.21.15.31:0/2931444662 >> 172.21.15.31:0/2931444662 conn(0x7f54cc102960 msgr2=0x7f54cc103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:33.571 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:32.971+0000 7f54d37e7700 1 -- 172.21.15.31:0/2931444662 shutdown_connections 2024-04-26T17:45:33.571 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:32.971+0000 7f54d37e7700 1 -- 172.21.15.31:0/2931444662 wait complete. 2024-04-26T17:45:33.571 INFO:teuthology.orchestra.run.smithi031.stdout:Deploying unmanaged mon service... 2024-04-26T17:45:34.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:33 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' 2024-04-26T17:45:34.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:33 smithi031 ceph-mon[29164]: Added host smithi031 2024-04-26T17:45:34.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:33 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:45:35.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-26T17:45:35.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.001+0000 7f727d65e700 1 Processor -- start 2024-04-26T17:45:35.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.001+0000 7f727d65e700 1 -- start start 2024-04-26T17:45:35.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.001+0000 7f727d65e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7278108d50 0x7f7278109170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:35.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.001+0000 7f727d65e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7278109740 con 0x7f7278108d50 2024-04-26T17:45:35.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.001+0000 7f7276ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7278108d50 0x7f7278109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:35.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.001+0000 7f7276ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7278108d50 0x7f7278109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53766/0 (socket says 172.21.15.31:53766) 2024-04-26T17:45:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.001+0000 7f7276ffd700 1 -- 172.21.15.31:0/4169072826 learned_addr learned my addr 172.21.15.31:0/4169072826 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.002+0000 7f7276ffd700 1 -- 172.21.15.31:0/4169072826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7278109f50 con 0x7f7278108d50 2024-04-26T17:45:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.002+0000 7f7276ffd700 1 --2- 172.21.15.31:0/4169072826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7278108d50 0x7f7278109170 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f72680098a0 tx=0x7f7268009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f218c2ef26ec708d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.002+0000 7f7275ffb700 1 -- 172.21.15.31:0/4169072826 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7268004160 con 0x7f7278108d50 2024-04-26T17:45:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.002+0000 7f7275ffb700 1 -- 172.21.15.31:0/4169072826 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7268004300 con 0x7f7278108d50 2024-04-26T17:45:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.002+0000 7f7275ffb700 1 -- 172.21.15.31:0/4169072826 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7268011af0 con 0x7f7278108d50 2024-04-26T17:45:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.003+0000 7f727d65e700 1 -- 172.21.15.31:0/4169072826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7278108d50 msgr2=0x7f7278109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.003+0000 7f727d65e700 1 --2- 172.21.15.31:0/4169072826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7278108d50 0x7f7278109170 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f72680098a0 tx=0x7f7268009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.003+0000 7f727d65e700 1 -- 172.21.15.31:0/4169072826 shutdown_connections 2024-04-26T17:45:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.003+0000 7f727d65e700 1 --2- 172.21.15.31:0/4169072826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7278108d50 0x7f7278109170 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.003+0000 7f727d65e700 1 -- 172.21.15.31:0/4169072826 >> 172.21.15.31:0/4169072826 conn(0x7f72781042b0 msgr2=0x7f7278106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.003+0000 7f727d65e700 1 -- 172.21.15.31:0/4169072826 shutdown_connections 2024-04-26T17:45:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.003+0000 7f727d65e700 1 -- 172.21.15.31:0/4169072826 wait complete. 2024-04-26T17:45:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.003+0000 7f727d65e700 1 Processor -- start 2024-04-26T17:45:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.004+0000 7f727d65e700 1 -- start start 2024-04-26T17:45:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.004+0000 7f727d65e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7278108d50 0x7f727807eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.004+0000 7f727d65e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f727810d300 con 0x7f7278108d50 2024-04-26T17:45:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.004+0000 7f7276ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7278108d50 0x7f727807eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.004+0000 7f7276ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7278108d50 0x7f727807eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53782/0 (socket says 172.21.15.31:53782) 2024-04-26T17:45:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.004+0000 7f7276ffd700 1 -- 172.21.15.31:0/3336952217 learned_addr learned my addr 172.21.15.31:0/3336952217 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.004+0000 7f7276ffd700 1 -- 172.21.15.31:0/3336952217 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f727807f0d0 con 0x7f7278108d50 2024-04-26T17:45:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.004+0000 7f7276ffd700 1 --2- 172.21.15.31:0/3336952217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7278108d50 0x7f727807eb90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f72680117b0 tx=0x7f7268011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.004+0000 7f725ffff700 1 -- 172.21.15.31:0/3336952217 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7268003af0 con 0x7f7278108d50 2024-04-26T17:45:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.004+0000 7f725ffff700 1 -- 172.21.15.31:0/3336952217 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7268020070 con 0x7f7278108d50 2024-04-26T17:45:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.004+0000 7f727d65e700 1 -- 172.21.15.31:0/3336952217 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f727807d2c0 con 0x7f7278108d50 2024-04-26T17:45:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.005+0000 7f725ffff700 1 -- 172.21.15.31:0/3336952217 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7268019630 con 0x7f7278108d50 2024-04-26T17:45:35.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.005+0000 7f727d65e700 1 -- 172.21.15.31:0/3336952217 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f727807d800 con 0x7f7278108d50 2024-04-26T17:45:35.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.005+0000 7f725ffff700 1 -- 172.21.15.31:0/3336952217 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f72680198a0 con 0x7f7278108d50 2024-04-26T17:45:35.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.006+0000 7f725ffff700 1 --2- 172.21.15.31:0/3336952217 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f7260039600 0x7f726003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:35.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.006+0000 7f727d65e700 1 -- 172.21.15.31:0/3336952217 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7278064a80 con 0x7f7278108d50 2024-04-26T17:45:35.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.006+0000 7f725ffff700 1 -- 172.21.15.31:0/3336952217 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f726804dde0 con 0x7f7278108d50 2024-04-26T17:45:35.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.006+0000 7f72767fc700 1 --2- 172.21.15.31:0/3336952217 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f7260039600 0x7f726003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:35.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.006+0000 7f72767fc700 1 --2- 172.21.15.31:0/3336952217 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f7260039600 0x7f726003bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f726c006fd0 tx=0x7f726c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:35.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.011+0000 7f725ffff700 1 -- 172.21.15.31:0/3336952217 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f726804d730 con 0x7f7278108d50 2024-04-26T17:45:35.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.245+0000 7f727d65e700 1 -- 172.21.15.31:0/3336952217 --> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f7278105590 con 0x7f7260039600 2024-04-26T17:45:35.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.253+0000 7f725ffff700 1 -- 172.21.15.31:0/3336952217 <== mgr.14118 v2:172.21.15.31:6800/153341213 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7278105590 con 0x7f7260039600 2024-04-26T17:45:35.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.255+0000 7f727d65e700 1 -- 172.21.15.31:0/3336952217 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f7260039600 msgr2=0x7f726003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:35.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.255+0000 7f727d65e700 1 --2- 172.21.15.31:0/3336952217 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f7260039600 0x7f726003bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f726c006fd0 tx=0x7f726c006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:35.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.255+0000 7f727d65e700 1 -- 172.21.15.31:0/3336952217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7278108d50 msgr2=0x7f727807eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:35.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.255+0000 7f727d65e700 1 --2- 172.21.15.31:0/3336952217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7278108d50 0x7f727807eb90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f72680117b0 tx=0x7f7268011b70 comp rx=0 tx=0).stop 2024-04-26T17:45:35.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.255+0000 7f727d65e700 1 -- 172.21.15.31:0/3336952217 shutdown_connections 2024-04-26T17:45:35.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.255+0000 7f727d65e700 1 --2- 172.21.15.31:0/3336952217 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f7260039600 0x7f726003bac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:35.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.255+0000 7f727d65e700 1 --2- 172.21.15.31:0/3336952217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7278108d50 0x7f727807eb90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:35.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.255+0000 7f727d65e700 1 -- 172.21.15.31:0/3336952217 >> 172.21.15.31:0/3336952217 conn(0x7f72781042b0 msgr2=0x7f7278104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:35.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.256+0000 7f727d65e700 1 -- 172.21.15.31:0/3336952217 shutdown_connections 2024-04-26T17:45:35.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:34.256+0000 7f727d65e700 1 -- 172.21.15.31:0/3336952217 wait complete. 2024-04-26T17:45:35.063 INFO:teuthology.orchestra.run.smithi031.stdout:Deploying unmanaged mgr service... 2024-04-26T17:45:35.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:35 smithi031 ceph-mon[29164]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:45:35.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:35 smithi031 ceph-mon[29164]: Saving service mon spec with placement count:5 2024-04-26T17:45:35.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:35 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' 2024-04-26T17:45:37.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-26T17:45:37.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.492+0000 7f1778a70700 1 Processor -- start 2024-04-26T17:45:37.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.492+0000 7f1778a70700 1 -- start start 2024-04-26T17:45:37.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.492+0000 7f1778a70700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1774109780 0x7f177410bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:37.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.492+0000 7f1778a70700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f177407ced0 con 0x7f1774109780 2024-04-26T17:45:37.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.492+0000 7f177259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1774109780 0x7f177410bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:37.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.493+0000 7f177259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1774109780 0x7f177410bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48152/0 (socket says 172.21.15.31:48152) 2024-04-26T17:45:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.493+0000 7f177259c700 1 -- 172.21.15.31:0/3054147166 learned_addr learned my addr 172.21.15.31:0/3054147166 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.493+0000 7f177259c700 1 -- 172.21.15.31:0/3054147166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177410c0b0 con 0x7f1774109780 2024-04-26T17:45:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.493+0000 7f177259c700 1 --2- 172.21.15.31:0/3054147166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1774109780 0x7f177410bb70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f17640098a0 tx=0x7f1764009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c135041d7a715e0d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.493+0000 7f177159a700 1 -- 172.21.15.31:0/3054147166 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1764004160 con 0x7f1774109780 2024-04-26T17:45:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.493+0000 7f177159a700 1 -- 172.21.15.31:0/3054147166 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1764004300 con 0x7f1774109780 2024-04-26T17:45:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.493+0000 7f177159a700 1 -- 172.21.15.31:0/3054147166 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1764011af0 con 0x7f1774109780 2024-04-26T17:45:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.494+0000 7f1778a70700 1 -- 172.21.15.31:0/3054147166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1774109780 msgr2=0x7f177410bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.494+0000 7f1778a70700 1 --2- 172.21.15.31:0/3054147166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1774109780 0x7f177410bb70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f17640098a0 tx=0x7f1764009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.494+0000 7f1778a70700 1 -- 172.21.15.31:0/3054147166 shutdown_connections 2024-04-26T17:45:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.494+0000 7f1778a70700 1 --2- 172.21.15.31:0/3054147166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1774109780 0x7f177410bb70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.494+0000 7f1778a70700 1 -- 172.21.15.31:0/3054147166 >> 172.21.15.31:0/3054147166 conn(0x7f1774103210 msgr2=0x7f1774105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.494+0000 7f1778a70700 1 -- 172.21.15.31:0/3054147166 shutdown_connections 2024-04-26T17:45:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.494+0000 7f1778a70700 1 -- 172.21.15.31:0/3054147166 wait complete. 2024-04-26T17:45:37.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.495+0000 7f1778a70700 1 Processor -- start 2024-04-26T17:45:37.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.495+0000 7f1778a70700 1 -- start start 2024-04-26T17:45:37.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.495+0000 7f1778a70700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1774109780 0x7f177419ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:37.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.495+0000 7f1778a70700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f177410d080 con 0x7f1774109780 2024-04-26T17:45:37.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.495+0000 7f177259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1774109780 0x7f177419ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:37.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.495+0000 7f177259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1774109780 0x7f177419ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48156/0 (socket says 172.21.15.31:48156) 2024-04-26T17:45:37.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.495+0000 7f177259c700 1 -- 172.21.15.31:0/4048580026 learned_addr learned my addr 172.21.15.31:0/4048580026 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:37.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.495+0000 7f177259c700 1 -- 172.21.15.31:0/4048580026 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177419d1f0 con 0x7f1774109780 2024-04-26T17:45:37.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.496+0000 7f177259c700 1 --2- 172.21.15.31:0/4048580026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1774109780 0x7f177419ccb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1764003f40 tx=0x7f1764011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:37.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.496+0000 7f17637fe700 1 -- 172.21.15.31:0/4048580026 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17640038b0 con 0x7f1774109780 2024-04-26T17:45:37.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.496+0000 7f1778a70700 1 -- 172.21.15.31:0/4048580026 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f177419d480 con 0x7f1774109780 2024-04-26T17:45:37.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.496+0000 7f1778a70700 1 -- 172.21.15.31:0/4048580026 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17741a0170 con 0x7f1774109780 2024-04-26T17:45:37.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.496+0000 7f17637fe700 1 -- 172.21.15.31:0/4048580026 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1764003a50 con 0x7f1774109780 2024-04-26T17:45:37.006 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.496+0000 7f17637fe700 1 -- 172.21.15.31:0/4048580026 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f176401a5c0 con 0x7f1774109780 2024-04-26T17:45:37.006 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.497+0000 7f1778a70700 1 -- 172.21.15.31:0/4048580026 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1754005360 con 0x7f1774109780 2024-04-26T17:45:37.006 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.497+0000 7f17637fe700 1 -- 172.21.15.31:0/4048580026 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f176401a870 con 0x7f1774109780 2024-04-26T17:45:37.006 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.497+0000 7f17637fe700 1 --2- 172.21.15.31:0/4048580026 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f175c0395b0 0x7f175c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:37.006 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.497+0000 7f17637fe700 1 -- 172.21.15.31:0/4048580026 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f176404da70 con 0x7f1774109780 2024-04-26T17:45:37.006 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.497+0000 7f1771d9b700 1 --2- 172.21.15.31:0/4048580026 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f175c0395b0 0x7f175c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:37.006 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.498+0000 7f1771d9b700 1 --2- 172.21.15.31:0/4048580026 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f175c0395b0 0x7f175c03ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1768006fd0 tx=0x7f1768006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:37.006 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.502+0000 7f17637fe700 1 -- 172.21.15.31:0/4048580026 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1764019c70 con 0x7f1774109780 2024-04-26T17:45:37.006 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.743+0000 7f1778a70700 1 -- 172.21.15.31:0/4048580026 --> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f1754000bf0 con 0x7f175c0395b0 2024-04-26T17:45:37.006 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.750+0000 7f17637fe700 1 -- 172.21.15.31:0/4048580026 <== mgr.14118 v2:172.21.15.31:6800/153341213 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1754000bf0 con 0x7f175c0395b0 2024-04-26T17:45:37.006 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.752+0000 7f1778a70700 1 -- 172.21.15.31:0/4048580026 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f175c0395b0 msgr2=0x7f175c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:37.006 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.752+0000 7f1778a70700 1 --2- 172.21.15.31:0/4048580026 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f175c0395b0 0x7f175c03ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1768006fd0 tx=0x7f1768006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:37.006 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.752+0000 7f1778a70700 1 -- 172.21.15.31:0/4048580026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1774109780 msgr2=0x7f177419ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:37.007 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.752+0000 7f1778a70700 1 --2- 172.21.15.31:0/4048580026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1774109780 0x7f177419ccb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1764003f40 tx=0x7f1764011890 comp rx=0 tx=0).stop 2024-04-26T17:45:37.007 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.753+0000 7f1778a70700 1 -- 172.21.15.31:0/4048580026 shutdown_connections 2024-04-26T17:45:37.007 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.753+0000 7f1778a70700 1 --2- 172.21.15.31:0/4048580026 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f175c0395b0 0x7f175c03ba70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:37.007 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.753+0000 7f1778a70700 1 --2- 172.21.15.31:0/4048580026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1774109780 0x7f177419ccb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:37.007 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.753+0000 7f1778a70700 1 -- 172.21.15.31:0/4048580026 >> 172.21.15.31:0/4048580026 conn(0x7f1774103210 msgr2=0x7f1774103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:37.007 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.753+0000 7f1778a70700 1 -- 172.21.15.31:0/4048580026 shutdown_connections 2024-04-26T17:45:37.007 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:35.753+0000 7f1778a70700 1 -- 172.21.15.31:0/4048580026 wait complete. 2024-04-26T17:45:37.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:36 smithi031 ceph-mon[29164]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:45:37.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:36 smithi031 ceph-mon[29164]: Saving service mgr spec with placement count:2 2024-04-26T17:45:37.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:36 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' 2024-04-26T17:45:37.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:36 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' 2024-04-26T17:45:38.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.450+0000 7fa185c86700 1 Processor -- start 2024-04-26T17:45:38.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.450+0000 7fa185c86700 1 -- start start 2024-04-26T17:45:38.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.450+0000 7fa185c86700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa180108d40 0x7fa180109160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:38.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.450+0000 7fa185c86700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa180109730 con 0x7fa180108d40 2024-04-26T17:45:38.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.451+0000 7fa17f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa180108d40 0x7fa180109160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:38.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.451+0000 7fa17f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa180108d40 0x7fa180109160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48160/0 (socket says 172.21.15.31:48160) 2024-04-26T17:45:38.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.451+0000 7fa17f7fe700 1 -- 172.21.15.31:0/4151188487 learned_addr learned my addr 172.21.15.31:0/4151188487 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:38.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.451+0000 7fa17f7fe700 1 -- 172.21.15.31:0/4151188487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa180109ef0 con 0x7fa180108d40 2024-04-26T17:45:38.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.451+0000 7fa17f7fe700 1 --2- 172.21.15.31:0/4151188487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa180108d40 0x7fa180109160 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa170009e80 tx=0x7fa17000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b7cbd5606f9281 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:38.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.451+0000 7fa17e7fc700 1 -- 172.21.15.31:0/4151188487 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa170017070 con 0x7fa180108d40 2024-04-26T17:45:38.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.451+0000 7fa17e7fc700 1 -- 172.21.15.31:0/4151188487 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa17000b8c0 con 0x7fa180108d40 2024-04-26T17:45:38.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.451+0000 7fa17e7fc700 1 -- 172.21.15.31:0/4151188487 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa170004490 con 0x7fa180108d40 2024-04-26T17:45:38.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.452+0000 7fa185c86700 1 -- 172.21.15.31:0/4151188487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa180108d40 msgr2=0x7fa180109160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:38.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.452+0000 7fa185c86700 1 --2- 172.21.15.31:0/4151188487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa180108d40 0x7fa180109160 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa170009e80 tx=0x7fa17000b330 comp rx=0 tx=0).stop 2024-04-26T17:45:38.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.452+0000 7fa185c86700 1 -- 172.21.15.31:0/4151188487 shutdown_connections 2024-04-26T17:45:38.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.452+0000 7fa185c86700 1 --2- 172.21.15.31:0/4151188487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa180108d40 0x7fa180109160 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:38.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.452+0000 7fa185c86700 1 -- 172.21.15.31:0/4151188487 >> 172.21.15.31:0/4151188487 conn(0x7fa180104300 msgr2=0x7fa180106720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:38.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.452+0000 7fa185c86700 1 -- 172.21.15.31:0/4151188487 shutdown_connections 2024-04-26T17:45:38.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.452+0000 7fa185c86700 1 -- 172.21.15.31:0/4151188487 wait complete. 2024-04-26T17:45:38.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.453+0000 7fa185c86700 1 Processor -- start 2024-04-26T17:45:38.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.453+0000 7fa185c86700 1 -- start start 2024-04-26T17:45:38.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.453+0000 7fa185c86700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa180108d40 0x7fa18007eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:38.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.453+0000 7fa185c86700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa18010d2a0 con 0x7fa180108d40 2024-04-26T17:45:38.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.453+0000 7fa17f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa180108d40 0x7fa18007eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:38.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.453+0000 7fa17f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa180108d40 0x7fa18007eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48174/0 (socket says 172.21.15.31:48174) 2024-04-26T17:45:38.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.453+0000 7fa17f7fe700 1 -- 172.21.15.31:0/2458189397 learned_addr learned my addr 172.21.15.31:0/2458189397 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:38.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.454+0000 7fa17f7fe700 1 -- 172.21.15.31:0/2458189397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa18007f0d0 con 0x7fa180108d40 2024-04-26T17:45:38.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.454+0000 7fa17f7fe700 1 --2- 172.21.15.31:0/2458189397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa180108d40 0x7fa18007eb90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa17000b7b0 tx=0x7fa17000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:38.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.454+0000 7fa17cff9700 1 -- 172.21.15.31:0/2458189397 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa170020020 con 0x7fa180108d40 2024-04-26T17:45:38.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.454+0000 7fa185c86700 1 -- 172.21.15.31:0/2458189397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa18007d2c0 con 0x7fa180108d40 2024-04-26T17:45:38.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.454+0000 7fa185c86700 1 -- 172.21.15.31:0/2458189397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa18007d800 con 0x7fa180108d40 2024-04-26T17:45:38.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.454+0000 7fa17cff9700 1 -- 172.21.15.31:0/2458189397 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa170003b50 con 0x7fa180108d40 2024-04-26T17:45:38.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.454+0000 7fa17cff9700 1 -- 172.21.15.31:0/2458189397 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa170017040 con 0x7fa180108d40 2024-04-26T17:45:38.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.455+0000 7fa17cff9700 1 -- 172.21.15.31:0/2458189397 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fa170023020 con 0x7fa180108d40 2024-04-26T17:45:38.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.455+0000 7fa185c86700 1 -- 172.21.15.31:0/2458189397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa180064a80 con 0x7fa180108d40 2024-04-26T17:45:38.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.455+0000 7fa17cff9700 1 --2- 172.21.15.31:0/2458189397 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7fa16c0395b0 0x7fa16c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:38.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.455+0000 7fa17cff9700 1 -- 172.21.15.31:0/2458189397 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa17000f050 con 0x7fa180108d40 2024-04-26T17:45:38.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.455+0000 7fa17effd700 1 --2- 172.21.15.31:0/2458189397 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7fa16c0395b0 0x7fa16c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:38.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.456+0000 7fa17effd700 1 --2- 172.21.15.31:0/2458189397 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7fa16c0395b0 0x7fa16c03ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa168006fd0 tx=0x7fa168006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:38.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.460+0000 7fa17cff9700 1 -- 172.21.15.31:0/2458189397 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa17002a020 con 0x7fa180108d40 2024-04-26T17:45:38.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.667+0000 7fa185c86700 1 -- 172.21.15.31:0/2458189397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fa18007df70 con 0x7fa180108d40 2024-04-26T17:45:38.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.670+0000 7fa17cff9700 1 -- 172.21.15.31:0/2458189397 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fa17002b260 con 0x7fa180108d40 2024-04-26T17:45:38.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.673+0000 7fa185c86700 1 -- 172.21.15.31:0/2458189397 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7fa16c0395b0 msgr2=0x7fa16c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:38.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.673+0000 7fa185c86700 1 --2- 172.21.15.31:0/2458189397 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7fa16c0395b0 0x7fa16c03ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa168006fd0 tx=0x7fa168006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:38.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.673+0000 7fa185c86700 1 -- 172.21.15.31:0/2458189397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa180108d40 msgr2=0x7fa18007eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:38.106 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.673+0000 7fa185c86700 1 --2- 172.21.15.31:0/2458189397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa180108d40 0x7fa18007eb90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa17000b7b0 tx=0x7fa17000bd30 comp rx=0 tx=0).stop 2024-04-26T17:45:38.106 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.673+0000 7fa185c86700 1 -- 172.21.15.31:0/2458189397 shutdown_connections 2024-04-26T17:45:38.106 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.673+0000 7fa185c86700 1 --2- 172.21.15.31:0/2458189397 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7fa16c0395b0 0x7fa16c03ba70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:38.106 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.673+0000 7fa185c86700 1 --2- 172.21.15.31:0/2458189397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa180108d40 0x7fa18007eb90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:38.106 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.673+0000 7fa185c86700 1 -- 172.21.15.31:0/2458189397 >> 172.21.15.31:0/2458189397 conn(0x7fa180104300 msgr2=0x7fa180104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:38.106 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.674+0000 7fa185c86700 1 -- 172.21.15.31:0/2458189397 shutdown_connections 2024-04-26T17:45:38.106 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:37.674+0000 7fa185c86700 1 -- 172.21.15.31:0/2458189397 wait complete. 2024-04-26T17:45:38.347 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:38 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' 2024-04-26T17:45:38.347 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:38 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2458189397' entity='client.admin' 2024-04-26T17:45:39.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.545+0000 7f63eccb6700 1 Processor -- start 2024-04-26T17:45:39.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.545+0000 7f63eccb6700 1 -- start start 2024-04-26T17:45:39.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.545+0000 7f63eccb6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108e00 0x7f63e8109220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:39.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.545+0000 7f63eccb6700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f63e81097f0 con 0x7f63e8108e00 2024-04-26T17:45:39.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.545+0000 7f63e659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108e00 0x7f63e8109220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:39.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.545+0000 7f63e659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108e00 0x7f63e8109220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48186/0 (socket says 172.21.15.31:48186) 2024-04-26T17:45:39.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.545+0000 7f63e659c700 1 -- 172.21.15.31:0/3360927228 learned_addr learned my addr 172.21.15.31:0/3360927228 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:39.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.546+0000 7f63e659c700 1 -- 172.21.15.31:0/3360927228 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e8109f60 con 0x7f63e8108e00 2024-04-26T17:45:39.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.546+0000 7f63e659c700 1 --2- 172.21.15.31:0/3360927228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108e00 0x7f63e8109220 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f63d80098a0 tx=0x7f63d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5dec7a392922e19 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:39.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.546+0000 7f63e559a700 1 -- 172.21.15.31:0/3360927228 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63d8004160 con 0x7f63e8108e00 2024-04-26T17:45:39.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.546+0000 7f63e559a700 1 -- 172.21.15.31:0/3360927228 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d8004300 con 0x7f63e8108e00 2024-04-26T17:45:39.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.546+0000 7f63e559a700 1 -- 172.21.15.31:0/3360927228 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63d8011a20 con 0x7f63e8108e00 2024-04-26T17:45:39.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.546+0000 7f63eccb6700 1 -- 172.21.15.31:0/3360927228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108e00 msgr2=0x7f63e8109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:39.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.546+0000 7f63eccb6700 1 --2- 172.21.15.31:0/3360927228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108e00 0x7f63e8109220 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f63d80098a0 tx=0x7f63d8009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:39.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.546+0000 7f63eccb6700 1 -- 172.21.15.31:0/3360927228 shutdown_connections 2024-04-26T17:45:39.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.546+0000 7f63eccb6700 1 --2- 172.21.15.31:0/3360927228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108e00 0x7f63e8109220 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:39.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.547+0000 7f63eccb6700 1 -- 172.21.15.31:0/3360927228 >> 172.21.15.31:0/3360927228 conn(0x7f63e81043c0 msgr2=0x7f63e81067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:39.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.547+0000 7f63eccb6700 1 -- 172.21.15.31:0/3360927228 shutdown_connections 2024-04-26T17:45:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.547+0000 7f63eccb6700 1 -- 172.21.15.31:0/3360927228 wait complete. 2024-04-26T17:45:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.547+0000 7f63eccb6700 1 Processor -- start 2024-04-26T17:45:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.547+0000 7f63eccb6700 1 -- start start 2024-04-26T17:45:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.548+0000 7f63eccb6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108e00 0x7f63e819cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.548+0000 7f63eccb6700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f63e810d310 con 0x7f63e8108e00 2024-04-26T17:45:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.548+0000 7f63e659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108e00 0x7f63e819cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.548+0000 7f63e659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108e00 0x7f63e819cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48196/0 (socket says 172.21.15.31:48196) 2024-04-26T17:45:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.548+0000 7f63e659c700 1 -- 172.21.15.31:0/3597768174 learned_addr learned my addr 172.21.15.31:0/3597768174 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.548+0000 7f63e659c700 1 -- 172.21.15.31:0/3597768174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e819d1d0 con 0x7f63e8108e00 2024-04-26T17:45:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.548+0000 7f63e659c700 1 --2- 172.21.15.31:0/3597768174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108e00 0x7f63e819cc90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f63d80119f0 tx=0x7f63d8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.548+0000 7f63d77fe700 1 -- 172.21.15.31:0/3597768174 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63d80039e0 con 0x7f63e8108e00 2024-04-26T17:45:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.548+0000 7f63d77fe700 1 -- 172.21.15.31:0/3597768174 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d801b8f0 con 0x7f63e8108e00 2024-04-26T17:45:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.548+0000 7f63d77fe700 1 -- 172.21.15.31:0/3597768174 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63d80237a0 con 0x7f63e8108e00 2024-04-26T17:45:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.548+0000 7f63eccb6700 1 -- 172.21.15.31:0/3597768174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63e819d460 con 0x7f63e8108e00 2024-04-26T17:45:39.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.548+0000 7f63eccb6700 1 -- 172.21.15.31:0/3597768174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63e81a0150 con 0x7f63e8108e00 2024-04-26T17:45:39.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.549+0000 7f63d77fe700 1 -- 172.21.15.31:0/3597768174 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f63d801b440 con 0x7f63e8108e00 2024-04-26T17:45:39.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.549+0000 7f63eccb6700 1 -- 172.21.15.31:0/3597768174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63e8064a80 con 0x7f63e8108e00 2024-04-26T17:45:39.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.549+0000 7f63d77fe700 1 --2- 172.21.15.31:0/3597768174 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f63d00395b0 0x7f63d003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:39.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.549+0000 7f63d77fe700 1 -- 172.21.15.31:0/3597768174 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f63d804dde0 con 0x7f63e8108e00 2024-04-26T17:45:39.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.550+0000 7f63e5d9b700 1 --2- 172.21.15.31:0/3597768174 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f63d00395b0 0x7f63d003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:39.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.550+0000 7f63e5d9b700 1 --2- 172.21.15.31:0/3597768174 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f63d00395b0 0x7f63d003ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f63dc006fd0 tx=0x7f63dc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:39.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.555+0000 7f63d77fe700 1 -- 172.21.15.31:0/3597768174 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f63d801f030 con 0x7f63e8108e00 2024-04-26T17:45:39.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.777+0000 7f63eccb6700 1 -- 172.21.15.31:0/3597768174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f63e81a03c0 con 0x7f63e8108e00 2024-04-26T17:45:39.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.780+0000 7f63d77fe700 1 -- 172.21.15.31:0/3597768174 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f63d8021dc0 con 0x7f63e8108e00 2024-04-26T17:45:39.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.782+0000 7f63eccb6700 1 -- 172.21.15.31:0/3597768174 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f63d00395b0 msgr2=0x7f63d003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:39.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.782+0000 7f63eccb6700 1 --2- 172.21.15.31:0/3597768174 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f63d00395b0 0x7f63d003ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f63dc006fd0 tx=0x7f63dc006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:39.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.782+0000 7f63eccb6700 1 -- 172.21.15.31:0/3597768174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108e00 msgr2=0x7f63e819cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:39.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.782+0000 7f63eccb6700 1 --2- 172.21.15.31:0/3597768174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108e00 0x7f63e819cc90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f63d80119f0 tx=0x7f63d8011ad0 comp rx=0 tx=0).stop 2024-04-26T17:45:39.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.782+0000 7f63eccb6700 1 -- 172.21.15.31:0/3597768174 shutdown_connections 2024-04-26T17:45:39.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.782+0000 7f63eccb6700 1 --2- 172.21.15.31:0/3597768174 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f63d00395b0 0x7f63d003ba70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:39.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.782+0000 7f63eccb6700 1 --2- 172.21.15.31:0/3597768174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108e00 0x7f63e819cc90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:39.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.782+0000 7f63eccb6700 1 -- 172.21.15.31:0/3597768174 >> 172.21.15.31:0/3597768174 conn(0x7f63e81043c0 msgr2=0x7f63e8104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:39.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.782+0000 7f63eccb6700 1 -- 172.21.15.31:0/3597768174 shutdown_connections 2024-04-26T17:45:39.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:38.782+0000 7f63eccb6700 1 -- 172.21.15.31:0/3597768174 wait complete. 2024-04-26T17:45:39.497 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling the dashboard module... 2024-04-26T17:45:40.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:39 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3597768174' entity='client.admin' 2024-04-26T17:45:41.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:40 smithi031 ceph-mon[29164]: from='mgr.14118 172.21.15.31:0/733593669' entity='mgr.x' 2024-04-26T17:45:41.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:40 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1318649555' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T17:45:41.310 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:41 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: ignoring --setuser ceph since I am not root 2024-04-26T17:45:41.310 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:41 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: ignoring --setgroup ceph since I am not root 2024-04-26T17:45:41.310 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:41 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:41.190+0000 7fe635d0b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T17:45:41.600 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:41 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:41.387+0000 7fe635d0b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:45:41.600 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:41 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:41.494+0000 7fe635d0b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T17:45:41.608 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.195+0000 7f47fa633700 1 Processor -- start 2024-04-26T17:45:41.608 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.195+0000 7f47fa633700 1 -- start start 2024-04-26T17:45:41.608 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.195+0000 7f47fa633700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f4107c10 0x7f47f4108030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:41.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.195+0000 7f47fa633700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f47f4108600 con 0x7f47f4107c10 2024-04-26T17:45:41.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.195+0000 7f47f3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f4107c10 0x7f47f4108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:41.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.195+0000 7f47f3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f4107c10 0x7f47f4108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48204/0 (socket says 172.21.15.31:48204) 2024-04-26T17:45:41.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.195+0000 7f47f3fff700 1 -- 172.21.15.31:0/2065704918 learned_addr learned my addr 172.21.15.31:0/2065704918 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:41.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.195+0000 7f47f3fff700 1 -- 172.21.15.31:0/2065704918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47f4108e10 con 0x7f47f4107c10 2024-04-26T17:45:41.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.195+0000 7f47f3fff700 1 --2- 172.21.15.31:0/2065704918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f4107c10 0x7f47f4108030 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f47e40098a0 tx=0x7f47e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8ba9851620c04a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:41.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.196+0000 7f47f2ffd700 1 -- 172.21.15.31:0/2065704918 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47e4004160 con 0x7f47f4107c10 2024-04-26T17:45:41.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.196+0000 7f47f2ffd700 1 -- 172.21.15.31:0/2065704918 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47e4004300 con 0x7f47f4107c10 2024-04-26T17:45:41.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.196+0000 7f47f2ffd700 1 -- 172.21.15.31:0/2065704918 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47e4011a20 con 0x7f47f4107c10 2024-04-26T17:45:41.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.196+0000 7f47fa633700 1 -- 172.21.15.31:0/2065704918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f4107c10 msgr2=0x7f47f4108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:41.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.196+0000 7f47fa633700 1 --2- 172.21.15.31:0/2065704918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f4107c10 0x7f47f4108030 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f47e40098a0 tx=0x7f47e4009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:41.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.196+0000 7f47fa633700 1 -- 172.21.15.31:0/2065704918 shutdown_connections 2024-04-26T17:45:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.196+0000 7f47fa633700 1 --2- 172.21.15.31:0/2065704918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f4107c10 0x7f47f4108030 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.196+0000 7f47fa633700 1 -- 172.21.15.31:0/2065704918 >> 172.21.15.31:0/2065704918 conn(0x7f47f4103210 msgr2=0x7f47f4105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.196+0000 7f47fa633700 1 -- 172.21.15.31:0/2065704918 shutdown_connections 2024-04-26T17:45:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.196+0000 7f47fa633700 1 -- 172.21.15.31:0/2065704918 wait complete. 2024-04-26T17:45:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.197+0000 7f47fa633700 1 Processor -- start 2024-04-26T17:45:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.197+0000 7f47fa633700 1 -- start start 2024-04-26T17:45:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.197+0000 7f47fa633700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f4107c10 0x7f47f419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.197+0000 7f47fa633700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f47f410c1c0 con 0x7f47f4107c10 2024-04-26T17:45:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.198+0000 7f47f3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f4107c10 0x7f47f419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.198+0000 7f47f3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f4107c10 0x7f47f419cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48212/0 (socket says 172.21.15.31:48212) 2024-04-26T17:45:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.198+0000 7f47f3fff700 1 -- 172.21.15.31:0/1318649555 learned_addr learned my addr 172.21.15.31:0/1318649555 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.198+0000 7f47f3fff700 1 -- 172.21.15.31:0/1318649555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47f419d1c0 con 0x7f47f4107c10 2024-04-26T17:45:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.198+0000 7f47f3fff700 1 --2- 172.21.15.31:0/1318649555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f4107c10 0x7f47f419cc80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009520 tx=0x7f47e4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.198+0000 7f47f17fa700 1 -- 172.21.15.31:0/1318649555 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47e40039e0 con 0x7f47f4107c10 2024-04-26T17:45:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.198+0000 7f47f17fa700 1 -- 172.21.15.31:0/1318649555 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47e401b8f0 con 0x7f47f4107c10 2024-04-26T17:45:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.198+0000 7f47fa633700 1 -- 172.21.15.31:0/1318649555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47f419d450 con 0x7f47f4107c10 2024-04-26T17:45:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.198+0000 7f47f17fa700 1 -- 172.21.15.31:0/1318649555 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47e4019770 con 0x7f47f4107c10 2024-04-26T17:45:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.198+0000 7f47fa633700 1 -- 172.21.15.31:0/1318649555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47f41a0140 con 0x7f47f4107c10 2024-04-26T17:45:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.199+0000 7f47f17fa700 1 -- 172.21.15.31:0/1318649555 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f47e4019a70 con 0x7f47f4107c10 2024-04-26T17:45:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.199+0000 7f47f17fa700 1 --2- 172.21.15.31:0/1318649555 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f47dc0395b0 0x7f47dc03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.199+0000 7f47fa633700 1 -- 172.21.15.31:0/1318649555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47f4064a80 con 0x7f47f4107c10 2024-04-26T17:45:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.199+0000 7f47f17fa700 1 -- 172.21.15.31:0/1318649555 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f47e404df00 con 0x7f47f4107c10 2024-04-26T17:45:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.199+0000 7f47f37fe700 1 --2- 172.21.15.31:0/1318649555 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f47dc0395b0 0x7f47dc03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.200+0000 7f47f37fe700 1 --2- 172.21.15.31:0/1318649555 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f47dc0395b0 0x7f47dc03ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f47e8006fd0 tx=0x7f47e8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.205+0000 7f47f17fa700 1 -- 172.21.15.31:0/1318649555 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f47e401f030 con 0x7f47f4107c10 2024-04-26T17:45:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:40.475+0000 7f47fa633700 1 -- 172.21.15.31:0/1318649555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f47f402e6d0 con 0x7f47f4107c10 2024-04-26T17:45:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:41.000+0000 7f47f17fa700 1 -- 172.21.15.31:0/1318649555 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f47e40172b0 con 0x7f47f4107c10 2024-04-26T17:45:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:41.002+0000 7f47f17fa700 1 -- 172.21.15.31:0/1318649555 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f47e404c9b0 con 0x7f47f4107c10 2024-04-26T17:45:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:41.004+0000 7f47fa633700 1 -- 172.21.15.31:0/1318649555 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f47dc0395b0 msgr2=0x7f47dc03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:41.004+0000 7f47fa633700 1 --2- 172.21.15.31:0/1318649555 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f47dc0395b0 0x7f47dc03ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f47e8006fd0 tx=0x7f47e8006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:41.004+0000 7f47fa633700 1 -- 172.21.15.31:0/1318649555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f4107c10 msgr2=0x7f47f419cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:41.004+0000 7f47fa633700 1 --2- 172.21.15.31:0/1318649555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f4107c10 0x7f47f419cc80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009520 tx=0x7f47e4011aa0 comp rx=0 tx=0).stop 2024-04-26T17:45:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:41.004+0000 7f47fa633700 1 -- 172.21.15.31:0/1318649555 shutdown_connections 2024-04-26T17:45:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:41.004+0000 7f47fa633700 1 --2- 172.21.15.31:0/1318649555 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f47dc0395b0 0x7f47dc03ba70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:41.004+0000 7f47fa633700 1 --2- 172.21.15.31:0/1318649555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f4107c10 0x7f47f419cc80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:41.004+0000 7f47fa633700 1 -- 172.21.15.31:0/1318649555 >> 172.21.15.31:0/1318649555 conn(0x7f47f4103210 msgr2=0x7f47f4104050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:41.004+0000 7f47fa633700 1 -- 172.21.15.31:0/1318649555 shutdown_connections 2024-04-26T17:45:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:41.004+0000 7f47fa633700 1 -- 172.21.15.31:0/1318649555 wait complete. 2024-04-26T17:45:41.920 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:41 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:41.835+0000 7fe635d0b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T17:45:42.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:42 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1318649555' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T17:45:42.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:42 smithi031 ceph-mon[29164]: mgrmap e9: x(active, since 19s) 2024-04-26T17:45:42.230 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:41 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:41.924+0000 7fe635d0b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T17:45:42.915 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-26T17:45:42.915 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-26T17:45:42.916 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T17:45:42.916 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T17:45:42.916 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T17:45:42.916 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-26T17:45:42.916 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.025+0000 7f5c645cc700 1 Processor -- start 2024-04-26T17:45:42.916 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.025+0000 7f5c645cc700 1 -- start start 2024-04-26T17:45:42.916 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.025+0000 7f5c645cc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c5c108e20 0x7f5c5c109240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:42.916 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.025+0000 7f5c645cc700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5c109810 con 0x7f5c5c108e20 2024-04-26T17:45:42.916 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.025+0000 7f5c62368700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c5c108e20 0x7f5c5c109240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:42.916 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.025+0000 7f5c62368700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c5c108e20 0x7f5c5c109240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48236/0 (socket says 172.21.15.31:48236) 2024-04-26T17:45:42.916 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.025+0000 7f5c62368700 1 -- 172.21.15.31:0/2249334713 learned_addr learned my addr 172.21.15.31:0/2249334713 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:42.917 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.025+0000 7f5c62368700 1 -- 172.21.15.31:0/2249334713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c5c109f80 con 0x7f5c5c108e20 2024-04-26T17:45:42.917 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.026+0000 7f5c62368700 1 --2- 172.21.15.31:0/2249334713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c5c108e20 0x7f5c5c109240 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5c580098a0 tx=0x7f5c58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fae033e53fb3d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:42.917 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.026+0000 7f5c61366700 1 -- 172.21.15.31:0/2249334713 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c58004160 con 0x7f5c5c108e20 2024-04-26T17:45:42.917 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.026+0000 7f5c61366700 1 -- 172.21.15.31:0/2249334713 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c58004300 con 0x7f5c5c108e20 2024-04-26T17:45:42.917 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.026+0000 7f5c61366700 1 -- 172.21.15.31:0/2249334713 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c58011a20 con 0x7f5c5c108e20 2024-04-26T17:45:42.917 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.026+0000 7f5c645cc700 1 -- 172.21.15.31:0/2249334713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c5c108e20 msgr2=0x7f5c5c109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:42.917 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.026+0000 7f5c645cc700 1 --2- 172.21.15.31:0/2249334713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c5c108e20 0x7f5c5c109240 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5c580098a0 tx=0x7f5c58009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:42.917 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.026+0000 7f5c645cc700 1 -- 172.21.15.31:0/2249334713 shutdown_connections 2024-04-26T17:45:42.917 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.026+0000 7f5c645cc700 1 --2- 172.21.15.31:0/2249334713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c5c108e20 0x7f5c5c109240 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:42.917 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.026+0000 7f5c645cc700 1 -- 172.21.15.31:0/2249334713 >> 172.21.15.31:0/2249334713 conn(0x7f5c5c104400 msgr2=0x7f5c5c106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:42.917 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.026+0000 7f5c645cc700 1 -- 172.21.15.31:0/2249334713 shutdown_connections 2024-04-26T17:45:42.917 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.027+0000 7f5c645cc700 1 -- 172.21.15.31:0/2249334713 wait complete. 2024-04-26T17:45:42.917 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.027+0000 7f5c645cc700 1 Processor -- start 2024-04-26T17:45:42.918 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.027+0000 7f5c645cc700 1 -- start start 2024-04-26T17:45:42.918 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.027+0000 7f5c645cc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c5c108e20 0x7f5c5c19cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:42.918 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.027+0000 7f5c645cc700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5c10d330 con 0x7f5c5c108e20 2024-04-26T17:45:42.918 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.027+0000 7f5c62368700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c5c108e20 0x7f5c5c19cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:42.918 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.027+0000 7f5c62368700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c5c108e20 0x7f5c5c19cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48246/0 (socket says 172.21.15.31:48246) 2024-04-26T17:45:42.918 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.027+0000 7f5c62368700 1 -- 172.21.15.31:0/63623403 learned_addr learned my addr 172.21.15.31:0/63623403 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:42.918 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.028+0000 7f5c62368700 1 -- 172.21.15.31:0/63623403 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c5c19d1e0 con 0x7f5c5c108e20 2024-04-26T17:45:42.918 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.028+0000 7f5c62368700 1 --2- 172.21.15.31:0/63623403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c5c108e20 0x7f5c5c19cca0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5c580119f0 tx=0x7f5c58011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:42.918 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.028+0000 7f5c537fe700 1 -- 172.21.15.31:0/63623403 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c580039e0 con 0x7f5c5c108e20 2024-04-26T17:45:42.918 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.028+0000 7f5c537fe700 1 -- 172.21.15.31:0/63623403 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c5801b8f0 con 0x7f5c5c108e20 2024-04-26T17:45:42.918 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.028+0000 7f5c645cc700 1 -- 172.21.15.31:0/63623403 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c5c19d470 con 0x7f5c5c108e20 2024-04-26T17:45:42.918 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.028+0000 7f5c537fe700 1 -- 172.21.15.31:0/63623403 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c580237a0 con 0x7f5c5c108e20 2024-04-26T17:45:42.919 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.028+0000 7f5c645cc700 1 -- 172.21.15.31:0/63623403 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c5c1a0160 con 0x7f5c5c108e20 2024-04-26T17:45:42.919 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.029+0000 7f5c537fe700 1 -- 172.21.15.31:0/63623403 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f5c5802c3f0 con 0x7f5c5c108e20 2024-04-26T17:45:42.919 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.029+0000 7f5c645cc700 1 -- 172.21.15.31:0/63623403 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c5c064a80 con 0x7f5c5c108e20 2024-04-26T17:45:42.919 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.029+0000 7f5c537fe700 1 --2- 172.21.15.31:0/63623403 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f5c48039610 0x7f5c4803bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:42.919 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.029+0000 7f5c537fe700 1 -- 172.21.15.31:0/63623403 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5c5804e050 con 0x7f5c5c108e20 2024-04-26T17:45:42.919 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.029+0000 7f5c61b67700 1 -- 172.21.15.31:0/63623403 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f5c48039610 msgr2=0x7f5c4803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/153341213 2024-04-26T17:45:42.919 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.029+0000 7f5c61b67700 1 --2- 172.21.15.31:0/63623403 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f5c48039610 0x7f5c4803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T17:45:42.919 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.034+0000 7f5c537fe700 1 -- 172.21.15.31:0/63623403 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5c5801f030 con 0x7f5c5c108e20 2024-04-26T17:45:42.919 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.230+0000 7f5c61b67700 1 -- 172.21.15.31:0/63623403 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f5c48039610 msgr2=0x7f5c4803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/153341213 2024-04-26T17:45:42.919 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.230+0000 7f5c61b67700 1 --2- 172.21.15.31:0/63623403 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f5c48039610 0x7f5c4803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T17:45:42.919 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.305+0000 7f5c645cc700 1 -- 172.21.15.31:0/63623403 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f5c5c1a0710 con 0x7f5c5c108e20 2024-04-26T17:45:42.919 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.306+0000 7f5c537fe700 1 -- 172.21.15.31:0/63623403 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f5c5801b440 con 0x7f5c5c108e20 2024-04-26T17:45:42.919 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.308+0000 7f5c645cc700 1 -- 172.21.15.31:0/63623403 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f5c48039610 msgr2=0x7f5c4803bad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T17:45:42.919 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.308+0000 7f5c645cc700 1 --2- 172.21.15.31:0/63623403 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f5c48039610 0x7f5c4803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:42.920 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.308+0000 7f5c645cc700 1 -- 172.21.15.31:0/63623403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c5c108e20 msgr2=0x7f5c5c19cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:42.920 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.308+0000 7f5c645cc700 1 --2- 172.21.15.31:0/63623403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c5c108e20 0x7f5c5c19cca0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5c580119f0 tx=0x7f5c58011ad0 comp rx=0 tx=0).stop 2024-04-26T17:45:42.920 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.308+0000 7f5c645cc700 1 -- 172.21.15.31:0/63623403 shutdown_connections 2024-04-26T17:45:42.920 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.308+0000 7f5c645cc700 1 --2- 172.21.15.31:0/63623403 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f5c48039610 0x7f5c4803bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:42.920 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.308+0000 7f5c645cc700 1 --2- 172.21.15.31:0/63623403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c5c108e20 0x7f5c5c19cca0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:42.920 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.308+0000 7f5c645cc700 1 -- 172.21.15.31:0/63623403 >> 172.21.15.31:0/63623403 conn(0x7f5c5c104400 msgr2=0x7f5c5c104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:42.920 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.308+0000 7f5c645cc700 1 -- 172.21.15.31:0/63623403 shutdown_connections 2024-04-26T17:45:42.920 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:42.308+0000 7f5c645cc700 1 -- 172.21.15.31:0/63623403 wait complete. 2024-04-26T17:45:42.920 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for the mgr to restart... 2024-04-26T17:45:42.920 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr epoch 9... 2024-04-26T17:45:43.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:43 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/63623403' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T17:45:43.309 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:43.200+0000 7fe635d0b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T17:45:43.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:43.384+0000 7fe635d0b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T17:45:43.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:43.561+0000 7fe635d0b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T17:45:44.172 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:43.864+0000 7fe635d0b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T17:45:44.173 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:43.961+0000 7fe635d0b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T17:45:44.173 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:44 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:44.172+0000 7fe635d0b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T17:45:44.559 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:44 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:44.374+0000 7fe635d0b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:45:45.309 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:45 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:45.026+0000 7fe635d0b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T17:45:45.783 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:45 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:45.467+0000 7fe635d0b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T17:45:45.783 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:45 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:45.570+0000 7fe635d0b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T17:45:45.783 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:45 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:45.783+0000 7fe635d0b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T17:45:46.683 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:46 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:46.331+0000 7fe635d0b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T17:45:46.683 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:46 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:46.428+0000 7fe635d0b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T17:45:47.044 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:46 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:46.683+0000 7fe635d0b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T17:45:47.044 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:46 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:46.823+0000 7fe635d0b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T17:45:47.044 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:46 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:46.923+0000 7fe635d0b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T17:45:47.044 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:47 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:47.044+0000 7fe635d0b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T17:45:47.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:47 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:47.355+0000 7fe635d0b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T17:45:48.269 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:47 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:47.989+0000 7fe635d0b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T17:45:48.269 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:45:48 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:45:48.216+0000 7fe635d0b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T17:45:48.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:48 smithi031 ceph-mon[29164]: Active manager daemon x restarted 2024-04-26T17:45:48.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:48 smithi031 ceph-mon[29164]: Activating manager daemon x 2024-04-26T17:45:48.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:48 smithi031 ceph-mon[29164]: osdmap e3: 0 total, 0 up, 0 in 2024-04-26T17:45:48.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:48 smithi031 ceph-mon[29164]: mgrmap e10: x(active, starting, since 0.00344754s) 2024-04-26T17:45:48.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:45:48.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T17:45:48.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T17:45:48.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T17:45:48.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T17:45:48.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:48 smithi031 ceph-mon[29164]: Manager daemon x is now available 2024-04-26T17:45:48.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:45:49.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T17:45:49.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T17:45:49.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:49 smithi031 ceph-mon[29164]: mgrmap e11: x(active, since 1.00578s) 2024-04-26T17:45:50.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-26T17:45:50.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-26T17:45:50.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T17:45:50.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-26T17:45:50.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.312+0000 7f28e4ae3700 1 Processor -- start 2024-04-26T17:45:50.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.312+0000 7f28e4ae3700 1 -- start start 2024-04-26T17:45:50.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.313+0000 7f28e4ae3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e0107c10 0x7f28e0108030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:50.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.313+0000 7f28e4ae3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28e0108600 con 0x7f28e0107c10 2024-04-26T17:45:50.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.313+0000 7f28de59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e0107c10 0x7f28e0108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:50.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.313+0000 7f28de59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e0107c10 0x7f28e0108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48256/0 (socket says 172.21.15.31:48256) 2024-04-26T17:45:50.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.313+0000 7f28de59c700 1 -- 172.21.15.31:0/612112322 learned_addr learned my addr 172.21.15.31:0/612112322 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:50.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.313+0000 7f28de59c700 1 -- 172.21.15.31:0/612112322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e0108e10 con 0x7f28e0107c10 2024-04-26T17:45:50.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.313+0000 7f28de59c700 1 --2- 172.21.15.31:0/612112322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e0107c10 0x7f28e0108030 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f28d00098a0 tx=0x7f28d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbbe65c1335c4b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:50.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.313+0000 7f28dd59a700 1 -- 172.21.15.31:0/612112322 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d0004160 con 0x7f28e0107c10 2024-04-26T17:45:50.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.314+0000 7f28dd59a700 1 -- 172.21.15.31:0/612112322 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28d0004300 con 0x7f28e0107c10 2024-04-26T17:45:50.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.314+0000 7f28dd59a700 1 -- 172.21.15.31:0/612112322 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d0011a20 con 0x7f28e0107c10 2024-04-26T17:45:50.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.314+0000 7f28e4ae3700 1 -- 172.21.15.31:0/612112322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e0107c10 msgr2=0x7f28e0108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:50.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.314+0000 7f28e4ae3700 1 --2- 172.21.15.31:0/612112322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e0107c10 0x7f28e0108030 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f28d00098a0 tx=0x7f28d0009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:50.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.314+0000 7f28e4ae3700 1 -- 172.21.15.31:0/612112322 shutdown_connections 2024-04-26T17:45:50.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.314+0000 7f28e4ae3700 1 --2- 172.21.15.31:0/612112322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e0107c10 0x7f28e0108030 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:50.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.314+0000 7f28e4ae3700 1 -- 172.21.15.31:0/612112322 >> 172.21.15.31:0/612112322 conn(0x7f28e0103210 msgr2=0x7f28e0105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:50.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.314+0000 7f28e4ae3700 1 -- 172.21.15.31:0/612112322 shutdown_connections 2024-04-26T17:45:50.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.314+0000 7f28e4ae3700 1 -- 172.21.15.31:0/612112322 wait complete. 2024-04-26T17:45:50.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.315+0000 7f28e4ae3700 1 Processor -- start 2024-04-26T17:45:50.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.315+0000 7f28e4ae3700 1 -- start start 2024-04-26T17:45:50.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.315+0000 7f28e4ae3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e0107c10 0x7f28e019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:50.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.315+0000 7f28e4ae3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28e010c1c0 con 0x7f28e0107c10 2024-04-26T17:45:50.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.315+0000 7f28de59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e0107c10 0x7f28e019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:50.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.315+0000 7f28de59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e0107c10 0x7f28e019cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48270/0 (socket says 172.21.15.31:48270) 2024-04-26T17:45:50.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.315+0000 7f28de59c700 1 -- 172.21.15.31:0/1612176093 learned_addr learned my addr 172.21.15.31:0/1612176093 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:50.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.316+0000 7f28de59c700 1 -- 172.21.15.31:0/1612176093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e019d1c0 con 0x7f28e0107c10 2024-04-26T17:45:50.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.316+0000 7f28de59c700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e0107c10 0x7f28e019cc80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f28d0009520 tx=0x7f28d0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:50.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.316+0000 7f28cf7fe700 1 -- 172.21.15.31:0/1612176093 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d00039e0 con 0x7f28e0107c10 2024-04-26T17:45:50.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.316+0000 7f28cf7fe700 1 -- 172.21.15.31:0/1612176093 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28d001b8f0 con 0x7f28e0107c10 2024-04-26T17:45:50.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.316+0000 7f28e4ae3700 1 -- 172.21.15.31:0/1612176093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e019d450 con 0x7f28e0107c10 2024-04-26T17:45:50.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.316+0000 7f28cf7fe700 1 -- 172.21.15.31:0/1612176093 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d0019770 con 0x7f28e0107c10 2024-04-26T17:45:50.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.316+0000 7f28e4ae3700 1 -- 172.21.15.31:0/1612176093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e01a0140 con 0x7f28e0107c10 2024-04-26T17:45:50.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.317+0000 7f28cf7fe700 1 -- 172.21.15.31:0/1612176093 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f28d0019a70 con 0x7f28e0107c10 2024-04-26T17:45:50.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.317+0000 7f28cf7fe700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f28c8039600 0x7f28c803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:50.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.317+0000 7f28e4ae3700 1 -- 172.21.15.31:0/1612176093 --> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f28e006a530 con 0x7f28c8039600 2024-04-26T17:45:50.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.317+0000 7f28cf7fe700 1 -- 172.21.15.31:0/1612176093 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f28d004df20 con 0x7f28e0107c10 2024-04-26T17:45:50.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.317+0000 7f28ddd9b700 1 -- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f28c8039600 msgr2=0x7f28c803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/153341213 2024-04-26T17:45:50.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.317+0000 7f28ddd9b700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f28c8039600 0x7f28c803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T17:45:50.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.518+0000 7f28ddd9b700 1 -- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f28c8039600 msgr2=0x7f28c803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/153341213 2024-04-26T17:45:50.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.518+0000 7f28ddd9b700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f28c8039600 0x7f28c803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T17:45:50.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.919+0000 7f28ddd9b700 1 -- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f28c8039600 msgr2=0x7f28c803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/153341213 2024-04-26T17:45:50.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:43.919+0000 7f28ddd9b700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f28c8039600 0x7f28c803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T17:45:50.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:44.720+0000 7f28ddd9b700 1 -- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f28c8039600 msgr2=0x7f28c803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/153341213 2024-04-26T17:45:50.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:44.720+0000 7f28ddd9b700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f28c8039600 0x7f28c803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T17:45:50.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:46.321+0000 7f28ddd9b700 1 -- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f28c8039600 msgr2=0x7f28c803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/153341213 2024-04-26T17:45:50.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:46.321+0000 7f28ddd9b700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f28c8039600 0x7f28c803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T17:45:50.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:48.219+0000 7f28cf7fe700 1 -- 172.21.15.31:0/1612176093 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f28d001a8f0 con 0x7f28e0107c10 2024-04-26T17:45:50.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:48.219+0000 7f28cf7fe700 1 -- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f28c8039600 msgr2=0x7f28c803bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T17:45:50.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:48.219+0000 7f28cf7fe700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f28c8039600 0x7f28c803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:50.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.221+0000 7f28cf7fe700 1 -- 172.21.15.31:0/1612176093 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f28d004fbd0 con 0x7f28e0107c10 2024-04-26T17:45:50.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.221+0000 7f28cf7fe700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f28c803cfb0 0x7f28c803f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:50.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.221+0000 7f28cf7fe700 1 -- 172.21.15.31:0/1612176093 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f28e006a530 con 0x7f28c803cfb0 2024-04-26T17:45:50.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.222+0000 7f28ddd9b700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f28c803cfb0 0x7f28c803f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:50.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.222+0000 7f28ddd9b700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f28c803cfb0 0x7f28c803f3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f28d4003a10 tx=0x7f28d40092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:50.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.223+0000 7f28cf7fe700 1 -- 172.21.15.31:0/1612176093 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f28e006a530 con 0x7f28c803cfb0 2024-04-26T17:45:50.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.231+0000 7f28e4ae3700 1 -- 172.21.15.31:0/1612176093 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f28e010c010 con 0x7f28c803cfb0 2024-04-26T17:45:50.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.232+0000 7f28cf7fe700 1 -- 172.21.15.31:0/1612176093 <== mgr.14150 v2:172.21.15.31:6800/2410008001 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f28e010c010 con 0x7f28c803cfb0 2024-04-26T17:45:50.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.232+0000 7f28e4ae3700 1 -- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f28c803cfb0 msgr2=0x7f28c803f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:50.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.232+0000 7f28e4ae3700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f28c803cfb0 0x7f28c803f3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f28d4003a10 tx=0x7f28d40092b0 comp rx=0 tx=0).stop 2024-04-26T17:45:50.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.232+0000 7f28e4ae3700 1 -- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e0107c10 msgr2=0x7f28e019cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:50.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.232+0000 7f28e4ae3700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e0107c10 0x7f28e019cc80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f28d0009520 tx=0x7f28d0011aa0 comp rx=0 tx=0).stop 2024-04-26T17:45:50.131 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.232+0000 7f28e4ae3700 1 -- 172.21.15.31:0/1612176093 shutdown_connections 2024-04-26T17:45:50.131 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.232+0000 7f28e4ae3700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f28c803cfb0 0x7f28c803f3a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:50.131 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.232+0000 7f28e4ae3700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e0107c10 0x7f28e019cc80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:50.131 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.232+0000 7f28e4ae3700 1 --2- 172.21.15.31:0/1612176093 >> [v2:172.21.15.31:6800/153341213,v1:172.21.15.31:6801/153341213] conn(0x7f28c8039600 0x7f28c803bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:50.131 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.232+0000 7f28e4ae3700 1 -- 172.21.15.31:0/1612176093 >> 172.21.15.31:0/1612176093 conn(0x7f28e0103210 msgr2=0x7f28e0104060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:50.131 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.232+0000 7f28e4ae3700 1 -- 172.21.15.31:0/1612176093 shutdown_connections 2024-04-26T17:45:50.131 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:49.232+0000 7f28e4ae3700 1 -- 172.21.15.31:0/1612176093 wait complete. 2024-04-26T17:45:50.131 INFO:teuthology.orchestra.run.smithi031.stdout:mgr epoch 9 is available 2024-04-26T17:45:50.131 INFO:teuthology.orchestra.run.smithi031.stdout:Generating a dashboard self-signed certificate... 2024-04-26T17:45:51.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:50 smithi031 ceph-mon[29164]: [26/Apr/2024:17:45:49] ENGINE Bus STARTING 2024-04-26T17:45:51.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:50 smithi031 ceph-mon[29164]: [26/Apr/2024:17:45:49] ENGINE Serving on http://172.21.15.31:8765 2024-04-26T17:45:51.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:50 smithi031 ceph-mon[29164]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T17:45:51.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:50 smithi031 ceph-mon[29164]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T17:45:51.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:50 smithi031 ceph-mon[29164]: [26/Apr/2024:17:45:49] ENGINE Serving on https://172.21.15.31:7150 2024-04-26T17:45:51.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:50 smithi031 ceph-mon[29164]: [26/Apr/2024:17:45:49] ENGINE Bus STARTED 2024-04-26T17:45:51.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:50 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:45:51.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-26T17:45:51.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.525+0000 7f7c2d9f1700 1 Processor -- start 2024-04-26T17:45:51.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.525+0000 7f7c2d9f1700 1 -- start start 2024-04-26T17:45:51.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.525+0000 7f7c2d9f1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c28107c70 0x7f7c28108090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:51.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.525+0000 7f7c2d9f1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7c28108660 con 0x7f7c28107c70 2024-04-26T17:45:51.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.525+0000 7f7c26ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c28107c70 0x7f7c28108090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:51.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.525+0000 7f7c26ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c28107c70 0x7f7c28108090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50772/0 (socket says 172.21.15.31:50772) 2024-04-26T17:45:51.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.525+0000 7f7c26ffd700 1 -- 172.21.15.31:0/309492464 learned_addr learned my addr 172.21.15.31:0/309492464 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:51.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.526+0000 7f7c26ffd700 1 -- 172.21.15.31:0/309492464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c28108e70 con 0x7f7c28107c70 2024-04-26T17:45:51.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.526+0000 7f7c26ffd700 1 --2- 172.21.15.31:0/309492464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c28107c70 0x7f7c28108090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7c180098a0 tx=0x7f7c18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b33b7156ba310d30 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:51.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.526+0000 7f7c25ffb700 1 -- 172.21.15.31:0/309492464 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c18004160 con 0x7f7c28107c70 2024-04-26T17:45:51.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.526+0000 7f7c25ffb700 1 -- 172.21.15.31:0/309492464 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c18004300 con 0x7f7c28107c70 2024-04-26T17:45:51.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.526+0000 7f7c25ffb700 1 -- 172.21.15.31:0/309492464 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c18011a20 con 0x7f7c28107c70 2024-04-26T17:45:51.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.527+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/309492464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c28107c70 msgr2=0x7f7c28108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:51.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.527+0000 7f7c2d9f1700 1 --2- 172.21.15.31:0/309492464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c28107c70 0x7f7c28108090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7c180098a0 tx=0x7f7c18009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:51.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.527+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/309492464 shutdown_connections 2024-04-26T17:45:51.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.527+0000 7f7c2d9f1700 1 --2- 172.21.15.31:0/309492464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c28107c70 0x7f7c28108090 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:51.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.527+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/309492464 >> 172.21.15.31:0/309492464 conn(0x7f7c28103210 msgr2=0x7f7c28105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:51.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.527+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/309492464 shutdown_connections 2024-04-26T17:45:51.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.527+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/309492464 wait complete. 2024-04-26T17:45:51.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.528+0000 7f7c2d9f1700 1 Processor -- start 2024-04-26T17:45:51.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.528+0000 7f7c2d9f1700 1 -- start start 2024-04-26T17:45:51.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.528+0000 7f7c2d9f1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c28107c70 0x7f7c28192de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:51.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.528+0000 7f7c2d9f1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2810c220 con 0x7f7c28107c70 2024-04-26T17:45:51.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.528+0000 7f7c26ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c28107c70 0x7f7c28192de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:51.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.528+0000 7f7c26ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c28107c70 0x7f7c28192de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50780/0 (socket says 172.21.15.31:50780) 2024-04-26T17:45:51.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.528+0000 7f7c26ffd700 1 -- 172.21.15.31:0/428698106 learned_addr learned my addr 172.21.15.31:0/428698106 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:51.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.528+0000 7f7c26ffd700 1 -- 172.21.15.31:0/428698106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c28193320 con 0x7f7c28107c70 2024-04-26T17:45:51.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.529+0000 7f7c26ffd700 1 --2- 172.21.15.31:0/428698106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c28107c70 0x7f7c28192de0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009520 tx=0x7f7c18011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:51.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.529+0000 7f7c2c9ef700 1 -- 172.21.15.31:0/428698106 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c180039e0 con 0x7f7c28107c70 2024-04-26T17:45:51.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.529+0000 7f7c2c9ef700 1 -- 172.21.15.31:0/428698106 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c1801b8f0 con 0x7f7c28107c70 2024-04-26T17:45:51.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.529+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/428698106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c28191510 con 0x7f7c28107c70 2024-04-26T17:45:51.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.529+0000 7f7c2c9ef700 1 -- 172.21.15.31:0/428698106 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c18019770 con 0x7f7c28107c70 2024-04-26T17:45:51.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.529+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/428698106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c28191a50 con 0x7f7c28107c70 2024-04-26T17:45:51.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.530+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/428698106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c28064a80 con 0x7f7c28107c70 2024-04-26T17:45:51.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.530+0000 7f7c2c9ef700 1 -- 172.21.15.31:0/428698106 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f7c18019a20 con 0x7f7c28107c70 2024-04-26T17:45:51.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.530+0000 7f7c2c9ef700 1 --2- 172.21.15.31:0/428698106 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f7c14039530 0x7f7c1403b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:51.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.530+0000 7f7c2c9ef700 1 -- 172.21.15.31:0/428698106 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7c1804de60 con 0x7f7c28107c70 2024-04-26T17:45:51.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.530+0000 7f7c267fc700 1 --2- 172.21.15.31:0/428698106 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f7c14039530 0x7f7c1403b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:51.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.531+0000 7f7c267fc700 1 --2- 172.21.15.31:0/428698106 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f7c14039530 0x7f7c1403b9f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7c10006fd0 tx=0x7f7c10006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:51.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.535+0000 7f7c2c9ef700 1 -- 172.21.15.31:0/428698106 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c1801f030 con 0x7f7c28107c70 2024-04-26T17:45:51.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.754+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/428698106 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f7c28196ba0 con 0x7f7c14039530 2024-04-26T17:45:51.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.781+0000 7f7c2c9ef700 1 -- 172.21.15.31:0/428698106 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f7c1801a8f0 con 0x7f7c28107c70 2024-04-26T17:45:51.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.832+0000 7f7c2c9ef700 1 -- 172.21.15.31:0/428698106 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f7c28196ba0 con 0x7f7c14039530 2024-04-26T17:45:51.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.834+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/428698106 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f7c14039530 msgr2=0x7f7c1403b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:51.370 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.834+0000 7f7c2d9f1700 1 --2- 172.21.15.31:0/428698106 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f7c14039530 0x7f7c1403b9f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7c10006fd0 tx=0x7f7c10006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:51.370 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.834+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/428698106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c28107c70 msgr2=0x7f7c28192de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:51.370 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.834+0000 7f7c2d9f1700 1 --2- 172.21.15.31:0/428698106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c28107c70 0x7f7c28192de0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009520 tx=0x7f7c18011aa0 comp rx=0 tx=0).stop 2024-04-26T17:45:51.370 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.834+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/428698106 shutdown_connections 2024-04-26T17:45:51.370 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.834+0000 7f7c2d9f1700 1 --2- 172.21.15.31:0/428698106 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f7c14039530 0x7f7c1403b9f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:51.370 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.834+0000 7f7c2d9f1700 1 --2- 172.21.15.31:0/428698106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c28107c70 0x7f7c28192de0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:51.370 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.834+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/428698106 >> 172.21.15.31:0/428698106 conn(0x7f7c28103210 msgr2=0x7f7c28104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:51.370 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.834+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/428698106 shutdown_connections 2024-04-26T17:45:51.370 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:50.834+0000 7f7c2d9f1700 1 -- 172.21.15.31:0/428698106 wait complete. 2024-04-26T17:45:51.370 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial admin user... 2024-04-26T17:45:51.911 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:51 smithi031 ceph-mon[29164]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:45:51.911 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:51 smithi031 ceph-mon[29164]: mgrmap e12: x(active, since 2s) 2024-04-26T17:45:51.911 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:51 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:45:51.911 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:51 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:45:51.911 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:51 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:45:52.901 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:52 smithi031 ceph-mon[29164]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:45:52.901 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:52 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:45:53.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$m.5e5A6G562U8bwNCvFvf.Gw5X1czdpCrWBwwwfJP60cdd4c/SYQa", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714153552, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-26T17:45:53.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.766+0000 7f2fc70ee700 1 Processor -- start 2024-04-26T17:45:53.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.766+0000 7f2fc70ee700 1 -- start start 2024-04-26T17:45:53.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.766+0000 7f2fc70ee700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fc0108e20 0x7f2fc0109240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:53.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.767+0000 7f2fc70ee700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc0109810 con 0x7f2fc0108e20 2024-04-26T17:45:53.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.767+0000 7f2fc4e8a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fc0108e20 0x7f2fc0109240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:53.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.767+0000 7f2fc4e8a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fc0108e20 0x7f2fc0109240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50796/0 (socket says 172.21.15.31:50796) 2024-04-26T17:45:53.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.767+0000 7f2fc4e8a700 1 -- 172.21.15.31:0/4140921891 learned_addr learned my addr 172.21.15.31:0/4140921891 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:53.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.767+0000 7f2fc4e8a700 1 -- 172.21.15.31:0/4140921891 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc0109f80 con 0x7f2fc0108e20 2024-04-26T17:45:53.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.767+0000 7f2fc4e8a700 1 --2- 172.21.15.31:0/4140921891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fc0108e20 0x7f2fc0109240 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40098a0 tx=0x7f2fb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89634f88886a2f6d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:53.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.767+0000 7f2fbf7fe700 1 -- 172.21.15.31:0/4140921891 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2fb4004160 con 0x7f2fc0108e20 2024-04-26T17:45:53.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.767+0000 7f2fbf7fe700 1 -- 172.21.15.31:0/4140921891 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fb4004300 con 0x7f2fc0108e20 2024-04-26T17:45:53.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.767+0000 7f2fbf7fe700 1 -- 172.21.15.31:0/4140921891 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2fb4011a20 con 0x7f2fc0108e20 2024-04-26T17:45:53.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.768+0000 7f2fc70ee700 1 -- 172.21.15.31:0/4140921891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fc0108e20 msgr2=0x7f2fc0109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:53.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.768+0000 7f2fc70ee700 1 --2- 172.21.15.31:0/4140921891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fc0108e20 0x7f2fc0109240 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40098a0 tx=0x7f2fb4009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:53.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.768+0000 7f2fc70ee700 1 -- 172.21.15.31:0/4140921891 shutdown_connections 2024-04-26T17:45:53.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.768+0000 7f2fc70ee700 1 --2- 172.21.15.31:0/4140921891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fc0108e20 0x7f2fc0109240 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:53.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.768+0000 7f2fc70ee700 1 -- 172.21.15.31:0/4140921891 >> 172.21.15.31:0/4140921891 conn(0x7f2fc0104400 msgr2=0x7f2fc0106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:53.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.768+0000 7f2fc70ee700 1 -- 172.21.15.31:0/4140921891 shutdown_connections 2024-04-26T17:45:53.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.768+0000 7f2fc70ee700 1 -- 172.21.15.31:0/4140921891 wait complete. 2024-04-26T17:45:53.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.769+0000 7f2fc70ee700 1 Processor -- start 2024-04-26T17:45:53.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.769+0000 7f2fc70ee700 1 -- start start 2024-04-26T17:45:53.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.769+0000 7f2fc70ee700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fc0108e20 0x7f2fc019ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:53.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.769+0000 7f2fc70ee700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc010d330 con 0x7f2fc0108e20 2024-04-26T17:45:53.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.769+0000 7f2fc4e8a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fc0108e20 0x7f2fc019ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:53.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.769+0000 7f2fc4e8a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fc0108e20 0x7f2fc019ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50812/0 (socket says 172.21.15.31:50812) 2024-04-26T17:45:53.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.769+0000 7f2fc4e8a700 1 -- 172.21.15.31:0/2934977284 learned_addr learned my addr 172.21.15.31:0/2934977284 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:53.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.769+0000 7f2fc4e8a700 1 -- 172.21.15.31:0/2934977284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc019d1f0 con 0x7f2fc0108e20 2024-04-26T17:45:53.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.769+0000 7f2fc4e8a700 1 --2- 172.21.15.31:0/2934977284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fc0108e20 0x7f2fc019ccb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40119f0 tx=0x7f2fb4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:53.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.770+0000 7f2fbdffb700 1 -- 172.21.15.31:0/2934977284 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2fb40039e0 con 0x7f2fc0108e20 2024-04-26T17:45:53.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.770+0000 7f2fbdffb700 1 -- 172.21.15.31:0/2934977284 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fb401b8f0 con 0x7f2fc0108e20 2024-04-26T17:45:53.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.770+0000 7f2fc70ee700 1 -- 172.21.15.31:0/2934977284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fc019d480 con 0x7f2fc0108e20 2024-04-26T17:45:53.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.770+0000 7f2fbdffb700 1 -- 172.21.15.31:0/2934977284 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2fb40237a0 con 0x7f2fc0108e20 2024-04-26T17:45:53.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.770+0000 7f2fc70ee700 1 -- 172.21.15.31:0/2934977284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fc01a0170 con 0x7f2fc0108e20 2024-04-26T17:45:53.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.771+0000 7f2fbdffb700 1 -- 172.21.15.31:0/2934977284 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f2fb4023a50 con 0x7f2fc0108e20 2024-04-26T17:45:53.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.771+0000 7f2fbdffb700 1 --2- 172.21.15.31:0/2934977284 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2fb0039610 0x7f2fb003bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:53.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.771+0000 7f2fbdffb700 1 -- 172.21.15.31:0/2934977284 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2fb404df60 con 0x7f2fc0108e20 2024-04-26T17:45:53.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.771+0000 7f2fc70ee700 1 -- 172.21.15.31:0/2934977284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fc0064a80 con 0x7f2fc0108e20 2024-04-26T17:45:53.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.771+0000 7f2fbffff700 1 --2- 172.21.15.31:0/2934977284 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2fb0039610 0x7f2fb003bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:53.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.771+0000 7f2fbffff700 1 --2- 172.21.15.31:0/2934977284 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2fb0039610 0x7f2fb003bad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2fac006fd0 tx=0x7f2fac006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:53.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:51.777+0000 7f2fbdffb700 1 -- 172.21.15.31:0/2934977284 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2fb401f030 con 0x7f2fc0108e20 2024-04-26T17:45:53.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:52.010+0000 7f2fc70ee700 1 -- 172.21.15.31:0/2934977284 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f2fc01a0340 con 0x7f2fb0039610 2024-04-26T17:45:53.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:52.257+0000 7f2fbdffb700 1 -- 172.21.15.31:0/2934977284 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f2fc01a0340 con 0x7f2fb0039610 2024-04-26T17:45:53.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:52.259+0000 7f2fc70ee700 1 -- 172.21.15.31:0/2934977284 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2fb0039610 msgr2=0x7f2fb003bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:53.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:52.259+0000 7f2fc70ee700 1 --2- 172.21.15.31:0/2934977284 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2fb0039610 0x7f2fb003bad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2fac006fd0 tx=0x7f2fac006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:53.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:52.259+0000 7f2fc70ee700 1 -- 172.21.15.31:0/2934977284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fc0108e20 msgr2=0x7f2fc019ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:53.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:52.259+0000 7f2fc70ee700 1 --2- 172.21.15.31:0/2934977284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fc0108e20 0x7f2fc019ccb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40119f0 tx=0x7f2fb4011ad0 comp rx=0 tx=0).stop 2024-04-26T17:45:53.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:52.259+0000 7f2fc70ee700 1 -- 172.21.15.31:0/2934977284 shutdown_connections 2024-04-26T17:45:53.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:52.259+0000 7f2fc70ee700 1 --2- 172.21.15.31:0/2934977284 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2fb0039610 0x7f2fb003bad0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:53.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:52.259+0000 7f2fc70ee700 1 --2- 172.21.15.31:0/2934977284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fc0108e20 0x7f2fc019ccb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:53.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:52.259+0000 7f2fc70ee700 1 -- 172.21.15.31:0/2934977284 >> 172.21.15.31:0/2934977284 conn(0x7f2fc0104400 msgr2=0x7f2fc0104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:53.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:52.259+0000 7f2fc70ee700 1 -- 172.21.15.31:0/2934977284 shutdown_connections 2024-04-26T17:45:53.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:52.259+0000 7f2fc70ee700 1 -- 172.21.15.31:0/2934977284 wait complete. 2024-04-26T17:45:53.274 INFO:teuthology.orchestra.run.smithi031.stdout:Fetching dashboard port number... 2024-04-26T17:45:54.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:53 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2773219250' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T17:45:54.452 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 8443 2024-04-26T17:45:54.452 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.726+0000 7ff620fd5700 1 Processor -- start 2024-04-26T17:45:54.452 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.726+0000 7ff620fd5700 1 -- start start 2024-04-26T17:45:54.452 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.727+0000 7ff620fd5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c1092a0 0x7ff61c1096c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:54.452 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.727+0000 7ff620fd5700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff61c109c90 con 0x7ff61c1092a0 2024-04-26T17:45:54.452 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.727+0000 7ff61a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c1092a0 0x7ff61c1096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:54.452 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.727+0000 7ff61a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c1092a0 0x7ff61c1096c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50818/0 (socket says 172.21.15.31:50818) 2024-04-26T17:45:54.452 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.727+0000 7ff61a59c700 1 -- 172.21.15.31:0/2585149048 learned_addr learned my addr 172.21.15.31:0/2585149048 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:54.452 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.727+0000 7ff61a59c700 1 -- 172.21.15.31:0/2585149048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff61c106da0 con 0x7ff61c1092a0 2024-04-26T17:45:54.452 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.728+0000 7ff61a59c700 1 --2- 172.21.15.31:0/2585149048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c1092a0 0x7ff61c1096c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0098a0 tx=0x7ff60c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f35fd5186409b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:54.452 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.728+0000 7ff61959a700 1 -- 172.21.15.31:0/2585149048 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff60c004160 con 0x7ff61c1092a0 2024-04-26T17:45:54.453 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.728+0000 7ff61959a700 1 -- 172.21.15.31:0/2585149048 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff60c004300 con 0x7ff61c1092a0 2024-04-26T17:45:54.453 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.728+0000 7ff61959a700 1 -- 172.21.15.31:0/2585149048 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff60c011a20 con 0x7ff61c1092a0 2024-04-26T17:45:54.453 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.728+0000 7ff620fd5700 1 -- 172.21.15.31:0/2585149048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c1092a0 msgr2=0x7ff61c1096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:54.453 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.728+0000 7ff620fd5700 1 --2- 172.21.15.31:0/2585149048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c1092a0 0x7ff61c1096c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0098a0 tx=0x7ff60c009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:54.453 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.728+0000 7ff620fd5700 1 -- 172.21.15.31:0/2585149048 shutdown_connections 2024-04-26T17:45:54.453 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.728+0000 7ff620fd5700 1 --2- 172.21.15.31:0/2585149048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c1092a0 0x7ff61c1096c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:54.453 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.728+0000 7ff620fd5700 1 -- 172.21.15.31:0/2585149048 >> 172.21.15.31:0/2585149048 conn(0x7ff61c102960 msgr2=0x7ff61c104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:54.453 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.728+0000 7ff620fd5700 1 -- 172.21.15.31:0/2585149048 shutdown_connections 2024-04-26T17:45:54.453 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.728+0000 7ff620fd5700 1 -- 172.21.15.31:0/2585149048 wait complete. 2024-04-26T17:45:54.453 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.729+0000 7ff620fd5700 1 Processor -- start 2024-04-26T17:45:54.453 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.729+0000 7ff620fd5700 1 -- start start 2024-04-26T17:45:54.453 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.729+0000 7ff620fd5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c1092a0 0x7ff61c19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:54.453 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.729+0000 7ff620fd5700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff61c107e00 con 0x7ff61c1092a0 2024-04-26T17:45:54.454 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.730+0000 7ff61a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c1092a0 0x7ff61c19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:54.454 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.730+0000 7ff61a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c1092a0 0x7ff61c19ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50826/0 (socket says 172.21.15.31:50826) 2024-04-26T17:45:54.454 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.730+0000 7ff61a59c700 1 -- 172.21.15.31:0/2773219250 learned_addr learned my addr 172.21.15.31:0/2773219250 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:54.454 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.730+0000 7ff61a59c700 1 -- 172.21.15.31:0/2773219250 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff61c19f350 con 0x7ff61c1092a0 2024-04-26T17:45:54.454 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.730+0000 7ff61a59c700 1 --2- 172.21.15.31:0/2773219250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c1092a0 0x7ff61c19ee10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff60c000c00 tx=0x7ff60c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:54.454 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.730+0000 7ff60b7fe700 1 -- 172.21.15.31:0/2773219250 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff60c003aa0 con 0x7ff61c1092a0 2024-04-26T17:45:54.454 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.730+0000 7ff60b7fe700 1 -- 172.21.15.31:0/2773219250 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff60c01b8f0 con 0x7ff61c1092a0 2024-04-26T17:45:54.454 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.730+0000 7ff620fd5700 1 -- 172.21.15.31:0/2773219250 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff61c19f5e0 con 0x7ff61c1092a0 2024-04-26T17:45:54.454 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.730+0000 7ff60b7fe700 1 -- 172.21.15.31:0/2773219250 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff60c023870 con 0x7ff61c1092a0 2024-04-26T17:45:54.454 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.730+0000 7ff620fd5700 1 -- 172.21.15.31:0/2773219250 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff61c1a22d0 con 0x7ff61c1092a0 2024-04-26T17:45:54.454 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.731+0000 7ff60b7fe700 1 -- 172.21.15.31:0/2773219250 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7ff60c023b20 con 0x7ff61c1092a0 2024-04-26T17:45:54.454 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.731+0000 7ff620fd5700 1 -- 172.21.15.31:0/2773219250 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff61c064a80 con 0x7ff61c1092a0 2024-04-26T17:45:54.454 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.731+0000 7ff60b7fe700 1 --2- 172.21.15.31:0/2773219250 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff604039660 0x7ff60403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:54.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.731+0000 7ff60b7fe700 1 -- 172.21.15.31:0/2773219250 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff60c04e0f0 con 0x7ff61c1092a0 2024-04-26T17:45:54.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.732+0000 7ff619d9b700 1 --2- 172.21.15.31:0/2773219250 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff604039660 0x7ff60403bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:54.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.732+0000 7ff619d9b700 1 --2- 172.21.15.31:0/2773219250 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff604039660 0x7ff60403bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff610006fd0 tx=0x7ff610006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:54.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.737+0000 7ff60b7fe700 1 -- 172.21.15.31:0/2773219250 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff60c01f030 con 0x7ff61c1092a0 2024-04-26T17:45:54.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.945+0000 7ff620fd5700 1 -- 172.21.15.31:0/2773219250 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7ff61c198410 con 0x7ff61c1092a0 2024-04-26T17:45:54.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.945+0000 7ff60b7fe700 1 -- 172.21.15.31:0/2773219250 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7ff61c198410 con 0x7ff61c1092a0 2024-04-26T17:45:54.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.947+0000 7ff620fd5700 1 -- 172.21.15.31:0/2773219250 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff604039660 msgr2=0x7ff60403bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:54.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.947+0000 7ff620fd5700 1 --2- 172.21.15.31:0/2773219250 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff604039660 0x7ff60403bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff610006fd0 tx=0x7ff610006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:54.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.947+0000 7ff620fd5700 1 -- 172.21.15.31:0/2773219250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c1092a0 msgr2=0x7ff61c19ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:54.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.947+0000 7ff620fd5700 1 --2- 172.21.15.31:0/2773219250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c1092a0 0x7ff61c19ee10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff60c000c00 tx=0x7ff60c011aa0 comp rx=0 tx=0).stop 2024-04-26T17:45:54.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.947+0000 7ff620fd5700 1 -- 172.21.15.31:0/2773219250 shutdown_connections 2024-04-26T17:45:54.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.947+0000 7ff620fd5700 1 --2- 172.21.15.31:0/2773219250 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff604039660 0x7ff60403bb20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:54.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.947+0000 7ff620fd5700 1 --2- 172.21.15.31:0/2773219250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff61c1092a0 0x7ff61c19ee10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:54.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.947+0000 7ff620fd5700 1 -- 172.21.15.31:0/2773219250 >> 172.21.15.31:0/2773219250 conn(0x7ff61c102960 msgr2=0x7ff61c1954f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:54.456 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.948+0000 7ff620fd5700 1 -- 172.21.15.31:0/2773219250 shutdown_connections 2024-04-26T17:45:54.456 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:53.948+0000 7ff620fd5700 1 -- 172.21.15.31:0/2773219250 wait complete. 2024-04-26T17:45:54.459 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T17:45:54.459 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-04-26T17:45:54.466 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T17:45:54.466 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-26T17:45:54.467 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-04-26T17:45:54.467 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-26T17:45:54.468 INFO:teuthology.orchestra.run.smithi031.stdout:Ceph Dashboard is now available at: 2024-04-26T17:45:54.468 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:45:54.468 INFO:teuthology.orchestra.run.smithi031.stdout: URL: https://smithi031:8443/ 2024-04-26T17:45:54.468 INFO:teuthology.orchestra.run.smithi031.stdout: User: admin 2024-04-26T17:45:54.468 INFO:teuthology.orchestra.run.smithi031.stdout: Password: 5218ns8jry 2024-04-26T17:45:54.469 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:45:54.469 INFO:teuthology.orchestra.run.smithi031.stdout:Saving cluster configuration to /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config directory 2024-04-26T17:45:54.469 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling autotune for osd_memory_target 2024-04-26T17:45:55.675 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.877+0000 7f2b13a0c700 1 Processor -- start 2024-04-26T17:45:55.675 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.877+0000 7f2b13a0c700 1 -- start start 2024-04-26T17:45:55.675 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.877+0000 7f2b13a0c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c108e00 0x7f2b0c109220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:55.675 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.877+0000 7f2b13a0c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c1097f0 con 0x7f2b0c108e00 2024-04-26T17:45:55.675 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.877+0000 7f2b117a8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c108e00 0x7f2b0c109220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:55.675 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.877+0000 7f2b117a8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c108e00 0x7f2b0c109220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60152/0 (socket says 172.21.15.31:60152) 2024-04-26T17:45:55.675 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.877+0000 7f2b117a8700 1 -- 172.21.15.31:0/2416442094 learned_addr learned my addr 172.21.15.31:0/2416442094 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:55.675 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.878+0000 7f2b117a8700 1 -- 172.21.15.31:0/2416442094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0c109f60 con 0x7f2b0c108e00 2024-04-26T17:45:55.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.878+0000 7f2b117a8700 1 --2- 172.21.15.31:0/2416442094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c108e00 0x7f2b0c109220 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2b080098a0 tx=0x7f2b08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205d8b2c39426337 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:55.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.878+0000 7f2b03fff700 1 -- 172.21.15.31:0/2416442094 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b08004160 con 0x7f2b0c108e00 2024-04-26T17:45:55.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.878+0000 7f2b03fff700 1 -- 172.21.15.31:0/2416442094 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b08004300 con 0x7f2b0c108e00 2024-04-26T17:45:55.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.878+0000 7f2b03fff700 1 -- 172.21.15.31:0/2416442094 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b08011a20 con 0x7f2b0c108e00 2024-04-26T17:45:55.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.878+0000 7f2b13a0c700 1 -- 172.21.15.31:0/2416442094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c108e00 msgr2=0x7f2b0c109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:55.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.878+0000 7f2b13a0c700 1 --2- 172.21.15.31:0/2416442094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c108e00 0x7f2b0c109220 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2b080098a0 tx=0x7f2b08009bb0 comp rx=0 tx=0).stop 2024-04-26T17:45:55.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.878+0000 7f2b13a0c700 1 -- 172.21.15.31:0/2416442094 shutdown_connections 2024-04-26T17:45:55.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.878+0000 7f2b13a0c700 1 --2- 172.21.15.31:0/2416442094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c108e00 0x7f2b0c109220 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:55.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.878+0000 7f2b13a0c700 1 -- 172.21.15.31:0/2416442094 >> 172.21.15.31:0/2416442094 conn(0x7f2b0c1043c0 msgr2=0x7f2b0c1067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:55.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.879+0000 7f2b13a0c700 1 -- 172.21.15.31:0/2416442094 shutdown_connections 2024-04-26T17:45:55.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.879+0000 7f2b13a0c700 1 -- 172.21.15.31:0/2416442094 wait complete. 2024-04-26T17:45:55.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.879+0000 7f2b13a0c700 1 Processor -- start 2024-04-26T17:45:55.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.879+0000 7f2b13a0c700 1 -- start start 2024-04-26T17:45:55.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.879+0000 7f2b13a0c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c108e00 0x7f2b0c19cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:55.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.879+0000 7f2b13a0c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c10d310 con 0x7f2b0c108e00 2024-04-26T17:45:55.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.879+0000 7f2b117a8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c108e00 0x7f2b0c19cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:55.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.880+0000 7f2b117a8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c108e00 0x7f2b0c19cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60156/0 (socket says 172.21.15.31:60156) 2024-04-26T17:45:55.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.880+0000 7f2b117a8700 1 -- 172.21.15.31:0/1483593859 learned_addr learned my addr 172.21.15.31:0/1483593859 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:55.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.880+0000 7f2b117a8700 1 -- 172.21.15.31:0/1483593859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0c19d1d0 con 0x7f2b0c108e00 2024-04-26T17:45:55.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.880+0000 7f2b117a8700 1 --2- 172.21.15.31:0/1483593859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c108e00 0x7f2b0c19cc90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b080119f0 tx=0x7f2b08011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:55.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.880+0000 7f2b027fc700 1 -- 172.21.15.31:0/1483593859 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b080039e0 con 0x7f2b0c108e00 2024-04-26T17:45:55.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.880+0000 7f2b027fc700 1 -- 172.21.15.31:0/1483593859 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b0801b8f0 con 0x7f2b0c108e00 2024-04-26T17:45:55.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.880+0000 7f2b13a0c700 1 -- 172.21.15.31:0/1483593859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b0c19d460 con 0x7f2b0c108e00 2024-04-26T17:45:55.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.880+0000 7f2b027fc700 1 -- 172.21.15.31:0/1483593859 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b080237a0 con 0x7f2b0c108e00 2024-04-26T17:45:55.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.880+0000 7f2b13a0c700 1 -- 172.21.15.31:0/1483593859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b0c1a0150 con 0x7f2b0c108e00 2024-04-26T17:45:55.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.881+0000 7f2b027fc700 1 -- 172.21.15.31:0/1483593859 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f2b0801b440 con 0x7f2b0c108e00 2024-04-26T17:45:55.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.881+0000 7f2b13a0c700 1 -- 172.21.15.31:0/1483593859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b0c064a80 con 0x7f2b0c108e00 2024-04-26T17:45:55.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.881+0000 7f2b027fc700 1 --2- 172.21.15.31:0/1483593859 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2af80395b0 0x7f2af803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:55.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.881+0000 7f2b027fc700 1 -- 172.21.15.31:0/1483593859 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2b0804df70 con 0x7f2b0c108e00 2024-04-26T17:45:55.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.881+0000 7f2b10fa7700 1 --2- 172.21.15.31:0/1483593859 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2af80395b0 0x7f2af803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:55.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.882+0000 7f2b10fa7700 1 --2- 172.21.15.31:0/1483593859 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2af80395b0 0x7f2af803ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2afc006fd0 tx=0x7f2afc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:55.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.886+0000 7f2b027fc700 1 -- 172.21.15.31:0/1483593859 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b0801f030 con 0x7f2b0c108e00 2024-04-26T17:45:55.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:54.998+0000 7f2b027fc700 1 -- 172.21.15.31:0/1483593859 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2b0801a3d0 con 0x7f2b0c108e00 2024-04-26T17:45:55.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:55.095+0000 7f2b13a0c700 1 -- 172.21.15.31:0/1483593859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f2b0c1a03c0 con 0x7f2b0c108e00 2024-04-26T17:45:55.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:55.096+0000 7f2b027fc700 1 -- 172.21.15.31:0/1483593859 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f2b08023e20 con 0x7f2b0c108e00 2024-04-26T17:45:55.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:55.098+0000 7f2b13a0c700 1 -- 172.21.15.31:0/1483593859 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2af80395b0 msgr2=0x7f2af803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:55.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:55.098+0000 7f2b13a0c700 1 --2- 172.21.15.31:0/1483593859 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2af80395b0 0x7f2af803ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2afc006fd0 tx=0x7f2afc006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:55.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:55.098+0000 7f2b13a0c700 1 -- 172.21.15.31:0/1483593859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c108e00 msgr2=0x7f2b0c19cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:55.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:55.098+0000 7f2b13a0c700 1 --2- 172.21.15.31:0/1483593859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c108e00 0x7f2b0c19cc90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b080119f0 tx=0x7f2b08011ad0 comp rx=0 tx=0).stop 2024-04-26T17:45:55.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:55.098+0000 7f2b13a0c700 1 -- 172.21.15.31:0/1483593859 shutdown_connections 2024-04-26T17:45:55.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:55.098+0000 7f2b13a0c700 1 --2- 172.21.15.31:0/1483593859 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2af80395b0 0x7f2af803ba70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:55.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:55.098+0000 7f2b13a0c700 1 --2- 172.21.15.31:0/1483593859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c108e00 0x7f2b0c19cc90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:55.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:55.098+0000 7f2b13a0c700 1 -- 172.21.15.31:0/1483593859 >> 172.21.15.31:0/1483593859 conn(0x7f2b0c1043c0 msgr2=0x7f2b0c104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:55.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:55.098+0000 7f2b13a0c700 1 -- 172.21.15.31:0/1483593859 shutdown_connections 2024-04-26T17:45:55.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:55.098+0000 7f2b13a0c700 1 -- 172.21.15.31:0/1483593859 wait complete. 2024-04-26T17:45:56.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:55 smithi031 ceph-mon[29164]: mgrmap e13: x(active, since 6s) 2024-04-26T17:45:56.973 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.055+0000 7fd705ba3700 1 Processor -- start 2024-04-26T17:45:56.973 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.055+0000 7fd705ba3700 1 -- start start 2024-04-26T17:45:56.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.056+0000 7fd705ba3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7001075d0 0x7fd7001079f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:56.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.056+0000 7fd705ba3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd70007ce70 con 0x7fd7001075d0 2024-04-26T17:45:56.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.056+0000 7fd6ff7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7001075d0 0x7fd7001079f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:56.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.056+0000 7fd6ff7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7001075d0 0x7fd7001079f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60160/0 (socket says 172.21.15.31:60160) 2024-04-26T17:45:56.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.056+0000 7fd6ff7fe700 1 -- 172.21.15.31:0/751585983 learned_addr learned my addr 172.21.15.31:0/751585983 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:56.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.056+0000 7fd6ff7fe700 1 -- 172.21.15.31:0/751585983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd700107f30 con 0x7fd7001075d0 2024-04-26T17:45:56.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.056+0000 7fd6ff7fe700 1 --2- 172.21.15.31:0/751585983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7001075d0 0x7fd7001079f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009e80 tx=0x7fd6f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b0fc0fae28328f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:56.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.056+0000 7fd6fe7fc700 1 -- 172.21.15.31:0/751585983 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6f0017070 con 0x7fd7001075d0 2024-04-26T17:45:56.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.057+0000 7fd6fe7fc700 1 -- 172.21.15.31:0/751585983 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6f000b8c0 con 0x7fd7001075d0 2024-04-26T17:45:56.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.057+0000 7fd6fe7fc700 1 -- 172.21.15.31:0/751585983 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6f0004490 con 0x7fd7001075d0 2024-04-26T17:45:56.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.057+0000 7fd705ba3700 1 -- 172.21.15.31:0/751585983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7001075d0 msgr2=0x7fd7001079f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:56.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.057+0000 7fd705ba3700 1 --2- 172.21.15.31:0/751585983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7001075d0 0x7fd7001079f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009e80 tx=0x7fd6f000b330 comp rx=0 tx=0).stop 2024-04-26T17:45:56.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.057+0000 7fd705ba3700 1 -- 172.21.15.31:0/751585983 shutdown_connections 2024-04-26T17:45:56.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.057+0000 7fd705ba3700 1 --2- 172.21.15.31:0/751585983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7001075d0 0x7fd7001079f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:56.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.057+0000 7fd705ba3700 1 -- 172.21.15.31:0/751585983 >> 172.21.15.31:0/751585983 conn(0x7fd700103210 msgr2=0x7fd700105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:56.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.057+0000 7fd705ba3700 1 -- 172.21.15.31:0/751585983 shutdown_connections 2024-04-26T17:45:56.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.057+0000 7fd705ba3700 1 -- 172.21.15.31:0/751585983 wait complete. 2024-04-26T17:45:56.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.058+0000 7fd705ba3700 1 Processor -- start 2024-04-26T17:45:56.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.058+0000 7fd705ba3700 1 -- start start 2024-04-26T17:45:56.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.058+0000 7fd705ba3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7001075d0 0x7fd70019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:56.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.058+0000 7fd705ba3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd700108fa0 con 0x7fd7001075d0 2024-04-26T17:45:56.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.058+0000 7fd6ff7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7001075d0 0x7fd70019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:56.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.058+0000 7fd6ff7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7001075d0 0x7fd70019eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60168/0 (socket says 172.21.15.31:60168) 2024-04-26T17:45:56.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.058+0000 7fd6ff7fe700 1 -- 172.21.15.31:0/41052550 learned_addr learned my addr 172.21.15.31:0/41052550 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:56.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.059+0000 7fd6ff7fe700 1 -- 172.21.15.31:0/41052550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70019f3e0 con 0x7fd7001075d0 2024-04-26T17:45:56.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.059+0000 7fd6ff7fe700 1 --2- 172.21.15.31:0/41052550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7001075d0 0x7fd70019eea0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0004220 tx=0x7fd6f000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:56.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.059+0000 7fd6fcff9700 1 -- 172.21.15.31:0/41052550 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6f0020020 con 0x7fd7001075d0 2024-04-26T17:45:56.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.059+0000 7fd6fcff9700 1 -- 172.21.15.31:0/41052550 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6f0003b50 con 0x7fd7001075d0 2024-04-26T17:45:56.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.059+0000 7fd705ba3700 1 -- 172.21.15.31:0/41052550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd70019f670 con 0x7fd7001075d0 2024-04-26T17:45:56.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.059+0000 7fd6fcff9700 1 -- 172.21.15.31:0/41052550 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6f0017040 con 0x7fd7001075d0 2024-04-26T17:45:56.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.059+0000 7fd705ba3700 1 -- 172.21.15.31:0/41052550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7001a2360 con 0x7fd7001075d0 2024-04-26T17:45:56.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.060+0000 7fd6fcff9700 1 -- 172.21.15.31:0/41052550 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fd6f0023020 con 0x7fd7001075d0 2024-04-26T17:45:56.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.060+0000 7fd705ba3700 1 -- 172.21.15.31:0/41052550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd700064a80 con 0x7fd7001075d0 2024-04-26T17:45:56.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.060+0000 7fd6fcff9700 1 --2- 172.21.15.31:0/41052550 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd6ec039650 0x7fd6ec03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:56.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.060+0000 7fd6fcff9700 1 -- 172.21.15.31:0/41052550 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd6f000f050 con 0x7fd7001075d0 2024-04-26T17:45:56.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.060+0000 7fd6feffd700 1 --2- 172.21.15.31:0/41052550 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd6ec039650 0x7fd6ec03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:56.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.061+0000 7fd6feffd700 1 --2- 172.21.15.31:0/41052550 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd6ec039650 0x7fd6ec03bb10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8006fd0 tx=0x7fd6e8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:56.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.065+0000 7fd6fcff9700 1 -- 172.21.15.31:0/41052550 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd6f002a020 con 0x7fd7001075d0 2024-04-26T17:45:56.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.387+0000 7fd705ba3700 1 -- 172.21.15.31:0/41052550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fd70002e6e0 con 0x7fd7001075d0 2024-04-26T17:45:56.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.389+0000 7fd6fcff9700 1 -- 172.21.15.31:0/41052550 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7fd6f0012430 con 0x7fd7001075d0 2024-04-26T17:45:56.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.391+0000 7fd705ba3700 1 -- 172.21.15.31:0/41052550 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd6ec039650 msgr2=0x7fd6ec03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:56.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.391+0000 7fd705ba3700 1 --2- 172.21.15.31:0/41052550 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd6ec039650 0x7fd6ec03bb10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8006fd0 tx=0x7fd6e8006e40 comp rx=0 tx=0).stop 2024-04-26T17:45:56.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.391+0000 7fd705ba3700 1 -- 172.21.15.31:0/41052550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7001075d0 msgr2=0x7fd70019eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:56.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.391+0000 7fd705ba3700 1 --2- 172.21.15.31:0/41052550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7001075d0 0x7fd70019eea0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0004220 tx=0x7fd6f000bd30 comp rx=0 tx=0).stop 2024-04-26T17:45:56.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.391+0000 7fd705ba3700 1 -- 172.21.15.31:0/41052550 shutdown_connections 2024-04-26T17:45:56.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.391+0000 7fd705ba3700 1 --2- 172.21.15.31:0/41052550 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd6ec039650 0x7fd6ec03bb10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:56.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.391+0000 7fd705ba3700 1 --2- 172.21.15.31:0/41052550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7001075d0 0x7fd70019eea0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:56.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.391+0000 7fd705ba3700 1 -- 172.21.15.31:0/41052550 >> 172.21.15.31:0/41052550 conn(0x7fd700103210 msgr2=0x7fd700195ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:56.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.392+0000 7fd705ba3700 1 -- 172.21.15.31:0/41052550 shutdown_connections 2024-04-26T17:45:56.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-26T17:45:56.392+0000 7fd705ba3700 1 -- 172.21.15.31:0/41052550 wait complete. 2024-04-26T17:45:56.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-26T17:45:56.978 INFO:teuthology.orchestra.run.smithi031.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-26T17:45:56.978 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:45:56.978 INFO:teuthology.orchestra.run.smithi031.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-26T17:45:56.978 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:45:56.978 INFO:teuthology.orchestra.run.smithi031.stdout:Or, if you are only running a single cluster on this host: 2024-04-26T17:45:56.978 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:45:56.978 INFO:teuthology.orchestra.run.smithi031.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-26T17:45:56.978 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:45:56.979 INFO:teuthology.orchestra.run.smithi031.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-26T17:45:56.979 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:45:56.979 INFO:teuthology.orchestra.run.smithi031.stdout: ceph telemetry on 2024-04-26T17:45:56.979 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:45:56.979 INFO:teuthology.orchestra.run.smithi031.stdout:For more information see: 2024-04-26T17:45:56.979 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:45:56.979 INFO:teuthology.orchestra.run.smithi031.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-26T17:45:56.979 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:45:56.979 INFO:teuthology.orchestra.run.smithi031.stdout:Bootstrap complete. 2024-04-26T17:45:57.013 INFO:tasks.cephadm:Fetching config... 2024-04-26T17:45:57.014 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:45:57.014 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-26T17:45:57.030 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-26T17:45:57.030 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:45:57.030 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-26T17:45:57.086 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-26T17:45:57.086 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:45:57.086 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-26T17:45:57.150 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-26T17:45:57.150 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:45:57.151 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-26T17:45:57.206 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-26T17:45:57.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu+NEeVJmhwe7hcSYRp4pMsWBs7o6C9+QTkNP4fbnQExDeY/OTVmIni1bcxbzMUptWE3ZBNOgC93mqeAfSms6AaqGxFhdxQei7I6+IPNoKU2O9ry2g8Db4Qwz0OIUJUDLuISMJj8El8+EfPfAvhSdLIdJ7sj3eQwVpZDxp7X7t0P4e5OaxOON21wh2/RPltchW9JsjG0HZrcH0Yv7q/mkJz1jgzcp7imaBurMZ+bgrkEg/n1k+J/dzEHXJZRlBM/nyd9vbSBCTrmqhf6xtuRicjjXH+n2f6NfbFtfcM4bUMLz7D5qLzC2kVKIu1mIUtbiyXiHipgZ3AJwqL86b+k8mxrw30uLExiyWeEXQKnAmOAmwq7umn9lxkZk0oQ4ZEK74c+9mxWLke1c/MitDCGMGopF3/RL0AbwpFLcpV/ga1My7Dvs5eW+EHV9Ks+i4v82V+R3KCGquk1vxEFJ2fxlCAVI61gvw5JC2L3Cis1Rg6Cc/BodMs46Zrl4xnytlHds= ceph-7e53e050-03f4-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T17:45:57.281 INFO:teuthology.orchestra.run.smithi031.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu+NEeVJmhwe7hcSYRp4pMsWBs7o6C9+QTkNP4fbnQExDeY/OTVmIni1bcxbzMUptWE3ZBNOgC93mqeAfSms6AaqGxFhdxQei7I6+IPNoKU2O9ry2g8Db4Qwz0OIUJUDLuISMJj8El8+EfPfAvhSdLIdJ7sj3eQwVpZDxp7X7t0P4e5OaxOON21wh2/RPltchW9JsjG0HZrcH0Yv7q/mkJz1jgzcp7imaBurMZ+bgrkEg/n1k+J/dzEHXJZRlBM/nyd9vbSBCTrmqhf6xtuRicjjXH+n2f6NfbFtfcM4bUMLz7D5qLzC2kVKIu1mIUtbiyXiHipgZ3AJwqL86b+k8mxrw30uLExiyWeEXQKnAmOAmwq7umn9lxkZk0oQ4ZEK74c+9mxWLke1c/MitDCGMGopF3/RL0AbwpFLcpV/ga1My7Dvs5eW+EHV9Ks+i4v82V+R3KCGquk1vxEFJ2fxlCAVI61gvw5JC2L3Cis1Rg6Cc/BodMs46Zrl4xnytlHds= ceph-7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:45:57.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu+NEeVJmhwe7hcSYRp4pMsWBs7o6C9+QTkNP4fbnQExDeY/OTVmIni1bcxbzMUptWE3ZBNOgC93mqeAfSms6AaqGxFhdxQei7I6+IPNoKU2O9ry2g8Db4Qwz0OIUJUDLuISMJj8El8+EfPfAvhSdLIdJ7sj3eQwVpZDxp7X7t0P4e5OaxOON21wh2/RPltchW9JsjG0HZrcH0Yv7q/mkJz1jgzcp7imaBurMZ+bgrkEg/n1k+J/dzEHXJZRlBM/nyd9vbSBCTrmqhf6xtuRicjjXH+n2f6NfbFtfcM4bUMLz7D5qLzC2kVKIu1mIUtbiyXiHipgZ3AJwqL86b+k8mxrw30uLExiyWeEXQKnAmOAmwq7umn9lxkZk0oQ4ZEK74c+9mxWLke1c/MitDCGMGopF3/RL0AbwpFLcpV/ga1My7Dvs5eW+EHV9Ks+i4v82V+R3KCGquk1vxEFJ2fxlCAVI61gvw5JC2L3Cis1Rg6Cc/BodMs46Zrl4xnytlHds= ceph-7e53e050-03f4-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T17:45:57.331 INFO:teuthology.orchestra.run.smithi097.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu+NEeVJmhwe7hcSYRp4pMsWBs7o6C9+QTkNP4fbnQExDeY/OTVmIni1bcxbzMUptWE3ZBNOgC93mqeAfSms6AaqGxFhdxQei7I6+IPNoKU2O9ry2g8Db4Qwz0OIUJUDLuISMJj8El8+EfPfAvhSdLIdJ7sj3eQwVpZDxp7X7t0P4e5OaxOON21wh2/RPltchW9JsjG0HZrcH0Yv7q/mkJz1jgzcp7imaBurMZ+bgrkEg/n1k+J/dzEHXJZRlBM/nyd9vbSBCTrmqhf6xtuRicjjXH+n2f6NfbFtfcM4bUMLz7D5qLzC2kVKIu1mIUtbiyXiHipgZ3AJwqL86b+k8mxrw30uLExiyWeEXQKnAmOAmwq7umn9lxkZk0oQ4ZEK74c+9mxWLke1c/MitDCGMGopF3/RL0AbwpFLcpV/ga1My7Dvs5eW+EHV9Ks+i4v82V+R3KCGquk1vxEFJ2fxlCAVI61gvw5JC2L3Cis1Rg6Cc/BodMs46Zrl4xnytlHds= ceph-7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:45:57.343 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu+NEeVJmhwe7hcSYRp4pMsWBs7o6C9+QTkNP4fbnQExDeY/OTVmIni1bcxbzMUptWE3ZBNOgC93mqeAfSms6AaqGxFhdxQei7I6+IPNoKU2O9ry2g8Db4Qwz0OIUJUDLuISMJj8El8+EfPfAvhSdLIdJ7sj3eQwVpZDxp7X7t0P4e5OaxOON21wh2/RPltchW9JsjG0HZrcH0Yv7q/mkJz1jgzcp7imaBurMZ+bgrkEg/n1k+J/dzEHXJZRlBM/nyd9vbSBCTrmqhf6xtuRicjjXH+n2f6NfbFtfcM4bUMLz7D5qLzC2kVKIu1mIUtbiyXiHipgZ3AJwqL86b+k8mxrw30uLExiyWeEXQKnAmOAmwq7umn9lxkZk0oQ4ZEK74c+9mxWLke1c/MitDCGMGopF3/RL0AbwpFLcpV/ga1My7Dvs5eW+EHV9Ks+i4v82V+R3KCGquk1vxEFJ2fxlCAVI61gvw5JC2L3Cis1Rg6Cc/BodMs46Zrl4xnytlHds= ceph-7e53e050-03f4-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T17:45:57.382 INFO:teuthology.orchestra.run.smithi186.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu+NEeVJmhwe7hcSYRp4pMsWBs7o6C9+QTkNP4fbnQExDeY/OTVmIni1bcxbzMUptWE3ZBNOgC93mqeAfSms6AaqGxFhdxQei7I6+IPNoKU2O9ry2g8Db4Qwz0OIUJUDLuISMJj8El8+EfPfAvhSdLIdJ7sj3eQwVpZDxp7X7t0P4e5OaxOON21wh2/RPltchW9JsjG0HZrcH0Yv7q/mkJz1jgzcp7imaBurMZ+bgrkEg/n1k+J/dzEHXJZRlBM/nyd9vbSBCTrmqhf6xtuRicjjXH+n2f6NfbFtfcM4bUMLz7D5qLzC2kVKIu1mIUtbiyXiHipgZ3AJwqL86b+k8mxrw30uLExiyWeEXQKnAmOAmwq7umn9lxkZk0oQ4ZEK74c+9mxWLke1c/MitDCGMGopF3/RL0AbwpFLcpV/ga1My7Dvs5eW+EHV9Ks+i4v82V+R3KCGquk1vxEFJ2fxlCAVI61gvw5JC2L3Cis1Rg6Cc/BodMs46Zrl4xnytlHds= ceph-7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:45:57.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-26T17:45:57.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:45:57 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/41052550' entity='client.admin' 2024-04-26T17:45:57.638 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:45:59.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.849+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/4153856444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b0101d80 msgr2=0x7fa8b01021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:45:59.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.849+0000 7fa8b4d2f700 1 --2- 172.21.15.31:0/4153856444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b0101d80 0x7fa8b01021a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4009910 tx=0x7fa8a4009c20 comp rx=0 tx=0).stop 2024-04-26T17:45:59.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.849+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/4153856444 shutdown_connections 2024-04-26T17:45:59.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.849+0000 7fa8b4d2f700 1 --2- 172.21.15.31:0/4153856444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b0101d80 0x7fa8b01021a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:45:59.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.849+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/4153856444 >> 172.21.15.31:0/4153856444 conn(0x7fa8b00fd320 msgr2=0x7fa8b00ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:45:59.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.849+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/4153856444 shutdown_connections 2024-04-26T17:45:59.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.849+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/4153856444 wait complete. 2024-04-26T17:45:59.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.850+0000 7fa8b4d2f700 1 Processor -- start 2024-04-26T17:45:59.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.850+0000 7fa8b4d2f700 1 -- start start 2024-04-26T17:45:59.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.850+0000 7fa8b4d2f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b0101d80 0x7fa8b01984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:59.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.850+0000 7fa8b4d2f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b0107cc0 con 0x7fa8b0101d80 2024-04-26T17:45:59.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.851+0000 7fa8ae59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b0101d80 0x7fa8b01984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:59.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.851+0000 7fa8ae59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b0101d80 0x7fa8b01984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60190/0 (socket says 172.21.15.31:60190) 2024-04-26T17:45:59.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.851+0000 7fa8ae59c700 1 -- 172.21.15.31:0/926204845 learned_addr learned my addr 172.21.15.31:0/926204845 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:45:59.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.851+0000 7fa8ae59c700 1 -- 172.21.15.31:0/926204845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b01989f0 con 0x7fa8b0101d80 2024-04-26T17:45:59.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.851+0000 7fa8ae59c700 1 --2- 172.21.15.31:0/926204845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b0101d80 0x7fa8b01984b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4005800 tx=0x7fa8a4009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:59.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.851+0000 7fa89f7fe700 1 -- 172.21.15.31:0/926204845 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8a4027070 con 0x7fa8b0101d80 2024-04-26T17:45:59.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.851+0000 7fa89f7fe700 1 -- 172.21.15.31:0/926204845 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8a400be10 con 0x7fa8b0101d80 2024-04-26T17:45:59.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.851+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/926204845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8b0198c80 con 0x7fa8b0101d80 2024-04-26T17:45:59.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.851+0000 7fa89f7fe700 1 -- 172.21.15.31:0/926204845 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8a400f460 con 0x7fa8b0101d80 2024-04-26T17:45:59.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.851+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/926204845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8b019b970 con 0x7fa8b0101d80 2024-04-26T17:45:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.852+0000 7fa89f7fe700 1 -- 172.21.15.31:0/926204845 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fa8a402a070 con 0x7fa8b0101d80 2024-04-26T17:45:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.852+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/926204845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8b0108440 con 0x7fa8b0101d80 2024-04-26T17:45:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.852+0000 7fa89f7fe700 1 --2- 172.21.15.31:0/926204845 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa8980396b0 0x7fa89803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:45:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.852+0000 7fa89f7fe700 1 -- 172.21.15.31:0/926204845 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa8a401e030 con 0x7fa8b0101d80 2024-04-26T17:45:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.853+0000 7fa8add9b700 1 --2- 172.21.15.31:0/926204845 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa8980396b0 0x7fa89803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:45:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.853+0000 7fa8add9b700 1 --2- 172.21.15.31:0/926204845 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa8980396b0 0x7fa89803bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa8a0006fd0 tx=0x7fa8a0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:45:59.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:45:59.858+0000 7fa89f7fe700 1 -- 172.21.15.31:0/926204845 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa8a4013070 con 0x7fa8b0101d80 2024-04-26T17:46:00.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:00.067+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/926204845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fa8b002e5a0 con 0x7fa8b0101d80 2024-04-26T17:46:00.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:00.069+0000 7fa89f7fe700 1 -- 172.21.15.31:0/926204845 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fa8a4031020 con 0x7fa8b0101d80 2024-04-26T17:46:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:00.072+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/926204845 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa8980396b0 msgr2=0x7fa89803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:00.072+0000 7fa8b4d2f700 1 --2- 172.21.15.31:0/926204845 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa8980396b0 0x7fa89803bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa8a0006fd0 tx=0x7fa8a0006e40 comp rx=0 tx=0).stop 2024-04-26T17:46:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:00.072+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/926204845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b0101d80 msgr2=0x7fa8b01984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:00.072+0000 7fa8b4d2f700 1 --2- 172.21.15.31:0/926204845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b0101d80 0x7fa8b01984b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4005800 tx=0x7fa8a4009f60 comp rx=0 tx=0).stop 2024-04-26T17:46:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:00.072+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/926204845 shutdown_connections 2024-04-26T17:46:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:00.072+0000 7fa8b4d2f700 1 --2- 172.21.15.31:0/926204845 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa8980396b0 0x7fa89803bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:00.072+0000 7fa8b4d2f700 1 --2- 172.21.15.31:0/926204845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b0101d80 0x7fa8b01984b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:00.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:00.072+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/926204845 >> 172.21.15.31:0/926204845 conn(0x7fa8b00fd320 msgr2=0x7fa8b006b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:00.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:00.072+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/926204845 shutdown_connections 2024-04-26T17:46:00.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:00.072+0000 7fa8b4d2f700 1 -- 172.21.15.31:0/926204845 wait complete. 2024-04-26T17:46:00.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:00 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:00.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:00 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:00.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:00 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi031", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:46:00.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:00 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:00.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:00 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:00.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:00 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:00.658 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-26T17:46:00.658 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-26T17:46:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:46:01.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:01 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/926204845' entity='client.admin' 2024-04-26T17:46:01.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:01.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:02.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.321+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/79845509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c18101da0 msgr2=0x7f2c181021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:02.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.321+0000 7f2c1d9fd700 1 --2- 172.21.15.31:0/79845509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c18101da0 0x7f2c181021c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009910 tx=0x7f2c08009c20 comp rx=0 tx=0).stop 2024-04-26T17:46:02.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.321+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/79845509 shutdown_connections 2024-04-26T17:46:02.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.321+0000 7f2c1d9fd700 1 --2- 172.21.15.31:0/79845509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c18101da0 0x7f2c181021c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:02.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.321+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/79845509 >> 172.21.15.31:0/79845509 conn(0x7f2c180fd360 msgr2=0x7f2c180ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.321+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/79845509 shutdown_connections 2024-04-26T17:46:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.321+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/79845509 wait complete. 2024-04-26T17:46:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.322+0000 7f2c1d9fd700 1 Processor -- start 2024-04-26T17:46:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.322+0000 7f2c1d9fd700 1 -- start start 2024-04-26T17:46:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.322+0000 7f2c1d9fd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c18101da0 0x7f2c18197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.322+0000 7f2c1d9fd700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2c18107ce0 con 0x7f2c18101da0 2024-04-26T17:46:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.323+0000 7f2c16ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c18101da0 0x7f2c18197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.323+0000 7f2c16ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c18101da0 0x7f2c18197440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60204/0 (socket says 172.21.15.31:60204) 2024-04-26T17:46:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.323+0000 7f2c16ffd700 1 -- 172.21.15.31:0/3405613375 learned_addr learned my addr 172.21.15.31:0/3405613375 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:46:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.323+0000 7f2c16ffd700 1 -- 172.21.15.31:0/3405613375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c18197980 con 0x7f2c18101da0 2024-04-26T17:46:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.323+0000 7f2c16ffd700 1 --2- 172.21.15.31:0/3405613375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c18101da0 0x7f2c18197440 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2c08004fb0 tx=0x7f2c08009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.323+0000 7f2c1c9fb700 1 -- 172.21.15.31:0/3405613375 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c08027070 con 0x7f2c18101da0 2024-04-26T17:46:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.323+0000 7f2c1c9fb700 1 -- 172.21.15.31:0/3405613375 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c0800be10 con 0x7f2c18101da0 2024-04-26T17:46:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.323+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/3405613375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c18197c10 con 0x7f2c18101da0 2024-04-26T17:46:02.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.323+0000 7f2c1c9fb700 1 -- 172.21.15.31:0/3405613375 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c08022de0 con 0x7f2c18101da0 2024-04-26T17:46:02.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.323+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/3405613375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c1819a900 con 0x7f2c18101da0 2024-04-26T17:46:02.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.324+0000 7f2c1c9fb700 1 -- 172.21.15.31:0/3405613375 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2c0802a070 con 0x7f2c18101da0 2024-04-26T17:46:02.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.324+0000 7f2c1c9fb700 1 --2- 172.21.15.31:0/3405613375 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2c04039650 0x7f2c0403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:02.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.324+0000 7f2c1c9fb700 1 -- 172.21.15.31:0/3405613375 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2c0801e030 con 0x7f2c18101da0 2024-04-26T17:46:02.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.324+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/3405613375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c18108460 con 0x7f2c18101da0 2024-04-26T17:46:02.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.324+0000 7f2c167fc700 1 --2- 172.21.15.31:0/3405613375 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2c04039650 0x7f2c0403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:02.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.325+0000 7f2c167fc700 1 --2- 172.21.15.31:0/3405613375 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2c04039650 0x7f2c0403bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2c00006fd0 tx=0x7f2c00006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:02.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.330+0000 7f2c1c9fb700 1 -- 172.21.15.31:0/3405613375 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c08031070 con 0x7f2c18101da0 2024-04-26T17:46:02.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.542+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/3405613375 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f2c180638f0 con 0x7f2c04039650 2024-04-26T17:46:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.545+0000 7f2c1c9fb700 1 -- 172.21.15.31:0/3405613375 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2c180638f0 con 0x7f2c04039650 2024-04-26T17:46:02.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.547+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/3405613375 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2c04039650 msgr2=0x7f2c0403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.547+0000 7f2c1d9fd700 1 --2- 172.21.15.31:0/3405613375 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2c04039650 0x7f2c0403bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2c00006fd0 tx=0x7f2c00006e40 comp rx=0 tx=0).stop 2024-04-26T17:46:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.547+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/3405613375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c18101da0 msgr2=0x7f2c18197440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.547+0000 7f2c1d9fd700 1 --2- 172.21.15.31:0/3405613375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c18101da0 0x7f2c18197440 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2c08004fb0 tx=0x7f2c08009f60 comp rx=0 tx=0).stop 2024-04-26T17:46:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.548+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/3405613375 shutdown_connections 2024-04-26T17:46:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.548+0000 7f2c1d9fd700 1 --2- 172.21.15.31:0/3405613375 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2c04039650 0x7f2c0403bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.548+0000 7f2c1d9fd700 1 --2- 172.21.15.31:0/3405613375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c18101da0 0x7f2c18197440 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.548+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/3405613375 >> 172.21.15.31:0/3405613375 conn(0x7f2c180fd360 msgr2=0x7f2c180fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.548+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/3405613375 shutdown_connections 2024-04-26T17:46:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:02.548+0000 7f2c1d9fd700 1 -- 172.21.15.31:0/3405613375 wait complete. 2024-04-26T17:46:03.275 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi097 2024-04-26T17:46:03.275 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-26T17:46:03.275 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/etc/ceph/ceph.conf 2024-04-26T17:46:03.295 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-26T17:46:03.295 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:46:03.351 INFO:tasks.cephadm:Adding host smithi097 to orchestrator... 2024-04-26T17:46:03.352 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch host add smithi097 2024-04-26T17:46:03.598 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:46:03.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:03 smithi031 ceph-mon[29164]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:46:03.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:03.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:03.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:03.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:03.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:03 smithi031 ceph-mon[29164]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-26T17:46:03.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:03 smithi031 ceph-mon[29164]: Updating smithi031:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:03.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:03 smithi031 ceph-mon[29164]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:46:03.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:03.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:03.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:04.634 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:04 smithi031 ceph-mon[29164]: Updating smithi031:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:46:05.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.078+0000 7fd646ca0700 1 -- 172.21.15.31:0/1831372523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6401030c0 msgr2=0x7fd6401034a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:05.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.078+0000 7fd646ca0700 1 --2- 172.21.15.31:0/1831372523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6401030c0 0x7fd6401034a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd634009910 tx=0x7fd634009c20 comp rx=0 tx=0).stop 2024-04-26T17:46:05.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.078+0000 7fd646ca0700 1 -- 172.21.15.31:0/1831372523 shutdown_connections 2024-04-26T17:46:05.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.078+0000 7fd646ca0700 1 --2- 172.21.15.31:0/1831372523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6401030c0 0x7fd6401034a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:05.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.078+0000 7fd646ca0700 1 -- 172.21.15.31:0/1831372523 >> 172.21.15.31:0/1831372523 conn(0x7fd6400fe950 msgr2=0x7fd640100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:05.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.078+0000 7fd646ca0700 1 -- 172.21.15.31:0/1831372523 shutdown_connections 2024-04-26T17:46:05.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.078+0000 7fd646ca0700 1 -- 172.21.15.31:0/1831372523 wait complete. 2024-04-26T17:46:05.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.078+0000 7fd646ca0700 1 Processor -- start 2024-04-26T17:46:05.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.078+0000 7fd646ca0700 1 -- start start 2024-04-26T17:46:05.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.079+0000 7fd646ca0700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6401030c0 0x7fd640075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:05.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.079+0000 7fd646ca0700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd64010d340 con 0x7fd6401030c0 2024-04-26T17:46:05.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.079+0000 7fd644a3c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6401030c0 0x7fd640075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:05.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.079+0000 7fd644a3c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6401030c0 0x7fd640075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43478/0 (socket says 172.21.15.31:43478) 2024-04-26T17:46:05.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.079+0000 7fd644a3c700 1 -- 172.21.15.31:0/1598785861 learned_addr learned my addr 172.21.15.31:0/1598785861 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:46:05.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.079+0000 7fd644a3c700 1 -- 172.21.15.31:0/1598785861 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd640077610 con 0x7fd6401030c0 2024-04-26T17:46:05.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.079+0000 7fd644a3c700 1 --2- 172.21.15.31:0/1598785861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6401030c0 0x7fd640075720 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd6340056d0 tx=0x7fd6340057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:05.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.079+0000 7fd63dffb700 1 -- 172.21.15.31:0/1598785861 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd634026070 con 0x7fd6401030c0 2024-04-26T17:46:05.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.079+0000 7fd63dffb700 1 -- 172.21.15.31:0/1598785861 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd63400bc00 con 0x7fd6401030c0 2024-04-26T17:46:05.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.079+0000 7fd646ca0700 1 -- 172.21.15.31:0/1598785861 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd640075cc0 con 0x7fd6401030c0 2024-04-26T17:46:05.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.079+0000 7fd63dffb700 1 -- 172.21.15.31:0/1598785861 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd634003920 con 0x7fd6401030c0 2024-04-26T17:46:05.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.080+0000 7fd646ca0700 1 -- 172.21.15.31:0/1598785861 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd640076180 con 0x7fd6401030c0 2024-04-26T17:46:05.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.080+0000 7fd63dffb700 1 -- 172.21.15.31:0/1598785861 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fd634029070 con 0x7fd6401030c0 2024-04-26T17:46:05.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.080+0000 7fd646ca0700 1 -- 172.21.15.31:0/1598785861 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6401045a0 con 0x7fd6401030c0 2024-04-26T17:46:05.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.081+0000 7fd63dffb700 1 --2- 172.21.15.31:0/1598785861 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd630039650 0x7fd63003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:05.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.081+0000 7fd63dffb700 1 -- 172.21.15.31:0/1598785861 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd63401e070 con 0x7fd6401030c0 2024-04-26T17:46:05.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.081+0000 7fd63ffff700 1 --2- 172.21.15.31:0/1598785861 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd630039650 0x7fd63003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:05.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.081+0000 7fd63ffff700 1 --2- 172.21.15.31:0/1598785861 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd630039650 0x7fd63003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd62c006fd0 tx=0x7fd62c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.086+0000 7fd63dffb700 1 -- 172.21.15.31:0/1598785861 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd634013070 con 0x7fd6401030c0 2024-04-26T17:46:05.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:05.298+0000 7fd646ca0700 1 -- 172.21.15.31:0/1598785861 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi097", "target": ["mon-mgr", ""]}) -- 0x7fd640076410 con 0x7fd630039650 2024-04-26T17:46:06.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:06 smithi031 ceph-mon[29164]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi097", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:46:06.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:06 smithi031 ceph-mon[29164]: Deploying cephadm binary to smithi097 2024-04-26T17:46:09.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:09 smithi031 ceph-mon[29164]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:09.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:09.576+0000 7fd63dffb700 1 -- 172.21.15.31:0/1598785861 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fd640076410 con 0x7fd630039650 2024-04-26T17:46:09.577 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi097' with addr '172.21.15.97' 2024-04-26T17:46:09.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:09.579+0000 7fd646ca0700 1 -- 172.21.15.31:0/1598785861 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd630039650 msgr2=0x7fd63003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:09.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:09.579+0000 7fd646ca0700 1 --2- 172.21.15.31:0/1598785861 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd630039650 0x7fd63003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd62c006fd0 tx=0x7fd62c006e40 comp rx=0 tx=0).stop 2024-04-26T17:46:09.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:09.579+0000 7fd646ca0700 1 -- 172.21.15.31:0/1598785861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6401030c0 msgr2=0x7fd640075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:09.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:09.579+0000 7fd646ca0700 1 --2- 172.21.15.31:0/1598785861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6401030c0 0x7fd640075720 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd6340056d0 tx=0x7fd6340057b0 comp rx=0 tx=0).stop 2024-04-26T17:46:09.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:09.579+0000 7fd646ca0700 1 -- 172.21.15.31:0/1598785861 shutdown_connections 2024-04-26T17:46:09.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:09.579+0000 7fd646ca0700 1 --2- 172.21.15.31:0/1598785861 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd630039650 0x7fd63003bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:09.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:09.579+0000 7fd646ca0700 1 --2- 172.21.15.31:0/1598785861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6401030c0 0x7fd640075720 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:09.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:09.579+0000 7fd646ca0700 1 -- 172.21.15.31:0/1598785861 >> 172.21.15.31:0/1598785861 conn(0x7fd6400fe950 msgr2=0x7fd640104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:09.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:09.579+0000 7fd646ca0700 1 -- 172.21.15.31:0/1598785861 shutdown_connections 2024-04-26T17:46:09.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:09.579+0000 7fd646ca0700 1 -- 172.21.15.31:0/1598785861 wait complete. 2024-04-26T17:46:10.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T17:46:10.447 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:46:10.901 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:10 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:10.902 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:10 smithi031 ceph-mon[29164]: Added host smithi097 2024-04-26T17:46:10.902 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:10 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:10.902 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:10 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:11.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:11 smithi031 ceph-mon[29164]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:11.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.064+0000 7f538860e700 1 -- 172.21.15.31:0/523946169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53801030c0 msgr2=0x7f53801034a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.064+0000 7f538860e700 1 --2- 172.21.15.31:0/523946169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53801030c0 0x7f53801034a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f537c009910 tx=0x7f537c009c20 comp rx=0 tx=0).stop 2024-04-26T17:46:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.064+0000 7f538860e700 1 -- 172.21.15.31:0/523946169 shutdown_connections 2024-04-26T17:46:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.064+0000 7f538860e700 1 --2- 172.21.15.31:0/523946169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53801030c0 0x7f53801034a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.064+0000 7f538860e700 1 -- 172.21.15.31:0/523946169 >> 172.21.15.31:0/523946169 conn(0x7f53800fe950 msgr2=0x7f5380100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.064+0000 7f538860e700 1 -- 172.21.15.31:0/523946169 shutdown_connections 2024-04-26T17:46:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.064+0000 7f538860e700 1 -- 172.21.15.31:0/523946169 wait complete. 2024-04-26T17:46:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.065+0000 7f538860e700 1 Processor -- start 2024-04-26T17:46:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.065+0000 7f538860e700 1 -- start start 2024-04-26T17:46:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.065+0000 7f538860e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53801030c0 0x7f53801a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.065+0000 7f538860e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f538010d340 con 0x7f53801030c0 2024-04-26T17:46:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.065+0000 7f53863aa700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53801030c0 0x7f53801a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.065+0000 7f53863aa700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53801030c0 0x7f53801a0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43496/0 (socket says 172.21.15.31:43496) 2024-04-26T17:46:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.065+0000 7f53863aa700 1 -- 172.21.15.31:0/2470422089 learned_addr learned my addr 172.21.15.31:0/2470422089 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:46:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.066+0000 7f53863aa700 1 -- 172.21.15.31:0/2470422089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53801a0c60 con 0x7f53801030c0 2024-04-26T17:46:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.066+0000 7f53863aa700 1 --2- 172.21.15.31:0/2470422089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53801030c0 0x7f53801a0720 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f537c0056d0 tx=0x7f537c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.066+0000 7f53777fe700 1 -- 172.21.15.31:0/2470422089 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f537c026070 con 0x7f53801030c0 2024-04-26T17:46:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.066+0000 7f53777fe700 1 -- 172.21.15.31:0/2470422089 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f537c00bba0 con 0x7f53801030c0 2024-04-26T17:46:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.066+0000 7f538860e700 1 -- 172.21.15.31:0/2470422089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53801a0f50 con 0x7f53801030c0 2024-04-26T17:46:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.066+0000 7f53777fe700 1 -- 172.21.15.31:0/2470422089 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f537c003920 con 0x7f53801030c0 2024-04-26T17:46:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.066+0000 7f538860e700 1 -- 172.21.15.31:0/2470422089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f538019a8e0 con 0x7f53801030c0 2024-04-26T17:46:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.067+0000 7f53777fe700 1 -- 172.21.15.31:0/2470422089 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f537c029070 con 0x7f53801030c0 2024-04-26T17:46:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.067+0000 7f538860e700 1 -- 172.21.15.31:0/2470422089 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53801045a0 con 0x7f53801030c0 2024-04-26T17:46:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.067+0000 7f53777fe700 1 --2- 172.21.15.31:0/2470422089 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f536c0396b0 0x7f536c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.067+0000 7f53777fe700 1 -- 172.21.15.31:0/2470422089 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f537c01e070 con 0x7f53801030c0 2024-04-26T17:46:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.067+0000 7f5385ba9700 1 --2- 172.21.15.31:0/2470422089 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f536c0396b0 0x7f536c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.068+0000 7f5385ba9700 1 --2- 172.21.15.31:0/2470422089 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f536c0396b0 0x7f536c03bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5370006fd0 tx=0x7f5370006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:12.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.072+0000 7f53777fe700 1 -- 172.21.15.31:0/2470422089 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f537c013070 con 0x7f53801030c0 2024-04-26T17:46:12.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.286+0000 7f538860e700 1 -- 172.21.15.31:0/2470422089 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f538019be80 con 0x7f536c0396b0 2024-04-26T17:46:12.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.288+0000 7f53777fe700 1 -- 172.21.15.31:0/2470422089 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f538019be80 con 0x7f536c0396b0 2024-04-26T17:46:12.287 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:46:12.288 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.97", "hostname": "smithi097", "labels": [], "status": ""}] 2024-04-26T17:46:12.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.290+0000 7f538860e700 1 -- 172.21.15.31:0/2470422089 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f536c0396b0 msgr2=0x7f536c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:12.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.290+0000 7f538860e700 1 --2- 172.21.15.31:0/2470422089 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f536c0396b0 0x7f536c03bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5370006fd0 tx=0x7f5370006e40 comp rx=0 tx=0).stop 2024-04-26T17:46:12.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.290+0000 7f538860e700 1 -- 172.21.15.31:0/2470422089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53801030c0 msgr2=0x7f53801a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:12.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.290+0000 7f538860e700 1 --2- 172.21.15.31:0/2470422089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53801030c0 0x7f53801a0720 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f537c0056d0 tx=0x7f537c0057b0 comp rx=0 tx=0).stop 2024-04-26T17:46:12.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.290+0000 7f538860e700 1 -- 172.21.15.31:0/2470422089 shutdown_connections 2024-04-26T17:46:12.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.290+0000 7f538860e700 1 --2- 172.21.15.31:0/2470422089 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f536c0396b0 0x7f536c03bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:12.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.290+0000 7f538860e700 1 --2- 172.21.15.31:0/2470422089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53801030c0 0x7f53801a0720 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:12.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.290+0000 7f538860e700 1 -- 172.21.15.31:0/2470422089 >> 172.21.15.31:0/2470422089 conn(0x7f53800fe950 msgr2=0x7f5380075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:12.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.290+0000 7f538860e700 1 -- 172.21.15.31:0/2470422089 shutdown_connections 2024-04-26T17:46:12.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:12.290+0000 7f538860e700 1 -- 172.21.15.31:0/2470422089 wait complete. 2024-04-26T17:46:12.849 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi186 2024-04-26T17:46:12.850 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T17:46:12.850 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.conf 2024-04-26T17:46:12.868 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T17:46:12.868 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:46:12.925 INFO:tasks.cephadm:Adding host smithi186 to orchestrator... 2024-04-26T17:46:12.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch host add smithi186 2024-04-26T17:46:13.169 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:46:13.498 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:13 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:13.498 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:13 smithi031 ceph-mon[29164]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:13.498 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:13 smithi031 ceph-mon[29164]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:46:14.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.699+0000 7f8925dc4700 1 -- 172.21.15.31:0/2226279556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89201030c0 msgr2=0x7f89201034a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:14.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.699+0000 7f8925dc4700 1 --2- 172.21.15.31:0/2226279556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89201030c0 0x7f89201034a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8918009910 tx=0x7f8918009c20 comp rx=0 tx=0).stop 2024-04-26T17:46:14.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.699+0000 7f8925dc4700 1 -- 172.21.15.31:0/2226279556 shutdown_connections 2024-04-26T17:46:14.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.699+0000 7f8925dc4700 1 --2- 172.21.15.31:0/2226279556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89201030c0 0x7f89201034a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:14.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.699+0000 7f8925dc4700 1 -- 172.21.15.31:0/2226279556 >> 172.21.15.31:0/2226279556 conn(0x7f89200fe950 msgr2=0x7f8920100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:14.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.699+0000 7f8925dc4700 1 -- 172.21.15.31:0/2226279556 shutdown_connections 2024-04-26T17:46:14.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.699+0000 7f8925dc4700 1 -- 172.21.15.31:0/2226279556 wait complete. 2024-04-26T17:46:14.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.700+0000 7f8925dc4700 1 Processor -- start 2024-04-26T17:46:14.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.700+0000 7f8925dc4700 1 -- start start 2024-04-26T17:46:14.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.700+0000 7f8925dc4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89201030c0 0x7f892019e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:14.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.700+0000 7f8925dc4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f892010d340 con 0x7f89201030c0 2024-04-26T17:46:14.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.700+0000 7f891f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89201030c0 0x7f892019e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:14.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.700+0000 7f891f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89201030c0 0x7f892019e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47806/0 (socket says 172.21.15.31:47806) 2024-04-26T17:46:14.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.700+0000 7f891f7fe700 1 -- 172.21.15.31:0/3792031193 learned_addr learned my addr 172.21.15.31:0/3792031193 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:46:14.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.700+0000 7f891f7fe700 1 -- 172.21.15.31:0/3792031193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f892019eb10 con 0x7f89201030c0 2024-04-26T17:46:14.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.701+0000 7f891f7fe700 1 --2- 172.21.15.31:0/3792031193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89201030c0 0x7f892019e5d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f89180056d0 tx=0x7f89180057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:14.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.701+0000 7f891cff9700 1 -- 172.21.15.31:0/3792031193 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8918026070 con 0x7f89201030c0 2024-04-26T17:46:14.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.701+0000 7f891cff9700 1 -- 172.21.15.31:0/3792031193 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f891800bba0 con 0x7f89201030c0 2024-04-26T17:46:14.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.701+0000 7f8925dc4700 1 -- 172.21.15.31:0/3792031193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f892019ee00 con 0x7f89201030c0 2024-04-26T17:46:14.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.701+0000 7f891cff9700 1 -- 172.21.15.31:0/3792031193 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8918003920 con 0x7f89201030c0 2024-04-26T17:46:14.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.701+0000 7f8925dc4700 1 -- 172.21.15.31:0/3792031193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8920198870 con 0x7f89201030c0 2024-04-26T17:46:14.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.702+0000 7f891cff9700 1 -- 172.21.15.31:0/3792031193 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8918029070 con 0x7f89201030c0 2024-04-26T17:46:14.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.702+0000 7f8925dc4700 1 -- 172.21.15.31:0/3792031193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89201045a0 con 0x7f89201030c0 2024-04-26T17:46:14.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.702+0000 7f891cff9700 1 --2- 172.21.15.31:0/3792031193 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f890c0396b0 0x7f890c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:14.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.702+0000 7f891cff9700 1 -- 172.21.15.31:0/3792031193 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f891801e070 con 0x7f89201030c0 2024-04-26T17:46:14.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.702+0000 7f891effd700 1 --2- 172.21.15.31:0/3792031193 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f890c0396b0 0x7f890c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:14.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.703+0000 7f891effd700 1 --2- 172.21.15.31:0/3792031193 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f890c0396b0 0x7f890c03bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8908006fd0 tx=0x7f8908006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:14.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.707+0000 7f891cff9700 1 -- 172.21.15.31:0/3792031193 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8918013070 con 0x7f89201030c0 2024-04-26T17:46:14.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:14.922+0000 7f8925dc4700 1 -- 172.21.15.31:0/3792031193 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}) -- 0x7f8920199bd0 con 0x7f890c0396b0 2024-04-26T17:46:15.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:15 smithi031 ceph-mon[29164]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:15.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:15 smithi031 ceph-mon[29164]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:46:16.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:16.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:16.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:16.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:16.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi097", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:46:16.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:16.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:16.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:16 smithi031 ceph-mon[29164]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-26T17:46:16.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:16 smithi031 ceph-mon[29164]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:16.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:16 smithi031 ceph-mon[29164]: Deploying cephadm binary to smithi186 2024-04-26T17:46:16.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:16 smithi031 ceph-mon[29164]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:46:17.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:17 smithi031 ceph-mon[29164]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:46:17.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:17 smithi031 ceph-mon[29164]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:17.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:17.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:17.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:19.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:19.459+0000 7f891cff9700 1 -- 172.21.15.31:0/3792031193 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f8920199bd0 con 0x7f890c0396b0 2024-04-26T17:46:19.460 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi186' with addr '172.21.15.186' 2024-04-26T17:46:19.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:19.462+0000 7f8925dc4700 1 -- 172.21.15.31:0/3792031193 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f890c0396b0 msgr2=0x7f890c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:19.462+0000 7f8925dc4700 1 --2- 172.21.15.31:0/3792031193 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f890c0396b0 0x7f890c03bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8908006fd0 tx=0x7f8908006e40 comp rx=0 tx=0).stop 2024-04-26T17:46:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:19.463+0000 7f8925dc4700 1 -- 172.21.15.31:0/3792031193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89201030c0 msgr2=0x7f892019e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:19.463+0000 7f8925dc4700 1 --2- 172.21.15.31:0/3792031193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89201030c0 0x7f892019e5d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f89180056d0 tx=0x7f89180057b0 comp rx=0 tx=0).stop 2024-04-26T17:46:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:19.463+0000 7f8925dc4700 1 -- 172.21.15.31:0/3792031193 shutdown_connections 2024-04-26T17:46:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:19.463+0000 7f8925dc4700 1 --2- 172.21.15.31:0/3792031193 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f890c0396b0 0x7f890c03bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:19.463+0000 7f8925dc4700 1 --2- 172.21.15.31:0/3792031193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89201030c0 0x7f892019e5d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:19.463+0000 7f8925dc4700 1 -- 172.21.15.31:0/3792031193 >> 172.21.15.31:0/3792031193 conn(0x7f89200fe950 msgr2=0x7f8920104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:19.463+0000 7f8925dc4700 1 -- 172.21.15.31:0/3792031193 shutdown_connections 2024-04-26T17:46:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:19.463+0000 7f8925dc4700 1 -- 172.21.15.31:0/3792031193 wait complete. 2024-04-26T17:46:19.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:19 smithi031 ceph-mon[29164]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:20.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T17:46:20.363 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:46:20.777 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:20 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:20.777 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:20 smithi031 ceph-mon[29164]: Added host smithi186 2024-04-26T17:46:20.777 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:20 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:20.777 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:20 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:21.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:21 smithi031 ceph-mon[29164]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:21.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:21 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.929+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/3017449392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28108fb0 msgr2=0x7f0f28109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.929+0000 7f0f2d7d3700 1 --2- 172.21.15.31:0/3017449392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28108fb0 0x7f0f28109390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009910 tx=0x7f0f18009c20 comp rx=0 tx=0).stop 2024-04-26T17:46:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.929+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/3017449392 shutdown_connections 2024-04-26T17:46:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.929+0000 7f0f2d7d3700 1 --2- 172.21.15.31:0/3017449392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28108fb0 0x7f0f28109390 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.929+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/3017449392 >> 172.21.15.31:0/3017449392 conn(0x7f0f280784a0 msgr2=0x7f0f280788b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.929+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/3017449392 shutdown_connections 2024-04-26T17:46:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.929+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/3017449392 wait complete. 2024-04-26T17:46:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.929+0000 7f0f2d7d3700 1 Processor -- start 2024-04-26T17:46:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.930+0000 7f0f2d7d3700 1 -- start start 2024-04-26T17:46:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.930+0000 7f0f2d7d3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28108fb0 0x7f0f2819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.930+0000 7f0f2d7d3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2810d460 con 0x7f0f28108fb0 2024-04-26T17:46:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.930+0000 7f0f26ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28108fb0 0x7f0f2819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.930+0000 7f0f26ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28108fb0 0x7f0f2819e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47828/0 (socket says 172.21.15.31:47828) 2024-04-26T17:46:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.930+0000 7f0f26ffd700 1 -- 172.21.15.31:0/1680515804 learned_addr learned my addr 172.21.15.31:0/1680515804 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:46:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.930+0000 7f0f26ffd700 1 -- 172.21.15.31:0/1680515804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2819ebc0 con 0x7f0f28108fb0 2024-04-26T17:46:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.931+0000 7f0f26ffd700 1 --2- 172.21.15.31:0/1680515804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28108fb0 0x7f0f2819e680 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0f18006010 tx=0x7f0f18005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.931+0000 7f0f0ffff700 1 -- 172.21.15.31:0/1680515804 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f18026070 con 0x7f0f28108fb0 2024-04-26T17:46:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.931+0000 7f0f0ffff700 1 -- 172.21.15.31:0/1680515804 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f1800bc00 con 0x7f0f28108fb0 2024-04-26T17:46:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.931+0000 7f0f0ffff700 1 -- 172.21.15.31:0/1680515804 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f18003910 con 0x7f0f28108fb0 2024-04-26T17:46:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.931+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/1680515804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f2819ee50 con 0x7f0f28108fb0 2024-04-26T17:46:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.931+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/1680515804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f28198750 con 0x7f0f28108fb0 2024-04-26T17:46:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.932+0000 7f0f0ffff700 1 -- 172.21.15.31:0/1680515804 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0f18029070 con 0x7f0f28108fb0 2024-04-26T17:46:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.932+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/1680515804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f281046d0 con 0x7f0f28108fb0 2024-04-26T17:46:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.932+0000 7f0f0ffff700 1 --2- 172.21.15.31:0/1680515804 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f0f100396b0 0x7f0f1003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.932+0000 7f0f0ffff700 1 -- 172.21.15.31:0/1680515804 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0f1801e070 con 0x7f0f28108fb0 2024-04-26T17:46:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.932+0000 7f0f267fc700 1 --2- 172.21.15.31:0/1680515804 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f0f100396b0 0x7f0f1003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.932+0000 7f0f267fc700 1 --2- 172.21.15.31:0/1680515804 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f0f100396b0 0x7f0f1003bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c006fd0 tx=0x7f0f1c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:21.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:21.937+0000 7f0f0ffff700 1 -- 172.21.15.31:0/1680515804 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0f18013070 con 0x7f0f28108fb0 2024-04-26T17:46:22.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:22.154+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/1680515804 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0f28199c80 con 0x7f0f100396b0 2024-04-26T17:46:22.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:22.155+0000 7f0f0ffff700 1 -- 172.21.15.31:0/1680515804 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f0f28199c80 con 0x7f0f100396b0 2024-04-26T17:46:22.155 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:46:22.155 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.97", "hostname": "smithi097", "labels": [], "status": ""}, {"addr": "172.21.15.186", "hostname": "smithi186", "labels": [], "status": ""}] 2024-04-26T17:46:22.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:22.157+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/1680515804 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f0f100396b0 msgr2=0x7f0f1003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:22.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:22.157+0000 7f0f2d7d3700 1 --2- 172.21.15.31:0/1680515804 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f0f100396b0 0x7f0f1003bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c006fd0 tx=0x7f0f1c006e40 comp rx=0 tx=0).stop 2024-04-26T17:46:22.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:22.157+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/1680515804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28108fb0 msgr2=0x7f0f2819e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:22.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:22.157+0000 7f0f2d7d3700 1 --2- 172.21.15.31:0/1680515804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28108fb0 0x7f0f2819e680 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0f18006010 tx=0x7f0f18005780 comp rx=0 tx=0).stop 2024-04-26T17:46:22.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:22.157+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/1680515804 shutdown_connections 2024-04-26T17:46:22.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:22.157+0000 7f0f2d7d3700 1 --2- 172.21.15.31:0/1680515804 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f0f100396b0 0x7f0f1003bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:22.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:22.157+0000 7f0f2d7d3700 1 --2- 172.21.15.31:0/1680515804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28108fb0 0x7f0f2819e680 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:22.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:22.157+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/1680515804 >> 172.21.15.31:0/1680515804 conn(0x7f0f280784a0 msgr2=0x7f0f28104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:22.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:22.157+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/1680515804 shutdown_connections 2024-04-26T17:46:22.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:22.157+0000 7f0f2d7d3700 1 -- 172.21.15.31:0/1680515804 wait complete. 2024-04-26T17:46:22.691 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-26T17:46:22.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-26T17:46:22.932 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:46:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:23 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:24.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:24 smithi031 ceph-mon[29164]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:46:24.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:24 smithi031 ceph-mon[29164]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:24.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.342+0000 7f0c9edf2700 1 -- 172.21.15.31:0/628620452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c98075740 msgr2=0x7f0c98075b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.342+0000 7f0c9edf2700 1 --2- 172.21.15.31:0/628620452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c98075740 0x7f0c98075b20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009910 tx=0x7f0c8c009c20 comp rx=0 tx=0).stop 2024-04-26T17:46:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.342+0000 7f0c9edf2700 1 -- 172.21.15.31:0/628620452 shutdown_connections 2024-04-26T17:46:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.342+0000 7f0c9edf2700 1 --2- 172.21.15.31:0/628620452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c98075740 0x7f0c98075b20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.342+0000 7f0c9edf2700 1 -- 172.21.15.31:0/628620452 >> 172.21.15.31:0/628620452 conn(0x7f0c980fe960 msgr2=0x7f0c98100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.342+0000 7f0c9edf2700 1 -- 172.21.15.31:0/628620452 shutdown_connections 2024-04-26T17:46:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.342+0000 7f0c9edf2700 1 -- 172.21.15.31:0/628620452 wait complete. 2024-04-26T17:46:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.342+0000 7f0c9edf2700 1 Processor -- start 2024-04-26T17:46:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.342+0000 7f0c9edf2700 1 -- start start 2024-04-26T17:46:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.343+0000 7f0c9edf2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c98075740 0x7f0c981a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:24.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.343+0000 7f0c9edf2700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9810f570 con 0x7f0c98075740 2024-04-26T17:46:24.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.343+0000 7f0c9cb8e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c98075740 0x7f0c981a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:24.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.343+0000 7f0c9cb8e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c98075740 0x7f0c981a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47850/0 (socket says 172.21.15.31:47850) 2024-04-26T17:46:24.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.343+0000 7f0c9cb8e700 1 -- 172.21.15.31:0/2896908694 learned_addr learned my addr 172.21.15.31:0/2896908694 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:46:24.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.343+0000 7f0c9cb8e700 1 -- 172.21.15.31:0/2896908694 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c981a0c90 con 0x7f0c98075740 2024-04-26T17:46:24.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.343+0000 7f0c9cb8e700 1 --2- 172.21.15.31:0/2896908694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c98075740 0x7f0c981a0750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0056d0 tx=0x7f0c8c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:24.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.343+0000 7f0c95ffb700 1 -- 172.21.15.31:0/2896908694 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c8c026070 con 0x7f0c98075740 2024-04-26T17:46:24.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.343+0000 7f0c95ffb700 1 -- 172.21.15.31:0/2896908694 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c8c004370 con 0x7f0c98075740 2024-04-26T17:46:24.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.343+0000 7f0c9edf2700 1 -- 172.21.15.31:0/2896908694 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c981a0f20 con 0x7f0c98075740 2024-04-26T17:46:24.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.343+0000 7f0c95ffb700 1 -- 172.21.15.31:0/2896908694 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c8c0039b0 con 0x7f0c98075740 2024-04-26T17:46:24.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.344+0000 7f0c9edf2700 1 -- 172.21.15.31:0/2896908694 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c9819a910 con 0x7f0c98075740 2024-04-26T17:46:24.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.344+0000 7f0c95ffb700 1 -- 172.21.15.31:0/2896908694 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0c8c029070 con 0x7f0c98075740 2024-04-26T17:46:24.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.344+0000 7f0c9edf2700 1 -- 172.21.15.31:0/2896908694 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c98076e80 con 0x7f0c98075740 2024-04-26T17:46:24.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.345+0000 7f0c95ffb700 1 --2- 172.21.15.31:0/2896908694 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f0c880396b0 0x7f0c8803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:24.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.345+0000 7f0c95ffb700 1 -- 172.21.15.31:0/2896908694 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0c8c01e070 con 0x7f0c98075740 2024-04-26T17:46:24.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.345+0000 7f0c97fff700 1 --2- 172.21.15.31:0/2896908694 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f0c880396b0 0x7f0c8803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:24.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.345+0000 7f0c97fff700 1 --2- 172.21.15.31:0/2896908694 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f0c880396b0 0x7f0c8803bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0c84006fd0 tx=0x7f0c84006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:24.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.350+0000 7f0c95ffb700 1 -- 172.21.15.31:0/2896908694 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0c8c013070 con 0x7f0c98075740 2024-04-26T17:46:24.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:24.565+0000 7f0c9edf2700 1 -- 172.21.15.31:0/2896908694 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f0c9819be40 con 0x7f0c98075740 2024-04-26T17:46:25.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:25.054+0000 7f0c95ffb700 1 -- 172.21.15.31:0/2896908694 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f0c8c00e3e0 con 0x7f0c98075740 2024-04-26T17:46:25.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:25.057+0000 7f0c9edf2700 1 -- 172.21.15.31:0/2896908694 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f0c880396b0 msgr2=0x7f0c8803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:25.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:25.057+0000 7f0c9edf2700 1 --2- 172.21.15.31:0/2896908694 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f0c880396b0 0x7f0c8803bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0c84006fd0 tx=0x7f0c84006e40 comp rx=0 tx=0).stop 2024-04-26T17:46:25.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:25.057+0000 7f0c9edf2700 1 -- 172.21.15.31:0/2896908694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c98075740 msgr2=0x7f0c981a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:25.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:25.057+0000 7f0c9edf2700 1 --2- 172.21.15.31:0/2896908694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c98075740 0x7f0c981a0750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0056d0 tx=0x7f0c8c0057b0 comp rx=0 tx=0).stop 2024-04-26T17:46:25.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:25.057+0000 7f0c9edf2700 1 -- 172.21.15.31:0/2896908694 shutdown_connections 2024-04-26T17:46:25.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:25.057+0000 7f0c9edf2700 1 --2- 172.21.15.31:0/2896908694 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f0c880396b0 0x7f0c8803bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:25.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:25.057+0000 7f0c9edf2700 1 --2- 172.21.15.31:0/2896908694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c98075740 0x7f0c981a0750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:25.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:25.057+0000 7f0c9edf2700 1 -- 172.21.15.31:0/2896908694 >> 172.21.15.31:0/2896908694 conn(0x7f0c980fe960 msgr2=0x7f0c98107150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:25.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:25.057+0000 7f0c9edf2700 1 -- 172.21.15.31:0/2896908694 shutdown_connections 2024-04-26T17:46:25.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:25.057+0000 7f0c9edf2700 1 -- 172.21.15.31:0/2896908694 wait complete. 2024-04-26T17:46:25.059 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2024-04-26T17:46:25.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:25 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2896908694' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T17:46:25.730 INFO:tasks.cephadm:Adding mon.a on smithi031 2024-04-26T17:46:25.730 INFO:tasks.cephadm:Adding mon.b on smithi097 2024-04-26T17:46:25.731 INFO:tasks.cephadm:Adding mon.c on smithi186 2024-04-26T17:46:25.731 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi031:172.21.15.31=a;smithi097:172.21.15.97=b;smithi186:172.21.15.186=c' 2024-04-26T17:46:25.965 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:26.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:26 smithi031 ceph-mon[29164]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:26.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:26 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2896908694' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T17:46:26.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:26 smithi031 ceph-mon[29164]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T17:46:26.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:26.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:26.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:26.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:26.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:46:26.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:26.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:26.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:26.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:26.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:27.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:27 smithi031 ceph-mon[29164]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T17:46:27.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:27 smithi031 ceph-mon[29164]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:27.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:27 smithi031 ceph-mon[29164]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:46:27.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:27 smithi031 ceph-mon[29164]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:46:27.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.464+0000 7f3327006700 1 -- 172.21.15.186:0/1482133877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3320101610 msgr2=0x7f33201019f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:27.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.464+0000 7f3327006700 1 --2- 172.21.15.186:0/1482133877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3320101610 0x7f33201019f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f331c009910 tx=0x7f331c009c20 comp rx=0 tx=0).stop 2024-04-26T17:46:27.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.464+0000 7f3327006700 1 -- 172.21.15.186:0/1482133877 shutdown_connections 2024-04-26T17:46:27.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.464+0000 7f3327006700 1 --2- 172.21.15.186:0/1482133877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3320101610 0x7f33201019f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:27.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.464+0000 7f3327006700 1 -- 172.21.15.186:0/1482133877 >> 172.21.15.186:0/1482133877 conn(0x7f33200784a0 msgr2=0x7f33200788b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:27.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.464+0000 7f3327006700 1 -- 172.21.15.186:0/1482133877 shutdown_connections 2024-04-26T17:46:27.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.464+0000 7f3327006700 1 -- 172.21.15.186:0/1482133877 wait complete. 2024-04-26T17:46:27.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.465+0000 7f3327006700 1 Processor -- start 2024-04-26T17:46:27.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.465+0000 7f3327006700 1 -- start start 2024-04-26T17:46:27.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.465+0000 7f3327006700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3320101610 0x7f332019e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:27.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.465+0000 7f3327006700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f332010d350 con 0x7f3320101610 2024-04-26T17:46:27.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.466+0000 7f3324da2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3320101610 0x7f332019e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:27.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.466+0000 7f3324da2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3320101610 0x7f332019e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.186:54956/0 (socket says 172.21.15.186:54956) 2024-04-26T17:46:27.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.466+0000 7f3324da2700 1 -- 172.21.15.186:0/2861906177 learned_addr learned my addr 172.21.15.186:0/2861906177 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T17:46:27.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.466+0000 7f3324da2700 1 -- 172.21.15.186:0/2861906177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f332019eb10 con 0x7f3320101610 2024-04-26T17:46:27.467 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.466+0000 7f3324da2700 1 --2- 172.21.15.186:0/2861906177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3320101610 0x7f332019e5d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f331c0056d0 tx=0x7f331c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:27.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.466+0000 7f3315ffb700 1 -- 172.21.15.186:0/2861906177 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f331c026070 con 0x7f3320101610 2024-04-26T17:46:27.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.466+0000 7f3327006700 1 -- 172.21.15.186:0/2861906177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f332019ee00 con 0x7f3320101610 2024-04-26T17:46:27.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.466+0000 7f3315ffb700 1 -- 172.21.15.186:0/2861906177 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f331c00bba0 con 0x7f3320101610 2024-04-26T17:46:27.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.466+0000 7f3315ffb700 1 -- 172.21.15.186:0/2861906177 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f331c003920 con 0x7f3320101610 2024-04-26T17:46:27.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.467+0000 7f3327006700 1 -- 172.21.15.186:0/2861906177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3320198870 con 0x7f3320101610 2024-04-26T17:46:27.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.467+0000 7f3327006700 1 -- 172.21.15.186:0/2861906177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33201046d0 con 0x7f3320101610 2024-04-26T17:46:27.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.467+0000 7f3315ffb700 1 -- 172.21.15.186:0/2861906177 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f331c029070 con 0x7f3320101610 2024-04-26T17:46:27.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.468+0000 7f3315ffb700 1 --2- 172.21.15.186:0/2861906177 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3308039650 0x7f330803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:27.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.468+0000 7f3315ffb700 1 -- 172.21.15.186:0/2861906177 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f331c01e070 con 0x7f3320101610 2024-04-26T17:46:27.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.468+0000 7f3317fff700 1 --2- 172.21.15.186:0/2861906177 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3308039650 0x7f330803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:27.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.468+0000 7f3317fff700 1 --2- 172.21.15.186:0/2861906177 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3308039650 0x7f330803bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3310006fd0 tx=0x7f3310006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:27.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.473+0000 7f3315ffb700 1 -- 172.21.15.186:0/2861906177 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f331c013070 con 0x7f3320101610 2024-04-26T17:46:27.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.681+0000 7f3327006700 1 -- 172.21.15.186:0/2861906177 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi097:172.21.15.97=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}) -- 0x7f3320199c90 con 0x7f3308039650 2024-04-26T17:46:27.687 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.687+0000 7f3315ffb700 1 -- 172.21.15.186:0/2861906177 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3320199c90 con 0x7f3308039650 2024-04-26T17:46:27.687 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mon update... 2024-04-26T17:46:27.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.689+0000 7f3327006700 1 -- 172.21.15.186:0/2861906177 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3308039650 msgr2=0x7f330803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:27.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.689+0000 7f3327006700 1 --2- 172.21.15.186:0/2861906177 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3308039650 0x7f330803bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3310006fd0 tx=0x7f3310006e40 comp rx=0 tx=0).stop 2024-04-26T17:46:27.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.689+0000 7f3327006700 1 -- 172.21.15.186:0/2861906177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3320101610 msgr2=0x7f332019e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:27.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.689+0000 7f3327006700 1 --2- 172.21.15.186:0/2861906177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3320101610 0x7f332019e5d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f331c0056d0 tx=0x7f331c0057b0 comp rx=0 tx=0).stop 2024-04-26T17:46:27.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.689+0000 7f3327006700 1 -- 172.21.15.186:0/2861906177 shutdown_connections 2024-04-26T17:46:27.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.689+0000 7f3327006700 1 --2- 172.21.15.186:0/2861906177 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3308039650 0x7f330803bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:27.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.689+0000 7f3327006700 1 --2- 172.21.15.186:0/2861906177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3320101610 0x7f332019e5d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:27.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.689+0000 7f3327006700 1 -- 172.21.15.186:0/2861906177 >> 172.21.15.186:0/2861906177 conn(0x7f33200784a0 msgr2=0x7f3320104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:27.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.689+0000 7f3327006700 1 -- 172.21.15.186:0/2861906177 shutdown_connections 2024-04-26T17:46:27.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:27.690+0000 7f3327006700 1 -- 172.21.15.186:0/2861906177 wait complete. 2024-04-26T17:46:28.259 DEBUG:teuthology.orchestra.run.smithi097:mon.b> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.b.service 2024-04-26T17:46:28.262 DEBUG:teuthology.orchestra.run.smithi186:mon.c> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.c.service 2024-04-26T17:46:28.264 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T17:46:28.265 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T17:46:28.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:28 smithi031 ceph-mon[29164]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:28.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:28 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:28.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:28 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:28.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:28 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:28.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:28 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:28.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:28 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:28.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:28 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:28.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:28 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:29.414 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:29 smithi031 ceph-mon[29164]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi097:172.21.15.97=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:46:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:29 smithi031 ceph-mon[29164]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi097:172.21.15.97=b;smithi186:172.21.15.186=c;count:3 2024-04-26T17:46:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:29 smithi031 ceph-mon[29164]: Deploying daemon mon.c on smithi186 2024-04-26T17:46:30.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:30 smithi031 ceph-mon[29164]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:32.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.160+0000 7fc70924c700 1 -- 172.21.15.186:0/3068681248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc704104070 msgr2=0x7fc704104450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:32.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.160+0000 7fc70924c700 1 --2- 172.21.15.186:0/3068681248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc704104070 0x7fc704104450 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009910 tx=0x7fc6f8009c20 comp rx=0 tx=0).stop 2024-04-26T17:46:32.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.160+0000 7fc70924c700 1 -- 172.21.15.186:0/3068681248 shutdown_connections 2024-04-26T17:46:32.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.160+0000 7fc70924c700 1 --2- 172.21.15.186:0/3068681248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc704104070 0x7fc704104450 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:32.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.160+0000 7fc70924c700 1 -- 172.21.15.186:0/3068681248 >> 172.21.15.186:0/3068681248 conn(0x7fc7040fd820 msgr2=0x7fc7040ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:32.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.160+0000 7fc70924c700 1 -- 172.21.15.186:0/3068681248 shutdown_connections 2024-04-26T17:46:32.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.160+0000 7fc70924c700 1 -- 172.21.15.186:0/3068681248 wait complete. 2024-04-26T17:46:32.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.161+0000 7fc70924c700 1 Processor -- start 2024-04-26T17:46:32.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.161+0000 7fc70924c700 1 -- start start 2024-04-26T17:46:32.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.161+0000 7fc70924c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc704104070 0x7fc704101cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:32.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.161+0000 7fc70924c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc70410d370 con 0x7fc704104070 2024-04-26T17:46:32.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.162+0000 7fc702d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc704104070 0x7fc704101cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:32.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.162+0000 7fc702d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc704104070 0x7fc704101cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.186:54970/0 (socket says 172.21.15.186:54970) 2024-04-26T17:46:32.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.162+0000 7fc702d9d700 1 -- 172.21.15.186:0/1152654140 learned_addr learned my addr 172.21.15.186:0/1152654140 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T17:46:32.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.162+0000 7fc702d9d700 1 -- 172.21.15.186:0/1152654140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc704103bc0 con 0x7fc704104070 2024-04-26T17:46:32.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.162+0000 7fc702d9d700 1 --2- 172.21.15.186:0/1152654140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc704104070 0x7fc704101cd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc6f80056d0 tx=0x7fc6f80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:32.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.162+0000 7fc6ebfff700 1 -- 172.21.15.186:0/1152654140 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6f8026070 con 0x7fc704104070 2024-04-26T17:46:32.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.162+0000 7fc6ebfff700 1 -- 172.21.15.186:0/1152654140 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6f8004370 con 0x7fc704104070 2024-04-26T17:46:32.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.162+0000 7fc70924c700 1 -- 172.21.15.186:0/1152654140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc704102210 con 0x7fc704104070 2024-04-26T17:46:32.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.162+0000 7fc6ebfff700 1 -- 172.21.15.186:0/1152654140 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6f80039b0 con 0x7fc704104070 2024-04-26T17:46:32.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.162+0000 7fc70924c700 1 -- 172.21.15.186:0/1152654140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7041026d0 con 0x7fc704104070 2024-04-26T17:46:32.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.163+0000 7fc70924c700 1 -- 172.21.15.186:0/1152654140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc70406b910 con 0x7fc704104070 2024-04-26T17:46:32.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.164+0000 7fc6ebfff700 1 -- 172.21.15.186:0/1152654140 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc6f8029070 con 0x7fc704104070 2024-04-26T17:46:32.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.164+0000 7fc6ebfff700 1 --2- 172.21.15.186:0/1152654140 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fc6ec039650 0x7fc6ec03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:32.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.164+0000 7fc6ebfff700 1 -- 172.21.15.186:0/1152654140 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc6f801e070 con 0x7fc704104070 2024-04-26T17:46:32.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.164+0000 7fc70259c700 1 --2- 172.21.15.186:0/1152654140 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fc6ec039650 0x7fc6ec03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:32.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.165+0000 7fc70259c700 1 --2- 172.21.15.186:0/1152654140 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fc6ec039650 0x7fc6ec03bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4006fd0 tx=0x7fc6f4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:32.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.170+0000 7fc6ebfff700 1 -- 172.21.15.186:0/1152654140 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc6f8013070 con 0x7fc704104070 2024-04-26T17:46:32.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 systemd[1]: Starting Ceph mon.c for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:46:32.477 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.476+0000 7fc70924c700 1 -- 172.21.15.186:0/1152654140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc7041032a0 con 0x7fc704104070 2024-04-26T17:46:32.477 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.477+0000 7fc6ebfff700 1 -- 172.21.15.186:0/1152654140 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fc6f8030090 con 0x7fc704104070 2024-04-26T17:46:32.477 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:46:32.477 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","modified":"2024-04-26T17:44:45.630672Z","created":"2024-04-26T17:44:45.630672Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-26T17:46:32.479 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.479+0000 7fc70924c700 1 -- 172.21.15.186:0/1152654140 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fc6ec039650 msgr2=0x7fc6ec03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:32.479 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.479+0000 7fc70924c700 1 --2- 172.21.15.186:0/1152654140 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fc6ec039650 0x7fc6ec03bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4006fd0 tx=0x7fc6f4006e40 comp rx=0 tx=0).stop 2024-04-26T17:46:32.479 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.479+0000 7fc70924c700 1 -- 172.21.15.186:0/1152654140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc704104070 msgr2=0x7fc704101cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:32.479 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.479+0000 7fc70924c700 1 --2- 172.21.15.186:0/1152654140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc704104070 0x7fc704101cd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc6f80056d0 tx=0x7fc6f80057b0 comp rx=0 tx=0).stop 2024-04-26T17:46:32.479 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.479+0000 7fc70924c700 1 -- 172.21.15.186:0/1152654140 shutdown_connections 2024-04-26T17:46:32.479 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.479+0000 7fc70924c700 1 --2- 172.21.15.186:0/1152654140 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fc6ec039650 0x7fc6ec03bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:32.479 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.479+0000 7fc70924c700 1 --2- 172.21.15.186:0/1152654140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc704104070 0x7fc704101cd0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:32.479 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.479+0000 7fc70924c700 1 -- 172.21.15.186:0/1152654140 >> 172.21.15.186:0/1152654140 conn(0x7fc7040fd820 msgr2=0x7fc704106e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:32.479 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.479+0000 7fc70924c700 1 -- 172.21.15.186:0/1152654140 shutdown_connections 2024-04-26T17:46:32.479 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:32.479+0000 7fc70924c700 1 -- 172.21.15.186:0/1152654140 wait complete. 2024-04-26T17:46:32.480 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-04-26T17:46:32.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:32 smithi031 ceph-mon[29164]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:32.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 podman[32765]: 2024-04-26 17:46:32.325322116 +0000 UTC m=+0.021978613 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:46:32.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 podman[32765]: 2024-04-26 17:46:32.425559596 +0000 UTC m=+0.122216080 container create 7c71567ce1a458c23c00f78464871d4818a88b7e82e63695d0d889b263593fbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T17:46:32.722 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 podman[32765]: 2024-04-26 17:46:32.723038925 +0000 UTC m=+0.419695408 container init 7c71567ce1a458c23c00f78464871d4818a88b7e82e63695d0d889b263593fbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-c, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T17:46:33.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 podman[32765]: 2024-04-26 17:46:32.726950163 +0000 UTC m=+0.423606647 container start 7c71567ce1a458c23c00f78464871d4818a88b7e82e63695d0d889b263593fbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-c, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-26T17:46:33.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T17:46:33.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev), process ceph-mon, pid 2 2024-04-26T17:46:33.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: pidfile_write: ignore empty --pid-file 2024-04-26T17:46:33.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: load: jerasure load: lrc 2024-04-26T17:46:33.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: RocksDB version: 7.9.2 2024-04-26T17:46:33.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Git sha 0 2024-04-26T17:46:33.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Compile date 2024-04-26 07:20:26 2024-04-26T17:46:33.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: DB SUMMARY 2024-04-26T17:46:33.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: DB Session ID: HR6GLMSI7LTDH4LGZVF3 2024-04-26T17:46:33.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: CURRENT file: CURRENT 2024-04-26T17:46:33.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T17:46:33.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-26T17:46:33.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-26T17:46:33.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-26T17:46:33.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.error_if_exists: 0 2024-04-26T17:46:33.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.create_if_missing: 0 2024-04-26T17:46:33.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.paranoid_checks: 1 2024-04-26T17:46:33.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T17:46:33.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T17:46:33.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T17:46:33.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.env: 0x559c9f0dbcc0 2024-04-26T17:46:33.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T17:46:33.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.info_log: 0x559ca030bd80 2024-04-26T17:46:33.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T17:46:33.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.statistics: (nil) 2024-04-26T17:46:33.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.use_fsync: 0 2024-04-26T17:46:33.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_log_file_size: 0 2024-04-26T17:46:33.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T17:46:33.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T17:46:33.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.allow_fallocate: 1 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.use_direct_reads: 0 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.db_log_dir: 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.wal_dir: 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T17:46:33.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T17:46:33.022 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T17:46:33.022 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T17:46:33.022 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.write_buffer_manager: 0x559ca02dd0e0 2024-04-26T17:46:33.022 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T17:46:33.022 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T17:46:33.022 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T17:46:33.022 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T17:46:33.022 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T17:46:33.022 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T17:46:33.022 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T17:46:33.022 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T17:46:33.022 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.unordered_write: 0 2024-04-26T17:46:33.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T17:46:33.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T17:46:33.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T17:46:33.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T17:46:33.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.row_cache: None 2024-04-26T17:46:33.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.wal_filter: None 2024-04-26T17:46:33.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T17:46:33.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T17:46:33.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.two_write_queues: 0 2024-04-26T17:46:33.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T17:46:33.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.wal_compression: 0 2024-04-26T17:46:33.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.atomic_flush: 0 2024-04-26T17:46:33.024 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T17:46:33.024 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T17:46:33.024 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T17:46:33.025 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.log_readahead_size: 0 2024-04-26T17:46:33.025 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T17:46:33.026 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T17:46:33.026 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T17:46:33.026 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T17:46:33.026 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T17:46:33.026 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T17:46:33.026 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T17:46:33.026 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_background_jobs: 2 2024-04-26T17:46:33.026 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_background_compactions: -1 2024-04-26T17:46:33.026 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_subcompactions: 1 2024-04-26T17:46:33.026 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T17:46:33.027 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T17:46:33.027 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T17:46:33.027 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T17:46:33.027 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T17:46:33.027 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T17:46:33.027 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T17:46:33.027 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T17:46:33.027 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_open_files: -1 2024-04-26T17:46:33.027 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T17:46:33.027 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T17:46:33.027 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T17:46:33.027 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T17:46:33.028 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_background_flushes: -1 2024-04-26T17:46:33.028 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Compression algorithms supported: 2024-04-26T17:46:33.028 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T17:46:33.028 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: kZSTD supported: 0 2024-04-26T17:46:33.028 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: kXpressCompression supported: 0 2024-04-26T17:46:33.028 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T17:46:33.028 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: kZlibCompression supported: 1 2024-04-26T17:46:33.028 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: kSnappyCompression supported: 1 2024-04-26T17:46:33.028 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: kLZ4Compression supported: 1 2024-04-26T17:46:33.028 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: kBZip2Compression supported: 0 2024-04-26T17:46:33.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T17:46:33.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T17:46:33.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-26T17:46:33.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T17:46:33.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T17:46:33.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.merge_operator: 2024-04-26T17:46:33.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compaction_filter: None 2024-04-26T17:46:33.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T17:46:33.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T17:46:33.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T17:46:33.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T17:46:33.030 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559ca030bf40) 2024-04-26T17:46:33.030 INFO:journalctl@ceph.mon.c.smithi186.stdout: cache_index_and_filter_blocks: 1 2024-04-26T17:46:33.030 INFO:journalctl@ceph.mon.c.smithi186.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T17:46:33.030 INFO:journalctl@ceph.mon.c.smithi186.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T17:46:33.030 INFO:journalctl@ceph.mon.c.smithi186.stdout: pin_top_level_index_and_filter: 1 2024-04-26T17:46:33.030 INFO:journalctl@ceph.mon.c.smithi186.stdout: index_type: 0 2024-04-26T17:46:33.030 INFO:journalctl@ceph.mon.c.smithi186.stdout: data_block_index_type: 0 2024-04-26T17:46:33.030 INFO:journalctl@ceph.mon.c.smithi186.stdout: index_shortening: 1 2024-04-26T17:46:33.030 INFO:journalctl@ceph.mon.c.smithi186.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T17:46:33.030 INFO:journalctl@ceph.mon.c.smithi186.stdout: checksum: 4 2024-04-26T17:46:33.031 INFO:journalctl@ceph.mon.c.smithi186.stdout: no_block_cache: 0 2024-04-26T17:46:33.031 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_cache: 0x559ca030f1f0 2024-04-26T17:46:33.031 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_cache_name: BinnedLRUCache 2024-04-26T17:46:33.031 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_cache_options: 2024-04-26T17:46:33.031 INFO:journalctl@ceph.mon.c.smithi186.stdout: capacity : 536870912 2024-04-26T17:46:33.031 INFO:journalctl@ceph.mon.c.smithi186.stdout: num_shard_bits : 4 2024-04-26T17:46:33.031 INFO:journalctl@ceph.mon.c.smithi186.stdout: strict_capacity_limit : 0 2024-04-26T17:46:33.031 INFO:journalctl@ceph.mon.c.smithi186.stdout: high_pri_pool_ratio: 0.000 2024-04-26T17:46:33.031 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_cache_compressed: (nil) 2024-04-26T17:46:33.031 INFO:journalctl@ceph.mon.c.smithi186.stdout: persistent_cache: (nil) 2024-04-26T17:46:33.031 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_size: 4096 2024-04-26T17:46:33.031 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_size_deviation: 10 2024-04-26T17:46:33.031 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_restart_interval: 16 2024-04-26T17:46:33.032 INFO:journalctl@ceph.mon.c.smithi186.stdout: index_block_restart_interval: 1 2024-04-26T17:46:33.032 INFO:journalctl@ceph.mon.c.smithi186.stdout: metadata_block_size: 4096 2024-04-26T17:46:33.032 INFO:journalctl@ceph.mon.c.smithi186.stdout: partition_filters: 0 2024-04-26T17:46:33.032 INFO:journalctl@ceph.mon.c.smithi186.stdout: use_delta_encoding: 1 2024-04-26T17:46:33.032 INFO:journalctl@ceph.mon.c.smithi186.stdout: filter_policy: bloomfilter 2024-04-26T17:46:33.032 INFO:journalctl@ceph.mon.c.smithi186.stdout: whole_key_filtering: 1 2024-04-26T17:46:33.032 INFO:journalctl@ceph.mon.c.smithi186.stdout: verify_compression: 0 2024-04-26T17:46:33.032 INFO:journalctl@ceph.mon.c.smithi186.stdout: read_amp_bytes_per_bit: 0 2024-04-26T17:46:33.032 INFO:journalctl@ceph.mon.c.smithi186.stdout: format_version: 5 2024-04-26T17:46:33.032 INFO:journalctl@ceph.mon.c.smithi186.stdout: enable_index_compression: 1 2024-04-26T17:46:33.032 INFO:journalctl@ceph.mon.c.smithi186.stdout: block_align: 0 2024-04-26T17:46:33.032 INFO:journalctl@ceph.mon.c.smithi186.stdout: max_auto_readahead_size: 262144 2024-04-26T17:46:33.033 INFO:journalctl@ceph.mon.c.smithi186.stdout: prepopulate_block_cache: 0 2024-04-26T17:46:33.033 INFO:journalctl@ceph.mon.c.smithi186.stdout: initial_auto_readahead_size: 8192 2024-04-26T17:46:33.033 INFO:journalctl@ceph.mon.c.smithi186.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T17:46:33.033 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T17:46:33.033 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T17:46:33.033 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compression: NoCompression 2024-04-26T17:46:33.033 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T17:46:33.033 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T17:46:33.033 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T17:46:33.033 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.num_levels: 7 2024-04-26T17:46:33.033 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T17:46:33.033 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T17:46:33.033 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T17:46:33.033 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T17:46:33.034 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T17:46:33.034 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T17:46:33.034 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T17:46:33.034 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T17:46:33.034 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T17:46:33.034 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T17:46:33.034 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T17:46:33.034 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T17:46:33.034 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T17:46:33.034 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T17:46:33.034 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T17:46:33.035 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T17:46:33.035 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T17:46:33.035 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T17:46:33.035 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T17:46:33.035 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T17:46:33.035 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T17:46:33.035 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T17:46:33.035 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T17:46:33.035 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T17:46:33.035 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T17:46:33.035 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T17:46:33.036 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T17:46:33.036 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T17:46:33.036 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T17:46:33.036 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T17:46:33.036 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T17:46:33.036 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T17:46:33.036 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T17:46:33.036 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T17:46:33.036 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T17:46:33.036 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T17:46:33.036 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T17:46:33.036 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T17:46:33.036 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T17:46:33.036 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T17:46:33.037 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T17:46:33.037 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T17:46:33.037 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T17:46:33.037 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T17:46:33.037 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T17:46:33.037 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T17:46:33.037 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T17:46:33.037 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T17:46:33.037 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T17:46:33.037 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T17:46:33.037 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T17:46:33.037 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T17:46:33.038 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T17:46:33.038 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T17:46:33.038 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.inplace_update_support: 0 2024-04-26T17:46:33.038 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T17:46:33.038 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T17:46:33.038 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T17:46:33.038 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T17:46:33.038 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.bloom_locality: 0 2024-04-26T17:46:33.038 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.max_successive_merges: 0 2024-04-26T17:46:33.038 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T17:46:33.038 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T17:46:33.038 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T17:46:33.038 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T17:46:33.039 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.ttl: 2592000 2024-04-26T17:46:33.039 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T17:46:33.039 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T17:46:33.039 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T17:46:33.039 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.enable_blob_files: false 2024-04-26T17:46:33.039 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.min_blob_size: 0 2024-04-26T17:46:33.039 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T17:46:33.039 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T17:46:33.039 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T17:46:33.039 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T17:46:33.039 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T17:46:33.039 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T17:46:33.039 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T17:46:33.040 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T17:46:33.040 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-26T17:46:33.040 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-26T17:46:33.040 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ffdb4a4a-375c-48d5-a67a-17db2fa3f65b 2024-04-26T17:46:33.040 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714153592756837, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-26T17:46:33.040 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-26T17:46:33.040 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714153592757324, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714153592, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ffdb4a4a-375c-48d5-a67a-17db2fa3f65b", "db_session_id": "HR6GLMSI7LTDH4LGZVF3", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-26T17:46:33.040 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714153592757410, "job": 1, "event": "recovery_finished"} 2024-04-26T17:46:33.040 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-26T17:46:33.040 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T17:46:33.040 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559ca042c000 2024-04-26T17:46:33.041 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: DB pointer 0x559ca0418000 2024-04-26T17:46:33.041 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T17:46:33.041 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T17:46:33.041 INFO:journalctl@ceph.mon.c.smithi186.stdout: ** DB Stats ** 2024-04-26T17:46:33.041 INFO:journalctl@ceph.mon.c.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T17:46:33.041 INFO:journalctl@ceph.mon.c.smithi186.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T17:46:33.041 INFO:journalctl@ceph.mon.c.smithi186.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T17:46:33.041 INFO:journalctl@ceph.mon.c.smithi186.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T17:46:33.041 INFO:journalctl@ceph.mon.c.smithi186.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T17:46:33.041 INFO:journalctl@ceph.mon.c.smithi186.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T17:46:33.041 INFO:journalctl@ceph.mon.c.smithi186.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T17:46:33.041 INFO:journalctl@ceph.mon.c.smithi186.stdout: 2024-04-26T17:46:33.041 INFO:journalctl@ceph.mon.c.smithi186.stdout: ** Compaction Stats [default] ** 2024-04-26T17:46:33.042 INFO:journalctl@ceph.mon.c.smithi186.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T17:46:33.042 INFO:journalctl@ceph.mon.c.smithi186.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T17:46:33.042 INFO:journalctl@ceph.mon.c.smithi186.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T17:46:33.042 INFO:journalctl@ceph.mon.c.smithi186.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T17:46:33.042 INFO:journalctl@ceph.mon.c.smithi186.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T17:46:33.042 INFO:journalctl@ceph.mon.c.smithi186.stdout: 2024-04-26T17:46:33.042 INFO:journalctl@ceph.mon.c.smithi186.stdout: ** Compaction Stats [default] ** 2024-04-26T17:46:33.042 INFO:journalctl@ceph.mon.c.smithi186.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T17:46:33.042 INFO:journalctl@ceph.mon.c.smithi186.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T17:46:33.042 INFO:journalctl@ceph.mon.c.smithi186.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T17:46:33.042 INFO:journalctl@ceph.mon.c.smithi186.stdout: 2024-04-26T17:46:33.042 INFO:journalctl@ceph.mon.c.smithi186.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T17:46:33.043 INFO:journalctl@ceph.mon.c.smithi186.stdout: 2024-04-26T17:46:33.043 INFO:journalctl@ceph.mon.c.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T17:46:33.043 INFO:journalctl@ceph.mon.c.smithi186.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T17:46:33.043 INFO:journalctl@ceph.mon.c.smithi186.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T17:46:33.043 INFO:journalctl@ceph.mon.c.smithi186.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T17:46:33.043 INFO:journalctl@ceph.mon.c.smithi186.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T17:46:33.043 INFO:journalctl@ceph.mon.c.smithi186.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T17:46:33.043 INFO:journalctl@ceph.mon.c.smithi186.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T17:46:33.043 INFO:journalctl@ceph.mon.c.smithi186.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T17:46:33.043 INFO:journalctl@ceph.mon.c.smithi186.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T17:46:33.043 INFO:journalctl@ceph.mon.c.smithi186.stdout: Block cache BinnedLRUCache@0x559ca030f1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-26T17:46:33.043 INFO:journalctl@ceph.mon.c.smithi186.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-26T17:46:33.044 INFO:journalctl@ceph.mon.c.smithi186.stdout: 2024-04-26T17:46:33.044 INFO:journalctl@ceph.mon.c.smithi186.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T17:46:33.044 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-26T17:46:33.044 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: using public_addr v2:172.21.15.186:0/0 -> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T17:46:33.044 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: starting mon.c rank -1 at public addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] at bind addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:46:33.044 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c@-1(???) e0 preinit fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:46:33.044 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c@-1(synchronizing).mds e1 new map 2024-04-26T17:46:33.044 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-26T17:46:33.044 INFO:journalctl@ceph.mon.c.smithi186.stdout: e1 2024-04-26T17:46:33.044 INFO:journalctl@ceph.mon.c.smithi186.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T17:46:33.044 INFO:journalctl@ceph.mon.c.smithi186.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T17:46:33.044 INFO:journalctl@ceph.mon.c.smithi186.stdout: legacy client fscid: -1 2024-04-26T17:46:33.045 INFO:journalctl@ceph.mon.c.smithi186.stdout: 2024-04-26T17:46:33.045 INFO:journalctl@ceph.mon.c.smithi186.stdout: No filesystems configured 2024-04-26T17:46:33.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-26T17:46:33.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-26T17:46:33.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-26T17:46:33.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-26T17:46:33.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-26T17:46:33.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-26T17:46:33.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T17:46:33.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T17:46:33.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T17:46:33.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T17:46:33.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:33.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:46:33.046 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.046 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.046 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.046 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.046 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi097", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:46:33.046 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:33.046 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:33.046 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-26T17:46:33.046 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:33.046 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: Deploying cephadm binary to smithi186 2024-04-26T17:46:33.046 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:46:33.046 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:46:33.047 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:33.047 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.047 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.047 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.047 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:33.047 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.047 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: Added host smithi186 2024-04-26T17:46:33.047 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:33.047 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.047 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:33.047 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2896908694' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2896908694' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:46:33.048 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:33.049 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:33.049 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.049 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.049 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.049 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T17:46:33.049 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:33.049 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:46:33.049 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:46:33.049 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:33.049 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.049 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:33.049 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:33.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:33.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:33.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:33.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:33.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi097:172.21.15.97=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:46:33.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi097:172.21.15.97=b;smithi186:172.21.15.186=c;count:3 2024-04-26T17:46:33.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: Deploying daemon mon.c on smithi186 2024-04-26T17:46:33.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:33.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:33.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 ceph-mon[32797]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T17:46:33.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 bash[32765]: 7c71567ce1a458c23c00f78464871d4818a88b7e82e63695d0d889b263593fbb 2024-04-26T17:46:33.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:32 smithi186 systemd[1]: Started Ceph mon.c for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:46:34.278 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T17:46:34.278 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T17:46:34.525 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T17:46:38.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:38.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:46:38.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: mon.a calling monitor election 2024-04-26T17:46:38.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:38.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:38.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:38.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:38.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: mon.c calling monitor election 2024-04-26T17:46:38.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:38.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:38.253 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:38.253 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T17:46:38.253 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: monmap epoch 2 2024-04-26T17:46:38.253 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:46:38.253 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: last_changed 2024-04-26T17:46:32.781577+0000 2024-04-26T17:46:38.253 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: created 2024-04-26T17:44:45.630672+0000 2024-04-26T17:46:38.253 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: min_mon_release 19 (squid) 2024-04-26T17:46:38.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: election_strategy: 1 2024-04-26T17:46:38.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-26T17:46:38.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T17:46:38.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: fsmap 2024-04-26T17:46:38.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T17:46:38.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: mgrmap e13: x(active, since 49s) 2024-04-26T17:46:38.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: overall HEALTH_OK 2024-04-26T17:46:38.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:38.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:38.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:38.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:38.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:38.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:38.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:38.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:46:38.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: mon.a calling monitor election 2024-04-26T17:46:38.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:38.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:38.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:38.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:38.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: mon.c calling monitor election 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: monmap epoch 2 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: last_changed 2024-04-26T17:46:32.781577+0000 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: created 2024-04-26T17:44:45.630672+0000 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: min_mon_release 19 (squid) 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: election_strategy: 1 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: fsmap 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T17:46:38.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: mgrmap e13: x(active, since 49s) 2024-04-26T17:46:38.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: overall HEALTH_OK 2024-04-26T17:46:38.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:38.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:38.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:38.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:38.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:38.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:38.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.987+0000 7fd532a1e700 1 -- 172.21.15.186:0/690507201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5140035c0 msgr2=0x7fd514005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:38.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.987+0000 7fd532a1e700 1 --2- 172.21.15.186:0/690507201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5140035c0 0x7fd514005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd52c06b710 tx=0x7fd51c00feb0 comp rx=0 tx=0).stop 2024-04-26T17:46:38.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.987+0000 7fd532a1e700 1 -- 172.21.15.186:0/690507201 shutdown_connections 2024-04-26T17:46:38.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.987+0000 7fd532a1e700 1 --2- 172.21.15.186:0/690507201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5140035c0 0x7fd514005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:38.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.987+0000 7fd532a1e700 1 -- 172.21.15.186:0/690507201 >> 172.21.15.186:0/690507201 conn(0x7fd52c0fd960 msgr2=0x7fd52c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:38.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.988+0000 7fd532a1e700 1 -- 172.21.15.186:0/690507201 shutdown_connections 2024-04-26T17:46:38.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.988+0000 7fd532a1e700 1 -- 172.21.15.186:0/690507201 wait complete. 2024-04-26T17:46:38.989 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.989+0000 7fd532a1e700 1 Processor -- start 2024-04-26T17:46:38.989 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.989+0000 7fd532a1e700 1 -- start start 2024-04-26T17:46:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.989+0000 7fd532a1e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5140035c0 0x7fd52c1a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.989+0000 7fd532a1e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd52c1a0d70 0x7fd52c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.989+0000 7fd532a1e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c10d070 con 0x7fd52c1a0d70 2024-04-26T17:46:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.989+0000 7fd532a1e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c10cef0 con 0x7fd5140035c0 2024-04-26T17:46:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd52bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5140035c0 0x7fd52c1a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd52b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd52c1a0d70 0x7fd52c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd52b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd52c1a0d70 0x7fd52c19a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.186:33672/0 (socket says 172.21.15.186:33672) 2024-04-26T17:46:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd52bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5140035c0 0x7fd52c1a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:35418/0 (socket says 172.21.15.186:35418) 2024-04-26T17:46:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd52b7fe700 1 -- 172.21.15.186:0/3245107766 learned_addr learned my addr 172.21.15.186:0/3245107766 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T17:46:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd52bfff700 1 -- 172.21.15.186:0/3245107766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5140035c0 msgr2=0x7fd52c1a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T17:46:38.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd52bfff700 1 -- 172.21.15.186:0/3245107766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5140035c0 msgr2=0x7fd52c1a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T17:46:38.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd52bfff700 1 --2- 172.21.15.186:0/3245107766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5140035c0 0x7fd52c1a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T17:46:38.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd52bfff700 1 --2- 172.21.15.186:0/3245107766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5140035c0 0x7fd52c1a0830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T17:46:38.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd52b7fe700 1 -- 172.21.15.186:0/3245107766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5140035c0 msgr2=0x7fd52c1a0830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T17:46:38.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd52b7fe700 1 --2- 172.21.15.186:0/3245107766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5140035c0 0x7fd52c1a0830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:38.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd52b7fe700 1 -- 172.21.15.186:0/3245107766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52c1a13b0 con 0x7fd52c1a0d70 2024-04-26T17:46:38.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd52b7fe700 1 --2- 172.21.15.186:0/3245107766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd52c1a0d70 0x7fd52c19a9f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd52000b700 tx=0x7fd52000bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:38.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd5297fa700 1 -- 172.21.15.186:0/3245107766 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fd520011630 con 0x7fd52c1a0d70 2024-04-26T17:46:38.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd5297fa700 1 -- 172.21.15.186:0/3245107766 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd520011c80 con 0x7fd52c1a0d70 2024-04-26T17:46:38.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd532a1e700 1 -- 172.21.15.186:0/3245107766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd52c19af30 con 0x7fd52c1a0d70 2024-04-26T17:46:38.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd5297fa700 1 -- 172.21.15.186:0/3245107766 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fd520010da0 con 0x7fd52c1a0d70 2024-04-26T17:46:38.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.990+0000 7fd532a1e700 1 -- 172.21.15.186:0/3245107766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd52c19b4c0 con 0x7fd52c1a0d70 2024-04-26T17:46:38.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.991+0000 7fd5297fa700 1 -- 172.21.15.186:0/3245107766 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fd520018020 con 0x7fd52c1a0d70 2024-04-26T17:46:38.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.992+0000 7fd5297fa700 1 --2- 172.21.15.186:0/3245107766 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd518039440 0x7fd51803b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:38.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.992+0000 7fd532a1e700 1 -- 172.21.15.186:0/3245107766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd52c06bf10 con 0x7fd52c1a0d70 2024-04-26T17:46:38.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.992+0000 7fd5297fa700 1 -- 172.21.15.186:0/3245107766 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd52004d240 con 0x7fd52c1a0d70 2024-04-26T17:46:38.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.992+0000 7fd52bfff700 1 --2- 172.21.15.186:0/3245107766 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd518039440 0x7fd51803b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:38.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.992+0000 7fd52bfff700 1 --2- 172.21.15.186:0/3245107766 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd518039440 0x7fd51803b900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00f8d0 tx=0x7fd51c005370 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:38.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:38.997+0000 7fd5297fa700 1 -- 172.21.15.186:0/3245107766 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd52001f020 con 0x7fd52c1a0d70 2024-04-26T17:46:39.214 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:38 smithi031 ceph-mon[29164]: Deploying daemon mon.b on smithi097 2024-04-26T17:46:39.215 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:38 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:39.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:38 smithi186 ceph-mon[32797]: Deploying daemon mon.b on smithi097 2024-04-26T17:46:39.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:38 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:39.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:39.284+0000 7fd532a1e700 1 -- 172.21.15.186:0/3245107766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fd52c19bf20 con 0x7fd52c1a0d70 2024-04-26T17:46:39.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:39.285+0000 7fd5297fa700 1 -- 172.21.15.186:0/3245107766 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fd520018570 con 0x7fd52c1a0d70 2024-04-26T17:46:39.285 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:46:39.285 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":2,"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","modified":"2024-04-26T17:46:32.781577Z","created":"2024-04-26T17:44:45.630672Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T17:46:39.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:39.287+0000 7fd532a1e700 1 -- 172.21.15.186:0/3245107766 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd518039440 msgr2=0x7fd51803b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:39.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:39.287+0000 7fd532a1e700 1 --2- 172.21.15.186:0/3245107766 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd518039440 0x7fd51803b900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00f8d0 tx=0x7fd51c005370 comp rx=0 tx=0).stop 2024-04-26T17:46:39.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:39.287+0000 7fd532a1e700 1 -- 172.21.15.186:0/3245107766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd52c1a0d70 msgr2=0x7fd52c19a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:39.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:39.287+0000 7fd532a1e700 1 --2- 172.21.15.186:0/3245107766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd52c1a0d70 0x7fd52c19a9f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd52000b700 tx=0x7fd52000bac0 comp rx=0 tx=0).stop 2024-04-26T17:46:39.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:39.287+0000 7fd532a1e700 1 -- 172.21.15.186:0/3245107766 shutdown_connections 2024-04-26T17:46:39.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:39.287+0000 7fd532a1e700 1 --2- 172.21.15.186:0/3245107766 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd518039440 0x7fd51803b900 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:39.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:39.287+0000 7fd532a1e700 1 --2- 172.21.15.186:0/3245107766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5140035c0 0x7fd52c1a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:39.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:39.287+0000 7fd532a1e700 1 --2- 172.21.15.186:0/3245107766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd52c1a0d70 0x7fd52c19a9f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:39.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:39.287+0000 7fd532a1e700 1 -- 172.21.15.186:0/3245107766 >> 172.21.15.186:0/3245107766 conn(0x7fd52c0fd960 msgr2=0x7fd52c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:39.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:39.287+0000 7fd532a1e700 1 -- 172.21.15.186:0/3245107766 shutdown_connections 2024-04-26T17:46:39.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:39.287+0000 7fd532a1e700 1 -- 172.21.15.186:0/3245107766 wait complete. 2024-04-26T17:46:39.288 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 2 2024-04-26T17:46:40.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:39 smithi186 ceph-mon[32797]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:40.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:39 smithi186 ceph-mon[32797]: from='client.? 172.21.15.186:0/3245107766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T17:46:40.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:39 smithi031 ceph-mon[29164]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:40.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:39 smithi031 ceph-mon[29164]: from='client.? 172.21.15.186:0/3245107766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T17:46:40.914 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T17:46:40.915 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T17:46:41.162 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T17:46:41.647 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-26T17:46:41.647 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-26T17:46:41.647 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-26T17:46:41.648 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-26T17:46:41.648 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-26T17:46:41.648 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-26T17:46:42.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T17:46:42.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T17:46:42.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T17:46:42.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi097", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: Deploying cephadm binary to smithi186 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:46:42.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:42.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:42.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: Added host smithi186 2024-04-26T17:46:42.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:42.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:42.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:46:42.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:42.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2896908694' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2896908694' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T17:46:42.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:42.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:46:42.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:46:42.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:42.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:42.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:42.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:42.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:42.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:42.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi097:172.21.15.97=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:46:42.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi097:172.21.15.97=b;smithi186:172.21.15.186=c;count:3 2024-04-26T17:46:42.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: Deploying daemon mon.c on smithi186 2024-04-26T17:46:42.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:42.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:42.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:42.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:46:42.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mon.a calling monitor election 2024-04-26T17:46:42.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:42.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:42.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:42.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:42.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mon.c calling monitor election 2024-04-26T17:46:42.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: monmap epoch 2 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: last_changed 2024-04-26T17:46:32.781577+0000 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: created 2024-04-26T17:44:45.630672+0000 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: min_mon_release 19 (squid) 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: election_strategy: 1 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: fsmap 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mgrmap e13: x(active, since 49s) 2024-04-26T17:46:42.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: overall HEALTH_OK 2024-04-26T17:46:42.141 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:42.141 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.141 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.141 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:42.141 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:42.141 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:42.141 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: Deploying daemon mon.b on smithi097 2024-04-26T17:46:42.141 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:42.141 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:42.141 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: from='client.? 172.21.15.186:0/3245107766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T17:46:42.141 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 ceph-mon[32419]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T17:46:42.141 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 bash[32400]: 81484a0a13bd3f3e8d890687339145d0166b978843c2dcdf87904bc30ea60ece 2024-04-26T17:46:42.141 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:41 smithi097 systemd[1]: Started Ceph mon.b for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:46:47.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:46:47.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: mon.a calling monitor election 2024-04-26T17:46:47.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: mon.c calling monitor election 2024-04-26T17:46:47.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:47.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:47.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:47.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:47.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:47.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:47.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:47.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:47.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:47.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T17:46:47.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: monmap epoch 3 2024-04-26T17:46:47.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:46:47.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: last_changed 2024-04-26T17:46:41.654574+0000 2024-04-26T17:46:47.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: created 2024-04-26T17:44:45.630672+0000 2024-04-26T17:46:47.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: min_mon_release 19 (squid) 2024-04-26T17:46:47.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: election_strategy: 1 2024-04-26T17:46:47.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-26T17:46:47.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T17:46:47.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: 2: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.b 2024-04-26T17:46:47.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: fsmap 2024-04-26T17:46:47.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T17:46:47.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: mgrmap e13: x(active, since 58s) 2024-04-26T17:46:47.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: overall HEALTH_OK 2024-04-26T17:46:47.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:47.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:47.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:47.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:47.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.033+0000 7feb3e0ed700 1 -- 172.21.15.186:0/731274337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb240057d0 msgr2=0x7feb24007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:47.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.033+0000 7feb3e0ed700 1 --2- 172.21.15.186:0/731274337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb240057d0 0x7feb24007c60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7feb38101d60 tx=0x7feb2800fe60 comp rx=0 tx=0).stop 2024-04-26T17:46:47.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.033+0000 7feb3e0ed700 1 -- 172.21.15.186:0/731274337 shutdown_connections 2024-04-26T17:46:47.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.033+0000 7feb3e0ed700 1 --2- 172.21.15.186:0/731274337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb240057d0 0x7feb24007c60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:47.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.033+0000 7feb3e0ed700 1 -- 172.21.15.186:0/731274337 >> 172.21.15.186:0/731274337 conn(0x7feb380fd920 msgr2=0x7feb380ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:47.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.033+0000 7feb3e0ed700 1 -- 172.21.15.186:0/731274337 shutdown_connections 2024-04-26T17:46:47.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.033+0000 7feb3e0ed700 1 -- 172.21.15.186:0/731274337 wait complete. 2024-04-26T17:46:47.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.033+0000 7feb3e0ed700 1 Processor -- start 2024-04-26T17:46:47.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.033+0000 7feb3e0ed700 1 -- start start 2024-04-26T17:46:47.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb3e0ed700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb240057d0 0x7feb381a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:47.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb3e0ed700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb381a0e80 0x7feb3819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:47.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb3e0ed700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb3819b040 0x7feb3819b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:47.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb3e0ed700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feb38108d10 con 0x7feb3819b040 2024-04-26T17:46:47.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb3e0ed700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7feb3806c4c0 con 0x7feb240057d0 2024-04-26T17:46:47.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb3e0ed700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feb38108e90 con 0x7feb381a0e80 2024-04-26T17:46:47.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb2ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb381a0e80 0x7feb3819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:47.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb2ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb381a0e80 0x7feb3819ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:37574/0 (socket says 172.21.15.186:37574) 2024-04-26T17:46:47.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb2ffff700 1 -- 172.21.15.186:0/1000278755 learned_addr learned my addr 172.21.15.186:0/1000278755 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T17:46:47.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb37fff700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb3819b040 0x7feb3819b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:47.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb377fe700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb240057d0 0x7feb381a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:47.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb2ffff700 1 -- 172.21.15.186:0/1000278755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb381a0e80 msgr2=0x7feb3819ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T17:46:47.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb2ffff700 1 -- 172.21.15.186:0/1000278755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb381a0e80 msgr2=0x7feb3819ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T17:46:47.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb2ffff700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb381a0e80 0x7feb3819ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T17:46:47.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb2ffff700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb381a0e80 0x7feb3819ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T17:46:47.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb377fe700 1 -- 172.21.15.186:0/1000278755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb240057d0 msgr2=0x7feb381a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T17:46:47.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb377fe700 1 -- 172.21.15.186:0/1000278755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb240057d0 msgr2=0x7feb381a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T17:46:47.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb377fe700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb240057d0 0x7feb381a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T17:46:47.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb377fe700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb240057d0 0x7feb381a0940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T17:46:47.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb37fff700 1 -- 172.21.15.186:0/1000278755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb381a0e80 msgr2=0x7feb3819ab00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T17:46:47.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb37fff700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb381a0e80 0x7feb3819ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:47.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb37fff700 1 -- 172.21.15.186:0/1000278755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb240057d0 msgr2=0x7feb381a0940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T17:46:47.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb37fff700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb240057d0 0x7feb381a0940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:47.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb37fff700 1 -- 172.21.15.186:0/1000278755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb381a7810 con 0x7feb3819b040 2024-04-26T17:46:47.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.034+0000 7feb37fff700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb3819b040 0x7feb3819b4f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7feb3000bd50 tx=0x7feb3000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:47.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.035+0000 7feb357fa700 1 -- 172.21.15.186:0/1000278755 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30019070 con 0x7feb3819b040 2024-04-26T17:46:47.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.035+0000 7feb357fa700 1 -- 172.21.15.186:0/1000278755 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb3000cd10 con 0x7feb3819b040 2024-04-26T17:46:47.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.035+0000 7feb357fa700 1 -- 172.21.15.186:0/1000278755 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30012700 con 0x7feb3819b040 2024-04-26T17:46:47.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.035+0000 7feb3e0ed700 1 -- 172.21.15.186:0/1000278755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb381a7a40 con 0x7feb3819b040 2024-04-26T17:46:47.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.035+0000 7feb3e0ed700 1 -- 172.21.15.186:0/1000278755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb381a7f30 con 0x7feb3819b040 2024-04-26T17:46:47.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.036+0000 7feb357fa700 1 -- 172.21.15.186:0/1000278755 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7feb3001c070 con 0x7feb3819b040 2024-04-26T17:46:47.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.036+0000 7feb3e0ed700 1 -- 172.21.15.186:0/1000278755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb38064a80 con 0x7feb3819b040 2024-04-26T17:46:47.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.036+0000 7feb357fa700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb18039ae0 0x7feb1803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:47.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.036+0000 7feb357fa700 1 -- 172.21.15.186:0/1000278755 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7feb3004c1f0 con 0x7feb3819b040 2024-04-26T17:46:47.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.036+0000 7feb377fe700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb18039ae0 0x7feb1803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:47.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.037+0000 7feb377fe700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb18039ae0 0x7feb1803bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7feb2800fa10 tx=0x7feb28005ad0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:47.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.042+0000 7feb357fa700 1 -- 172.21.15.186:0/1000278755 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feb30014a10 con 0x7feb3819b040 2024-04-26T17:46:47.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:46:47.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: mon.a calling monitor election 2024-04-26T17:46:47.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: mon.c calling monitor election 2024-04-26T17:46:47.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:47.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:47.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:47.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:47.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:47.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:47.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:47.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:47.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:47.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T17:46:47.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: monmap epoch 3 2024-04-26T17:46:47.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:46:47.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: last_changed 2024-04-26T17:46:41.654574+0000 2024-04-26T17:46:47.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: created 2024-04-26T17:44:45.630672+0000 2024-04-26T17:46:47.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: min_mon_release 19 (squid) 2024-04-26T17:46:47.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: election_strategy: 1 2024-04-26T17:46:47.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-26T17:46:47.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T17:46:47.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: 2: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.b 2024-04-26T17:46:47.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: fsmap 2024-04-26T17:46:47.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T17:46:47.062 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: mgrmap e13: x(active, since 58s) 2024-04-26T17:46:47.062 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: overall HEALTH_OK 2024-04-26T17:46:47.062 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:47.062 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:47.062 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:47.062 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:47.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.328+0000 7feb3e0ed700 1 -- 172.21.15.186:0/1000278755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7feb3802e650 con 0x7feb3819b040 2024-04-26T17:46:47.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.328+0000 7feb357fa700 1 -- 172.21.15.186:0/1000278755 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7feb30012910 con 0x7feb3819b040 2024-04-26T17:46:47.329 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T17:46:47.329 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":3,"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","modified":"2024-04-26T17:46:41.654574Z","created":"2024-04-26T17:44:45.630672Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T17:46:47.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.331+0000 7feb3e0ed700 1 -- 172.21.15.186:0/1000278755 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb18039ae0 msgr2=0x7feb1803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:47.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.331+0000 7feb3e0ed700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb18039ae0 0x7feb1803bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7feb2800fa10 tx=0x7feb28005ad0 comp rx=0 tx=0).stop 2024-04-26T17:46:47.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.331+0000 7feb3e0ed700 1 -- 172.21.15.186:0/1000278755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb3819b040 msgr2=0x7feb3819b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:47.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.331+0000 7feb3e0ed700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb3819b040 0x7feb3819b4f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7feb3000bd50 tx=0x7feb3000be30 comp rx=0 tx=0).stop 2024-04-26T17:46:47.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.331+0000 7feb3e0ed700 1 -- 172.21.15.186:0/1000278755 shutdown_connections 2024-04-26T17:46:47.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.331+0000 7feb3e0ed700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb18039ae0 0x7feb1803bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:47.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.331+0000 7feb3e0ed700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb240057d0 0x7feb381a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:47.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.331+0000 7feb3e0ed700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb3819b040 0x7feb3819b4f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:47.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.331+0000 7feb3e0ed700 1 --2- 172.21.15.186:0/1000278755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb381a0e80 0x7feb3819ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:47.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.331+0000 7feb3e0ed700 1 -- 172.21.15.186:0/1000278755 >> 172.21.15.186:0/1000278755 conn(0x7feb380fd920 msgr2=0x7feb380feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:47.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.331+0000 7feb3e0ed700 1 -- 172.21.15.186:0/1000278755 shutdown_connections 2024-04-26T17:46:47.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:47.331+0000 7feb3e0ed700 1 -- 172.21.15.186:0/1000278755 wait complete. 2024-04-26T17:46:47.332 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 3 2024-04-26T17:46:47.886 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:47 smithi186 ceph-mon[32797]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:47.887 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:47 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:47.887 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:47 smithi186 ceph-mon[32797]: from='client.? 172.21.15.186:0/1000278755' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T17:46:47.887 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:47 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:47.913 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-26T17:46:47.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-26T17:46:48.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:47 smithi031 ceph-mon[29164]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:48.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:47 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:48.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:47 smithi031 ceph-mon[29164]: from='client.? 172.21.15.186:0/1000278755' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T17:46:48.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:47 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:48.169 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:46:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:46:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: mon.a calling monitor election 2024-04-26T17:46:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: mon.c calling monitor election 2024-04-26T17:46:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:46:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:49.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:49.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:49.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:49.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:49.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:49.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:49.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T17:46:49.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: monmap epoch 3 2024-04-26T17:46:49.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:46:49.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: last_changed 2024-04-26T17:46:41.654574+0000 2024-04-26T17:46:49.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: created 2024-04-26T17:44:45.630672+0000 2024-04-26T17:46:49.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: min_mon_release 19 (squid) 2024-04-26T17:46:49.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: election_strategy: 1 2024-04-26T17:46:49.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-26T17:46:49.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T17:46:49.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: 2: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.b 2024-04-26T17:46:49.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: fsmap 2024-04-26T17:46:49.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T17:46:49.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: mgrmap e13: x(active, since 58s) 2024-04-26T17:46:49.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: overall HEALTH_OK 2024-04-26T17:46:49.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:49.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:49.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='client.? 172.21.15.186:0/1000278755' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T17:46:49.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:49.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: mon.b calling monitor election 2024-04-26T17:46:49.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:49.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: mon.b calling monitor election 2024-04-26T17:46:49.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: mon.c calling monitor election 2024-04-26T17:46:49.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: mon.a calling monitor election 2024-04-26T17:46:49.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T17:46:49.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: monmap epoch 3 2024-04-26T17:46:49.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:46:49.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: last_changed 2024-04-26T17:46:41.654574+0000 2024-04-26T17:46:49.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: created 2024-04-26T17:44:45.630672+0000 2024-04-26T17:46:49.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: min_mon_release 19 (squid) 2024-04-26T17:46:49.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: election_strategy: 1 2024-04-26T17:46:49.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-26T17:46:49.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T17:46:49.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: 2: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.b 2024-04-26T17:46:49.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: fsmap 2024-04-26T17:46:49.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T17:46:49.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: mgrmap e13: x(active, since 60s) 2024-04-26T17:46:49.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: overall HEALTH_OK 2024-04-26T17:46:49.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:49.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:49.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-26T17:46:49.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-26T17:46:49.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T17:46:49.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:49.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:49.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: Updating smithi031:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:49.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: mon.b calling monitor election 2024-04-26T17:46:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: mon.b calling monitor election 2024-04-26T17:46:49.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: mon.c calling monitor election 2024-04-26T17:46:49.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: mon.a calling monitor election 2024-04-26T17:46:49.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T17:46:49.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: monmap epoch 3 2024-04-26T17:46:49.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:46:49.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: last_changed 2024-04-26T17:46:41.654574+0000 2024-04-26T17:46:49.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: created 2024-04-26T17:44:45.630672+0000 2024-04-26T17:46:49.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: min_mon_release 19 (squid) 2024-04-26T17:46:49.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: election_strategy: 1 2024-04-26T17:46:49.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-26T17:46:49.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T17:46:49.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: 2: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.b 2024-04-26T17:46:49.636 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: fsmap 2024-04-26T17:46:49.636 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T17:46:49.636 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: mgrmap e13: x(active, since 60s) 2024-04-26T17:46:49.636 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: overall HEALTH_OK 2024-04-26T17:46:49.636 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.636 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.636 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:49.636 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:49.636 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-26T17:46:49.636 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-26T17:46:49.636 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T17:46:49.636 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:49.636 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:49.637 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: Updating smithi031:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:49.637 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.637 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.637 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.637 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: mon.b calling monitor election 2024-04-26T17:46:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: mon.b calling monitor election 2024-04-26T17:46:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: mon.c calling monitor election 2024-04-26T17:46:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: mon.a calling monitor election 2024-04-26T17:46:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T17:46:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: monmap epoch 3 2024-04-26T17:46:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:46:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: last_changed 2024-04-26T17:46:41.654574+0000 2024-04-26T17:46:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: created 2024-04-26T17:44:45.630672+0000 2024-04-26T17:46:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: min_mon_release 19 (squid) 2024-04-26T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: election_strategy: 1 2024-04-26T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-26T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: 2: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.b 2024-04-26T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: fsmap 2024-04-26T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: mgrmap e13: x(active, since 60s) 2024-04-26T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: overall HEALTH_OK 2024-04-26T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-26T17:46:49.769 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-26T17:46:49.769 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T17:46:49.769 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:49.769 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:49.769 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: Updating smithi031:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:46:49.769 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.769 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.769 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.769 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:49.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.817+0000 7f8b28761700 1 -- 172.21.15.31:0/2231312883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b20104030 msgr2=0x7f8b20104410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:49.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.817+0000 7f8b28761700 1 --2- 172.21.15.31:0/2231312883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b20104030 0x7f8b20104410 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c009910 tx=0x7f8b1c009c20 comp rx=0 tx=0).stop 2024-04-26T17:46:49.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.817+0000 7f8b28761700 1 -- 172.21.15.31:0/2231312883 shutdown_connections 2024-04-26T17:46:49.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.817+0000 7f8b28761700 1 --2- 172.21.15.31:0/2231312883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b20104030 0x7f8b20104410 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:49.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.817+0000 7f8b28761700 1 -- 172.21.15.31:0/2231312883 >> 172.21.15.31:0/2231312883 conn(0x7f8b200fd880 msgr2=0x7f8b200ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:49.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.817+0000 7f8b28761700 1 -- 172.21.15.31:0/2231312883 shutdown_connections 2024-04-26T17:46:49.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.817+0000 7f8b28761700 1 -- 172.21.15.31:0/2231312883 wait complete. 2024-04-26T17:46:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.818+0000 7f8b28761700 1 Processor -- start 2024-04-26T17:46:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.818+0000 7f8b28761700 1 -- start start 2024-04-26T17:46:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.818+0000 7f8b28761700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8b20104030 0x7f8b2019e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.818+0000 7f8b28761700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b2019ed00 0x7f8b20198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.818+0000 7f8b28761700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b20198ec0 0x7f8b20199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b28761700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2010d3b0 con 0x7f8b20198ec0 2024-04-26T17:46:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b28761700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2010d230 con 0x7f8b20104030 2024-04-26T17:46:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b28761700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2010d530 con 0x7f8b2019ed00 2024-04-26T17:46:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b26cfe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b20198ec0 0x7f8b20199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b26cfe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b20198ec0 0x7f8b20199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39654/0 (socket says 172.21.15.31:39654) 2024-04-26T17:46:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b26cfe700 1 -- 172.21.15.31:0/3410002655 learned_addr learned my addr 172.21.15.31:0/3410002655 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:46:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b264fd700 1 --2- 172.21.15.31:0/3410002655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8b20104030 0x7f8b2019e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b25cfc700 1 --2- 172.21.15.31:0/3410002655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b2019ed00 0x7f8b20198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b26cfe700 1 -- 172.21.15.31:0/3410002655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b2019ed00 msgr2=0x7f8b20198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b26cfe700 1 --2- 172.21.15.31:0/3410002655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b2019ed00 0x7f8b20198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b26cfe700 1 -- 172.21.15.31:0/3410002655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8b20104030 msgr2=0x7f8b2019e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b26cfe700 1 --2- 172.21.15.31:0/3410002655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8b20104030 0x7f8b2019e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b26cfe700 1 -- 172.21.15.31:0/3410002655 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b201a5690 con 0x7f8b20198ec0 2024-04-26T17:46:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b26cfe700 1 --2- 172.21.15.31:0/3410002655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b20198ec0 0x7f8b20199370 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800ba50 tx=0x7f8b1800bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b177fe700 1 -- 172.21.15.31:0/3410002655 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18019070 con 0x7f8b20198ec0 2024-04-26T17:46:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b177fe700 1 -- 172.21.15.31:0/3410002655 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b1800cde0 con 0x7f8b20198ec0 2024-04-26T17:46:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b28761700 1 -- 172.21.15.31:0/3410002655 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b201a58c0 con 0x7f8b20198ec0 2024-04-26T17:46:49.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.819+0000 7f8b177fe700 1 -- 172.21.15.31:0/3410002655 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18007350 con 0x7f8b20198ec0 2024-04-26T17:46:49.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.820+0000 7f8b28761700 1 -- 172.21.15.31:0/3410002655 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b201a5e00 con 0x7f8b20198ec0 2024-04-26T17:46:49.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.820+0000 7f8b177fe700 1 -- 172.21.15.31:0/3410002655 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8b1801c070 con 0x7f8b20198ec0 2024-04-26T17:46:49.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.820+0000 7f8b28761700 1 -- 172.21.15.31:0/3410002655 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b20064a80 con 0x7f8b20198ec0 2024-04-26T17:46:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.821+0000 7f8b177fe700 1 --2- 172.21.15.31:0/3410002655 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8b0c039ae0 0x7f8b0c03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.821+0000 7f8b177fe700 1 -- 172.21.15.31:0/3410002655 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8b1804bed0 con 0x7f8b20198ec0 2024-04-26T17:46:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.821+0000 7f8b264fd700 1 --2- 172.21.15.31:0/3410002655 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8b0c039ae0 0x7f8b0c03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.821+0000 7f8b264fd700 1 --2- 172.21.15.31:0/3410002655 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8b0c039ae0 0x7f8b0c03bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c006010 tx=0x7f8b1c01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:49.826+0000 7f8b177fe700 1 -- 172.21.15.31:0/3410002655 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b180145d0 con 0x7f8b20198ec0 2024-04-26T17:46:50.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:50.041+0000 7f8b28761700 1 -- 172.21.15.31:0/3410002655 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f8b201a6030 con 0x7f8b20198ec0 2024-04-26T17:46:50.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:50.041+0000 7f8b177fe700 1 -- 172.21.15.31:0/3410002655 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f8b1801e440 con 0x7f8b20198ec0 2024-04-26T17:46:50.042 INFO:teuthology.orchestra.run.smithi031.stdout:# minimal ceph.conf for 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:46:50.042 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-04-26T17:46:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:46:50.042 INFO:teuthology.orchestra.run.smithi031.stdout: mon_host = [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T17:46:50.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:50.043+0000 7f8b28761700 1 -- 172.21.15.31:0/3410002655 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8b0c039ae0 msgr2=0x7f8b0c03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:50.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:50.043+0000 7f8b28761700 1 --2- 172.21.15.31:0/3410002655 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8b0c039ae0 0x7f8b0c03bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c006010 tx=0x7f8b1c01a040 comp rx=0 tx=0).stop 2024-04-26T17:46:50.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:50.043+0000 7f8b28761700 1 -- 172.21.15.31:0/3410002655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b20198ec0 msgr2=0x7f8b20199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:50.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:50.043+0000 7f8b28761700 1 --2- 172.21.15.31:0/3410002655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b20198ec0 0x7f8b20199370 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800ba50 tx=0x7f8b1800bd60 comp rx=0 tx=0).stop 2024-04-26T17:46:50.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:50.043+0000 7f8b28761700 1 -- 172.21.15.31:0/3410002655 shutdown_connections 2024-04-26T17:46:50.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:50.043+0000 7f8b28761700 1 --2- 172.21.15.31:0/3410002655 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8b0c039ae0 0x7f8b0c03bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:50.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:50.043+0000 7f8b28761700 1 --2- 172.21.15.31:0/3410002655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8b20104030 0x7f8b2019e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:50.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:50.043+0000 7f8b28761700 1 --2- 172.21.15.31:0/3410002655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b20198ec0 0x7f8b20199370 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:50.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:50.043+0000 7f8b28761700 1 --2- 172.21.15.31:0/3410002655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b2019ed00 0x7f8b20198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:50.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:50.043+0000 7f8b28761700 1 -- 172.21.15.31:0/3410002655 >> 172.21.15.31:0/3410002655 conn(0x7f8b200fd880 msgr2=0x7f8b20104f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:50.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:50.043+0000 7f8b28761700 1 -- 172.21.15.31:0/3410002655 shutdown_connections 2024-04-26T17:46:50.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:46:50.043+0000 7f8b28761700 1 -- 172.21.15.31:0/3410002655 wait complete. 2024-04-26T17:46:50.347 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:50 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.347 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:50 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.347 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:50 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.347 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:50 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.347 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:50 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:50 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:50 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:50 smithi031 ceph-mon[29164]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T17:46:50.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:50 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:50.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:50 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T17:46:50.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:50 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:50.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:50 smithi031 ceph-mon[29164]: Reconfiguring daemon mon.a on smithi031 2024-04-26T17:46:50.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:50 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:50.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:50 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3410002655' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:50.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:50 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:50 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:50 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:50 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:50 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:50 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:50 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:50 smithi097 ceph-mon[32419]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T17:46:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:50 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:50 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T17:46:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:50 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:50 smithi097 ceph-mon[32419]: Reconfiguring daemon mon.a on smithi031 2024-04-26T17:46:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:50 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:50 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3410002655' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:50.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:50 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:50 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:50 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:50 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:50 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:50 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:50 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:50 smithi186 ceph-mon[32797]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T17:46:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:50 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:50 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T17:46:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:50 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:50 smithi186 ceph-mon[32797]: Reconfiguring daemon mon.a on smithi031 2024-04-26T17:46:50.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:50 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:46:50.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:50 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3410002655' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:50.777 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-26T17:46:50.777 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:46:50.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T17:46:50.804 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:46:50.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:46:50.869 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-26T17:46:50.869 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T17:46:50.899 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-26T17:46:50.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:46:50.964 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T17:46:50.964 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T17:46:50.995 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T17:46:50.995 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:46:51.063 INFO:tasks.cephadm:Adding mgr.x on smithi031 2024-04-26T17:46:51.063 INFO:tasks.cephadm:Adding mgr.y on smithi097 2024-04-26T17:46:51.063 INFO:tasks.cephadm:Adding mgr.z on smithi186 2024-04-26T17:46:51.063 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi031=x;smithi097=y;smithi186=z' 2024-04-26T17:46:51.337 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T17:46:51.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:51 smithi097 ceph-mon[32419]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:51.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:51 smithi031 ceph-mon[29164]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:51.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:51 smithi186 ceph-mon[32797]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:52.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.804+0000 7fed96586700 1 -- 172.21.15.186:0/2879128086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed90101290 msgr2=0x7fed901016b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:52.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.804+0000 7fed96586700 1 --2- 172.21.15.186:0/2879128086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed90101290 0x7fed901016b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fed80009910 tx=0x7fed80009c20 comp rx=0 tx=0).stop 2024-04-26T17:46:52.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.804+0000 7fed96586700 1 -- 172.21.15.186:0/2879128086 shutdown_connections 2024-04-26T17:46:52.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.804+0000 7fed96586700 1 --2- 172.21.15.186:0/2879128086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed90101290 0x7fed901016b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:52.806 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.804+0000 7fed96586700 1 -- 172.21.15.186:0/2879128086 >> 172.21.15.186:0/2879128086 conn(0x7fed900fcbd0 msgr2=0x7fed900ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:52.806 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.804+0000 7fed96586700 1 -- 172.21.15.186:0/2879128086 shutdown_connections 2024-04-26T17:46:52.806 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.804+0000 7fed96586700 1 -- 172.21.15.186:0/2879128086 wait complete. 2024-04-26T17:46:52.806 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.805+0000 7fed96586700 1 Processor -- start 2024-04-26T17:46:52.806 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.805+0000 7fed96586700 1 -- start start 2024-04-26T17:46:52.806 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.805+0000 7fed96586700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed90101290 0x7fed90198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:52.806 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.805+0000 7fed96586700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed90198d60 0x7fed9019ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:52.806 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.805+0000 7fed96586700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed901991e0 0x7fed901a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:52.806 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.805+0000 7fed96586700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fed90108f10 con 0x7fed90198d60 2024-04-26T17:46:52.806 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.805+0000 7fed96586700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fed90108d90 con 0x7fed901991e0 2024-04-26T17:46:52.806 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.805+0000 7fed96586700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fed90109090 con 0x7fed90101290 2024-04-26T17:46:52.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.805+0000 7fed8ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed90101290 0x7fed90198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:52.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.805+0000 7fed8ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed90101290 0x7fed90198820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43254/0 (socket says 172.21.15.186:43254) 2024-04-26T17:46:52.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.805+0000 7fed8ffff700 1 -- 172.21.15.186:0/1165005157 learned_addr learned my addr 172.21.15.186:0/1165005157 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T17:46:52.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.805+0000 7fed8f7fe700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed90198d60 0x7fed9019ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:52.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.806+0000 7fed94b23700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed901991e0 0x7fed901a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:52.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.806+0000 7fed8ffff700 1 -- 172.21.15.186:0/1165005157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed901991e0 msgr2=0x7fed901a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:52.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.806+0000 7fed8ffff700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed901991e0 0x7fed901a0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:52.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.806+0000 7fed8ffff700 1 -- 172.21.15.186:0/1165005157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed90198d60 msgr2=0x7fed9019ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:52.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.806+0000 7fed8ffff700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed90198d60 0x7fed9019ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:52.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.806+0000 7fed8ffff700 1 -- 172.21.15.186:0/1165005157 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed901a0c30 con 0x7fed90101290 2024-04-26T17:46:52.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.806+0000 7fed94b23700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed901991e0 0x7fed901a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:46:52.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.806+0000 7fed8ffff700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed90101290 0x7fed90198820 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fed80004f80 tx=0x7fed80004fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:52.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.806+0000 7fed8d7fa700 1 -- 172.21.15.186:0/1165005157 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80026070 con 0x7fed90101290 2024-04-26T17:46:52.807 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.806+0000 7fed8d7fa700 1 -- 172.21.15.186:0/1165005157 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed8000b750 con 0x7fed90101290 2024-04-26T17:46:52.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.806+0000 7fed96586700 1 -- 172.21.15.186:0/1165005157 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed901a0ec0 con 0x7fed90101290 2024-04-26T17:46:52.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.806+0000 7fed8d7fa700 1 -- 172.21.15.186:0/1165005157 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed800038c0 con 0x7fed90101290 2024-04-26T17:46:52.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.806+0000 7fed96586700 1 -- 172.21.15.186:0/1165005157 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed901a1330 con 0x7fed90101290 2024-04-26T17:46:52.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.807+0000 7fed8d7fa700 1 -- 172.21.15.186:0/1165005157 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fed80029070 con 0x7fed90101290 2024-04-26T17:46:52.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.807+0000 7fed8d7fa700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fed78039ae0 0x7fed7803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:46:52.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.807+0000 7fed8d7fa700 1 -- 172.21.15.186:0/1165005157 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fed8004d500 con 0x7fed90101290 2024-04-26T17:46:52.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.807+0000 7fed96586700 1 -- 172.21.15.186:0/1165005157 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed90064a80 con 0x7fed90101290 2024-04-26T17:46:52.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.807+0000 7fed8f7fe700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fed78039ae0 0x7fed7803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:46:52.809 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.808+0000 7fed8f7fe700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fed78039ae0 0x7fed7803bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fed84009ad0 tx=0x7fed84009210 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:46:52.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:52.813+0000 7fed8d7fa700 1 -- 172.21.15.186:0/1165005157 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed80030020 con 0x7fed90101290 2024-04-26T17:46:52.966 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:52 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:52.966 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:52 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:52.966 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:52 smithi097 ceph-mon[32419]: Reconfiguring mon.b (monmap changed)... 2024-04-26T17:46:52.966 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:52 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:52.966 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:52 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T17:46:52.966 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:52 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:52.966 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:52 smithi097 ceph-mon[32419]: Reconfiguring daemon mon.b on smithi097 2024-04-26T17:46:53.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:52 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:53.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:52 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:53.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:52 smithi186 ceph-mon[32797]: Reconfiguring mon.b (monmap changed)... 2024-04-26T17:46:53.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:52 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:53.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:52 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T17:46:53.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:52 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:53.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:52 smithi186 ceph-mon[32797]: Reconfiguring daemon mon.b on smithi097 2024-04-26T17:46:53.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:53.023+0000 7fed96586700 1 -- 172.21.15.186:0/1165005157 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi097=y;smithi186=z", "target": ["mon-mgr", ""]}) -- 0x7fed900feb60 con 0x7fed78039ae0 2024-04-26T17:46:53.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:53.029+0000 7fed8d7fa700 1 -- 172.21.15.186:0/1165005157 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fed900feb60 con 0x7fed78039ae0 2024-04-26T17:46:53.028 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mgr update... 2024-04-26T17:46:53.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:53.031+0000 7fed96586700 1 -- 172.21.15.186:0/1165005157 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fed78039ae0 msgr2=0x7fed7803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:53.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:53.031+0000 7fed96586700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fed78039ae0 0x7fed7803bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fed84009ad0 tx=0x7fed84009210 comp rx=0 tx=0).stop 2024-04-26T17:46:53.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:53.031+0000 7fed96586700 1 -- 172.21.15.186:0/1165005157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed90101290 msgr2=0x7fed90198820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:46:53.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:53.031+0000 7fed96586700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed90101290 0x7fed90198820 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fed80004f80 tx=0x7fed80004fb0 comp rx=0 tx=0).stop 2024-04-26T17:46:53.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:53.032+0000 7fed96586700 1 -- 172.21.15.186:0/1165005157 shutdown_connections 2024-04-26T17:46:53.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:53.032+0000 7fed96586700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fed78039ae0 0x7fed7803bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:53.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:53.032+0000 7fed96586700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed901991e0 0x7fed901a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:53.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:53.032+0000 7fed96586700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed90101290 0x7fed90198820 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:53.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:53.032+0000 7fed96586700 1 --2- 172.21.15.186:0/1165005157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed90198d60 0x7fed9019ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:46:53.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:53.032+0000 7fed96586700 1 -- 172.21.15.186:0/1165005157 >> 172.21.15.186:0/1165005157 conn(0x7fed900fcbd0 msgr2=0x7fed900fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:46:53.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:53.032+0000 7fed96586700 1 -- 172.21.15.186:0/1165005157 shutdown_connections 2024-04-26T17:46:53.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:46:53.032+0000 7fed96586700 1 -- 172.21.15.186:0/1165005157 wait complete. 2024-04-26T17:46:53.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:52 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:53.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:52 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:53.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:52 smithi031 ceph-mon[29164]: Reconfiguring mon.b (monmap changed)... 2024-04-26T17:46:53.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:52 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:53.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:52 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T17:46:53.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:52 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:53.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:52 smithi031 ceph-mon[29164]: Reconfiguring daemon mon.b on smithi097 2024-04-26T17:46:53.665 DEBUG:teuthology.orchestra.run.smithi097:mgr.y> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mgr.y.service 2024-04-26T17:46:53.668 DEBUG:teuthology.orchestra.run.smithi186:mgr.z> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mgr.z.service 2024-04-26T17:46:53.670 INFO:tasks.cephadm:Deploying OSDs... 2024-04-26T17:46:53.670 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:46:53.670 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T17:46:53.691 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T17:46:53.691 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-04-26T17:46:53.749 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T17:46:53.749 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T17:46:53.749 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-26T17:46:53.749 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T17:46:53.749 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T17:46:53.749 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-26 17:45:57.969716607 +0000 2024-04-26T17:46:53.750 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-26 17:41:07.415177128 +0000 2024-04-26T17:46:53.750 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-26 17:41:07.415177128 +0000 2024-04-26T17:46:53.750 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-26 17:41:07.415177128 +0000 2024-04-26T17:46:53.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T17:46:53.816 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-26T17:46:53.816 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-26T17:46:53.816 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000225746 s, 2.3 MB/s 2024-04-26T17:46:53.817 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T17:46:53.875 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-04-26T17:46:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T17:46:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T17:46:53.932 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-26T17:46:53.932 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T17:46:53.932 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T17:46:53.932 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-26 17:45:58.005716032 +0000 2024-04-26T17:46:53.932 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-26 17:41:07.790170675 +0000 2024-04-26T17:46:53.932 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-26 17:41:07.790170675 +0000 2024-04-26T17:46:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-26 17:41:07.790170675 +0000 2024-04-26T17:46:53.933 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T17:46:53.998 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-26T17:46:53.998 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-26T17:46:53.998 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000205501 s, 2.5 MB/s 2024-04-26T17:46:53.999 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T17:46:54.057 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-04-26T17:46:54.114 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T17:46:54.114 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T17:46:54.114 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-26T17:46:54.114 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T17:46:54.114 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T17:46:54.114 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-26 17:45:58.040715472 +0000 2024-04-26T17:46:54.114 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-26 17:41:08.133164772 +0000 2024-04-26T17:46:54.114 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-26 17:41:08.133164772 +0000 2024-04-26T17:46:54.114 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-26 17:41:08.133164772 +0000 2024-04-26T17:46:54.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T17:46:54.176 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:54 smithi031 ceph-mon[29164]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:54.177 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:54 smithi031 ceph-mon[29164]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi097=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:46:54.177 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:54 smithi031 ceph-mon[29164]: Saving service mgr spec with placement smithi031=x;smithi097=y;smithi186=z;count:3 2024-04-26T17:46:54.177 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:54.177 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:54.177 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:54.177 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:54.177 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T17:46:54.177 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:54.179 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-26T17:46:54.179 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-26T17:46:54.179 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000194543 s, 2.6 MB/s 2024-04-26T17:46:54.180 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T17:46:54.189 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:54 smithi186 ceph-mon[32797]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:54.189 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:54 smithi186 ceph-mon[32797]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi097=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:46:54.189 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:54 smithi186 ceph-mon[32797]: Saving service mgr spec with placement smithi031=x;smithi097=y;smithi186=z;count:3 2024-04-26T17:46:54.189 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:54.189 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:54.189 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:54.189 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:54.189 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T17:46:54.190 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:54.237 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-04-26T17:46:54.293 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T17:46:54.293 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T17:46:54.293 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-26T17:46:54.294 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T17:46:54.294 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T17:46:54.294 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-26 17:45:58.075714913 +0000 2024-04-26T17:46:54.294 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-26 17:41:08.483158749 +0000 2024-04-26T17:46:54.294 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-26 17:41:08.483158749 +0000 2024-04-26T17:46:54.294 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-26 17:41:08.483158749 +0000 2024-04-26T17:46:54.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T17:46:54.357 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-26T17:46:54.357 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-26T17:46:54.357 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.00022953 s, 2.2 MB/s 2024-04-26T17:46:54.358 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T17:46:54.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:54 smithi097 ceph-mon[32419]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:54.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:54 smithi097 ceph-mon[32419]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi097=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:46:54.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:54 smithi097 ceph-mon[32419]: Saving service mgr spec with placement smithi031=x;smithi097=y;smithi186=z;count:3 2024-04-26T17:46:54.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:54.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:54.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:54.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T17:46:54.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T17:46:54.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:54.415 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-26T17:46:54.415 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T17:46:54.434 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T17:46:54.434 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2024-04-26T17:46:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T17:46:54.492 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T17:46:54.492 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-26T17:46:54.492 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T17:46:54.492 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T17:46:54.492 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-26 17:46:41.085958414 +0000 2024-04-26T17:46:54.493 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-26 17:41:22.163807677 +0000 2024-04-26T17:46:54.493 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-26 17:41:22.163807677 +0000 2024-04-26T17:46:54.493 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-04-26 17:41:22.163807677 +0000 2024-04-26T17:46:54.493 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T17:46:54.559 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-26T17:46:54.559 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-26T17:46:54.559 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000385235 s, 1.3 MB/s 2024-04-26T17:46:54.560 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T17:46:54.619 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2024-04-26T17:46:54.677 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T17:46:54.677 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T17:46:54.677 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-26T17:46:54.677 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T17:46:54.677 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T17:46:54.677 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-26 17:46:41.085958414 +0000 2024-04-26T17:46:54.678 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-26 17:41:22.555800746 +0000 2024-04-26T17:46:54.678 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-26 17:41:22.555800746 +0000 2024-04-26T17:46:54.678 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-04-26 17:41:22.555800746 +0000 2024-04-26T17:46:54.678 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T17:46:54.742 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-26T17:46:54.743 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-26T17:46:54.743 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000188615 s, 2.7 MB/s 2024-04-26T17:46:54.744 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T17:46:54.801 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2024-04-26T17:46:54.857 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T17:46:54.857 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T17:46:54.857 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-26T17:46:54.857 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T17:46:54.857 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T17:46:54.857 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-26 17:46:41.086958398 +0000 2024-04-26T17:46:54.857 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-26 17:41:22.916794364 +0000 2024-04-26T17:46:54.857 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-26 17:41:22.916794364 +0000 2024-04-26T17:46:54.857 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-04-26 17:41:22.916794364 +0000 2024-04-26T17:46:54.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T17:46:54.921 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-26T17:46:54.921 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-26T17:46:54.921 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000200506 s, 2.6 MB/s 2024-04-26T17:46:54.922 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T17:46:54.977 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2024-04-26T17:46:55.033 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T17:46:55.033 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T17:46:55.033 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-26T17:46:55.034 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T17:46:55.034 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T17:46:55.034 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-26 17:46:41.086958398 +0000 2024-04-26T17:46:55.034 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-26 17:41:23.246788529 +0000 2024-04-26T17:46:55.034 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-26 17:41:23.246788529 +0000 2024-04-26T17:46:55.034 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-04-26 17:41:23.246788529 +0000 2024-04-26T17:46:55.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T17:46:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-26T17:46:55.096 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-26T17:46:55.096 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000213837 s, 2.4 MB/s 2024-04-26T17:46:55.097 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T17:46:55.152 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T17:46:55.152 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T17:46:55.172 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T17:46:55.172 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-04-26T17:46:55.229 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T17:46:55.229 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T17:46:55.229 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-26T17:46:55.229 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T17:46:55.229 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T17:46:55.230 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 17:46:32.094671396 +0000 2024-04-26T17:46:55.230 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 17:41:24.550554623 +0000 2024-04-26T17:46:55.230 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 17:41:24.550554623 +0000 2024-04-26T17:46:55.230 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-26 17:41:24.550554623 +0000 2024-04-26T17:46:55.230 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T17:46:55.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:55 smithi186 ceph-mon[32797]: Reconfiguring mon.c (monmap changed)... 2024-04-26T17:46:55.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:55 smithi186 ceph-mon[32797]: Reconfiguring daemon mon.c on smithi186 2024-04-26T17:46:55.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:55 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:55.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:55 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:55.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:55 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:55.293 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T17:46:55.293 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T17:46:55.293 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000300612 s, 1.7 MB/s 2024-04-26T17:46:55.294 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T17:46:55.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:55 smithi031 ceph-mon[29164]: Reconfiguring mon.c (monmap changed)... 2024-04-26T17:46:55.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:55 smithi031 ceph-mon[29164]: Reconfiguring daemon mon.c on smithi186 2024-04-26T17:46:55.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:55 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:55.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:55 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:55.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:55 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:55.352 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-04-26T17:46:55.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:55 smithi097 ceph-mon[32419]: Reconfiguring mon.c (monmap changed)... 2024-04-26T17:46:55.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:55 smithi097 ceph-mon[32419]: Reconfiguring daemon mon.c on smithi186 2024-04-26T17:46:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:55 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:55 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:55 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:46:55.409 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T17:46:55.409 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T17:46:55.409 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-26T17:46:55.409 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T17:46:55.410 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T17:46:55.410 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 17:46:32.094671396 +0000 2024-04-26T17:46:55.410 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 17:41:24.889548150 +0000 2024-04-26T17:46:55.410 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 17:41:24.889548150 +0000 2024-04-26T17:46:55.410 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-26 17:41:24.889548150 +0000 2024-04-26T17:46:55.410 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T17:46:55.474 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T17:46:55.474 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T17:46:55.474 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000209242 s, 2.4 MB/s 2024-04-26T17:46:55.475 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T17:46:55.532 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-04-26T17:46:55.588 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T17:46:55.588 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T17:46:55.588 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-26T17:46:55.588 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T17:46:55.588 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T17:46:55.588 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 17:46:32.095671379 +0000 2024-04-26T17:46:55.588 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 17:41:25.251541238 +0000 2024-04-26T17:46:55.588 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 17:41:25.251541238 +0000 2024-04-26T17:46:55.588 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-26 17:41:25.251541238 +0000 2024-04-26T17:46:55.589 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T17:46:55.650 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T17:46:55.650 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T17:46:55.650 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000195757 s, 2.6 MB/s 2024-04-26T17:46:55.651 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T17:46:55.707 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-04-26T17:46:55.761 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T17:46:55.761 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T17:46:55.762 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-26T17:46:55.762 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T17:46:55.762 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T17:46:55.762 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 17:46:32.095671379 +0000 2024-04-26T17:46:55.762 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 17:41:25.603534516 +0000 2024-04-26T17:46:55.762 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 17:41:25.603534516 +0000 2024-04-26T17:46:55.762 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-26 17:41:25.603534516 +0000 2024-04-26T17:46:55.762 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T17:46:55.825 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T17:46:55.825 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T17:46:55.825 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.0002053 s, 2.5 MB/s 2024-04-26T17:46:55.826 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T17:46:55.883 INFO:tasks.cephadm:Deploying osd.0 on smithi031 with /dev/vg_nvme/lv_4... 2024-04-26T17:46:55.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T17:46:56.126 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:46:56.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:56 smithi031 ceph-mon[29164]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:56.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:56.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:56.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:56.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T17:46:56.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T17:46:56.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T17:46:56.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:56.370 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:56 smithi186 ceph-mon[32797]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:56.370 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:56.371 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:56.371 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:56.371 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T17:46:56.371 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T17:46:56.371 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T17:46:56.371 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:56.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:56 smithi097 ceph-mon[32419]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:56.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:56.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:46:56.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:56.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T17:46:56.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T17:46:56.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T17:46:56.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:57.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:57 smithi031 ceph-mon[29164]: Deploying daemon mgr.z on smithi186 2024-04-26T17:46:57.330 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:57 smithi186 ceph-mon[32797]: Deploying daemon mgr.z on smithi186 2024-04-26T17:46:57.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:57 smithi097 ceph-mon[32419]: Deploying daemon mgr.z on smithi186 2024-04-26T17:46:57.701 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:46:57 smithi186 systemd[1]: Starting Ceph mgr.z for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:46:57.995 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:46:57 smithi186 podman[34638]: 2024-04-26 17:46:57.702352875 +0000 UTC m=+0.105593085 container create e4e8d064514ae8881acad3cd8fd0dc94004d35d08ba9d4836e5e08fa35c2c0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-26T17:46:57.995 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:46:57 smithi186 podman[34638]: 2024-04-26 17:46:57.614811505 +0000 UTC m=+0.018051661 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:46:57.995 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:46:57 smithi186 podman[34638]: 2024-04-26 17:46:57.904700187 +0000 UTC m=+0.307940325 container init e4e8d064514ae8881acad3cd8fd0dc94004d35d08ba9d4836e5e08fa35c2c0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-26T17:46:57.995 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:46:57 smithi186 podman[34638]: 2024-04-26 17:46:57.907151494 +0000 UTC m=+0.310391634 container start e4e8d064514ae8881acad3cd8fd0dc94004d35d08ba9d4836e5e08fa35c2c0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T17:46:57.995 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:46:57 smithi186 ceph-mgr[34656]: -- 172.21.15.186:0/2501698922 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55db8c19a5a0 con 0x55db8c16ec00 2024-04-26T17:46:57.995 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:46:57 smithi186 bash[34638]: e4e8d064514ae8881acad3cd8fd0dc94004d35d08ba9d4836e5e08fa35c2c0b2 2024-04-26T17:46:58.003 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T17:46:58.003 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T17:46:58.003 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-04-26T17:46:58.003 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-04-26T17:46:58.003 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013114 s, 800 MB/s 2024-04-26T17:46:58.003 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-04-26T17:46:58.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:58 smithi186 ceph-mon[32797]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:58.267 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:46:57 smithi186 systemd[1]: Started Ceph mgr.z for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:46:58.267 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:46:58 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:46:58.058+0000 7f72d76ff200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T17:46:58.267 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:46:58 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:46:58.244+0000 7f72d76ff200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:46:58.304 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:58 smithi097 ceph-mon[32419]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:58.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:58 smithi031 ceph-mon[29164]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:46:58.651 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:46:58 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:46:58.345+0000 7f72d76ff200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T17:46:58.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_4 2024-04-26T17:46:58.917 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:46:59.016 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:46:58 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:46:58.652+0000 7f72d76ff200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T17:46:59.017 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:46:58 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:46:58.735+0000 7f72d76ff200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T17:46:59.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:59.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:59.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:59.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T17:46:59.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T17:46:59.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T17:46:59.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:46:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:59.357 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:59.357 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:59.357 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:59.357 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T17:46:59.357 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T17:46:59.357 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T17:46:59.357 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:46:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:46:59.371 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:59.372 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:59.372 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:46:59.372 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T17:46:59.372 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T17:46:59.372 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T17:46:59.372 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:46:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:00.158 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:00 smithi031 ceph-mon[29164]: Deploying daemon mgr.y on smithi097 2024-04-26T17:47:00.158 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:00 smithi031 ceph-mon[29164]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:00.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:00 smithi186 ceph-mon[32797]: Deploying daemon mgr.y on smithi097 2024-04-26T17:47:00.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:00 smithi186 ceph-mon[32797]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:00.267 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:46:59 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:46:59.962+0000 7f72d76ff200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T17:47:00.267 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:00 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:00.135+0000 7f72d76ff200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T17:47:00.338 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:00 smithi097 ceph-mon[32419]: Deploying daemon mgr.y on smithi097 2024-04-26T17:47:00.338 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:00 smithi097 ceph-mon[32419]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:00.338 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:00 smithi097 systemd[1]: Starting Ceph mgr.y for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:47:00.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.378+0000 7f468a8b9700 1 -- 172.21.15.31:0/2083405559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4684103070 msgr2=0x7f46841034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:47:00.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.378+0000 7f468a8b9700 1 --2- 172.21.15.31:0/2083405559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4684103070 0x7f46841034f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f467c00b0a0 tx=0x7f467c00b3b0 comp rx=0 tx=0).stop 2024-04-26T17:47:00.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.378+0000 7f468a8b9700 1 -- 172.21.15.31:0/2083405559 shutdown_connections 2024-04-26T17:47:00.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.378+0000 7f468a8b9700 1 --2- 172.21.15.31:0/2083405559 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4684101d50 0x7f4684102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:00.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.378+0000 7f468a8b9700 1 --2- 172.21.15.31:0/2083405559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4684103070 0x7f46841034f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:00.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.378+0000 7f468a8b9700 1 --2- 172.21.15.31:0/2083405559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46841026b0 0x7f4684102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.378+0000 7f468a8b9700 1 -- 172.21.15.31:0/2083405559 >> 172.21.15.31:0/2083405559 conn(0x7f46840fd910 msgr2=0x7f46840ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:47:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.378+0000 7f468a8b9700 1 -- 172.21.15.31:0/2083405559 shutdown_connections 2024-04-26T17:47:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.378+0000 7f468a8b9700 1 -- 172.21.15.31:0/2083405559 wait complete. 2024-04-26T17:47:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.378+0000 7f468a8b9700 1 Processor -- start 2024-04-26T17:47:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.378+0000 7f468a8b9700 1 -- start start 2024-04-26T17:47:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f468a8b9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4684101d50 0x7f468419b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:47:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f468a8b9700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46841026b0 0x7f468419bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:47:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f468a8b9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4684103070 0x7f46841a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:47:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f468a8b9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4684111140 con 0x7f4684103070 2024-04-26T17:47:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f468a8b9700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4684110fc0 con 0x7f46841026b0 2024-04-26T17:47:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f468a8b9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f46841112c0 con 0x7f4684101d50 2024-04-26T17:47:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f4688e56700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4684103070 0x7f46841a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:47:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f4688e56700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4684103070 0x7f46841a2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43442/0 (socket says 172.21.15.31:43442) 2024-04-26T17:47:00.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f4688e56700 1 -- 172.21.15.31:0/2490882742 learned_addr learned my addr 172.21.15.31:0/2490882742 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:47:00.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f4683fff700 1 --2- 172.21.15.31:0/2490882742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4684101d50 0x7f468419b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:47:00.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f46837fe700 1 --2- 172.21.15.31:0/2490882742 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46841026b0 0x7f468419bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:47:00.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f4688e56700 1 -- 172.21.15.31:0/2490882742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4684101d50 msgr2=0x7f468419b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:47:00.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f4688e56700 1 --2- 172.21.15.31:0/2490882742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4684101d50 0x7f468419b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:00.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f4688e56700 1 -- 172.21.15.31:0/2490882742 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46841026b0 msgr2=0x7f468419bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:47:00.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f4688e56700 1 --2- 172.21.15.31:0/2490882742 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46841026b0 0x7f468419bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:00.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f4688e56700 1 -- 172.21.15.31:0/2490882742 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46841a3420 con 0x7f4684103070 2024-04-26T17:47:00.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.379+0000 7f4688e56700 1 --2- 172.21.15.31:0/2490882742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4684103070 0x7f46841a2d80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f467c00b070 tx=0x7f467c004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:47:00.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.380+0000 7f46817fa700 1 -- 172.21.15.31:0/2490882742 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c021070 con 0x7f4684103070 2024-04-26T17:47:00.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.380+0000 7f46817fa700 1 -- 172.21.15.31:0/2490882742 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f467c007c00 con 0x7f4684103070 2024-04-26T17:47:00.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.380+0000 7f468a8b9700 1 -- 172.21.15.31:0/2490882742 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46841a36b0 con 0x7f4684103070 2024-04-26T17:47:00.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.380+0000 7f46817fa700 1 -- 172.21.15.31:0/2490882742 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c0047c0 con 0x7f4684103070 2024-04-26T17:47:00.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.380+0000 7f468a8b9700 1 -- 172.21.15.31:0/2490882742 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46841a3c40 con 0x7f4684103070 2024-04-26T17:47:00.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.381+0000 7f46817fa700 1 -- 172.21.15.31:0/2490882742 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f467c01d450 con 0x7f4684103070 2024-04-26T17:47:00.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.381+0000 7f46817fa700 1 --2- 172.21.15.31:0/2490882742 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f466c039890 0x7f466c03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:47:00.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.381+0000 7f46817fa700 1 -- 172.21.15.31:0/2490882742 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f467c00f020 con 0x7f4684103070 2024-04-26T17:47:00.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.381+0000 7f4683fff700 1 --2- 172.21.15.31:0/2490882742 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f466c039890 0x7f466c03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:47:00.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.381+0000 7f468a8b9700 1 -- 172.21.15.31:0/2490882742 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f468406a110 con 0x7f4684103070 2024-04-26T17:47:00.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.381+0000 7f4683fff700 1 --2- 172.21.15.31:0/2490882742 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f466c039890 0x7f466c03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4674009a10 tx=0x7f4674009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:47:00.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.386+0000 7f46817fa700 1 -- 172.21.15.31:0/2490882742 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f467c02a020 con 0x7f4684103070 2024-04-26T17:47:00.597 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:00 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:00.311+0000 7f72d76ff200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T17:47:00.602 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:00 smithi097 podman[33792]: 2024-04-26 17:47:00.241720384 +0000 UTC m=+0.018145895 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:47:00.602 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:00 smithi097 podman[33792]: 2024-04-26 17:47:00.342835946 +0000 UTC m=+0.119261445 container create d1c415b7a384d01efce34f8308d3cab5aa73f55051b70d93e40aabdba2e1cdba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T17:47:00.602 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:00 smithi097 podman[33792]: 2024-04-26 17:47:00.536866753 +0000 UTC m=+0.313292253 container init d1c415b7a384d01efce34f8308d3cab5aa73f55051b70d93e40aabdba2e1cdba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T17:47:00.602 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:00 smithi097 podman[33792]: 2024-04-26 17:47:00.539415672 +0000 UTC m=+0.315841176 container start d1c415b7a384d01efce34f8308d3cab5aa73f55051b70d93e40aabdba2e1cdba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-26T17:47:00.602 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:00 smithi097 ceph-mgr[33811]: -- 172.21.15.97:0/4199321166 <== mon.2 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559697b7c5a0 con 0x559697b51000 2024-04-26T17:47:00.603 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:00 smithi097 bash[33792]: d1c415b7a384d01efce34f8308d3cab5aa73f55051b70d93e40aabdba2e1cdba 2024-04-26T17:47:00.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:00.604+0000 7f468a8b9700 1 -- 172.21.15.31:0/2490882742 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4684063860 con 0x7f466c039890 2024-04-26T17:47:00.879 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:00 smithi097 systemd[1]: Started Ceph mgr.y for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:47:00.879 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:00 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:00.693+0000 7f4992931200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T17:47:00.879 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:00 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:00.879+0000 7f4992931200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:47:00.895 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:00 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:00.598+0000 7f72d76ff200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T17:47:00.895 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:00 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:00.690+0000 7f72d76ff200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T17:47:01.134 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:00 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:00.986+0000 7f4992931200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T17:47:01.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:47:01.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:47:01.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:01.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:01.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:01.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:01.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:01.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:01.266 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:00 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:00.896+0000 7f72d76ff200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T17:47:01.267 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:01 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:01.098+0000 7f72d76ff200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:47:01.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:47:01.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:47:01.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:01.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:01.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:01.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:01.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:01.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:01.322 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:47:01.322 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:47:01.322 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:01.322 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:01.322 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:01.322 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:01.323 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:01.323 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:01.633 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:01 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:01.310+0000 7f4992931200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T17:47:01.634 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:01 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:01.403+0000 7f4992931200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T17:47:02.016 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:01 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:01.742+0000 7f72d76ff200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T17:47:02.072 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:02 smithi097 ceph-mon[32419]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:02.072 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:02 smithi097 ceph-mon[32419]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:47:02.298 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:02 smithi031 ceph-mon[29164]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:02.298 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:02 smithi031 ceph-mon[29164]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:47:02.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:02 smithi186 ceph-mon[32797]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:02.483 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:02 smithi186 ceph-mon[32797]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:47:02.484 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:02 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:02.186+0000 7f72d76ff200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T17:47:02.484 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:02 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:02.281+0000 7f72d76ff200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T17:47:02.484 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:02 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:02.484+0000 7f72d76ff200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T17:47:02.883 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:02 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:02.626+0000 7f4992931200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T17:47:02.884 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:02 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:02.792+0000 7f4992931200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T17:47:03.245 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:02 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:02.960+0000 7f4992931200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T17:47:03.245 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:03.244+0000 7f4992931200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T17:47:03.245 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.245 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:03.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:03.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-mon[32419]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:03.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-mon[32419]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T17:47:03.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T17:47:03.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T17:47:03.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:03.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-mon[32419]: Reconfiguring daemon mgr.x on smithi031 2024-04-26T17:47:03.266 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:03.006+0000 7f72d76ff200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T17:47:03.266 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:03.097+0000 7f72d76ff200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T17:47:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-mon[32797]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-mon[32797]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T17:47:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T17:47:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T17:47:03.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:03.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-mon[32797]: Reconfiguring daemon mgr.x on smithi031 2024-04-26T17:47:03.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:03.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:03.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:03.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:03 smithi031 ceph-mon[29164]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:03.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:03 smithi031 ceph-mon[29164]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T17:47:03.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T17:47:03.537 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T17:47:03.537 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:03.537 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:03 smithi031 ceph-mon[29164]: Reconfiguring daemon mgr.x on smithi031 2024-04-26T17:47:03.537 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:03.337+0000 7f4992931200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T17:47:03.686 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:03.345+0000 7f72d76ff200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T17:47:03.686 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:03.476+0000 7f72d76ff200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T17:47:03.686 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:03.571+0000 7f72d76ff200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T17:47:03.686 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:03.687+0000 7f72d76ff200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T17:47:03.884 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:03.537+0000 7f4992931200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T17:47:03.884 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:03 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:03.728+0000 7f4992931200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:47:04.266 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:03.992+0000 7f72d76ff200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T17:47:04.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:04 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/4035587817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f684ee5-eeee-454a-a00b-e5d6d2901572"}]: dispatch 2024-04-26T17:47:04.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:04 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/4035587817' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f684ee5-eeee-454a-a00b-e5d6d2901572"}]': finished 2024-04-26T17:47:04.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:04 smithi186 ceph-mon[32797]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T17:47:04.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:04 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:04.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:04 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:04.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:04 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1235795345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:47:04.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:04 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/4035587817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f684ee5-eeee-454a-a00b-e5d6d2901572"}]: dispatch 2024-04-26T17:47:04.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:04 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/4035587817' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f684ee5-eeee-454a-a00b-e5d6d2901572"}]': finished 2024-04-26T17:47:04.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:04 smithi031 ceph-mon[29164]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T17:47:04.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:04 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:04.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:04 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:04.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:04 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1235795345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:47:04.633 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:04 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:04.322+0000 7f4992931200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T17:47:04.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:04 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/4035587817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f684ee5-eeee-454a-a00b-e5d6d2901572"}]: dispatch 2024-04-26T17:47:04.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:04 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/4035587817' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f684ee5-eeee-454a-a00b-e5d6d2901572"}]': finished 2024-04-26T17:47:04.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:04 smithi097 ceph-mon[32419]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T17:47:04.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:04 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:04.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:04 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:04.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:04 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1235795345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:47:05.016 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:04 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:04.592+0000 7f72d76ff200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T17:47:05.016 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:47:04 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:47:04.807+0000 7f72d76ff200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T17:47:05.032 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:04 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:04.738+0000 7f4992931200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T17:47:05.032 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:04 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:04.834+0000 7f4992931200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T17:47:05.032 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:05.031+0000 7f4992931200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T17:47:05.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:05.188+0000 7f46817fa700 1 -- 172.21.15.31:0/2490882742 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f467c01d790 con 0x7f4684103070 2024-04-26T17:47:05.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:05 smithi097 ceph-mon[32419]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:05.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:05 smithi097 ceph-mon[32419]: Standby manager daemon z started 2024-04-26T17:47:05.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:05 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.186:0/1558990095' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T17:47:05.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:05 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.186:0/1558990095' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T17:47:05.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:05 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.186:0/1558990095' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T17:47:05.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:05 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.186:0/1558990095' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T17:47:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:05 smithi186 ceph-mon[32797]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:05 smithi186 ceph-mon[32797]: Standby manager daemon z started 2024-04-26T17:47:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:05 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.186:0/1558990095' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T17:47:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:05 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.186:0/1558990095' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T17:47:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:05 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.186:0/1558990095' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T17:47:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:05 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.186:0/1558990095' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T17:47:05.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:05 smithi031 ceph-mon[29164]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:05.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:05 smithi031 ceph-mon[29164]: Standby manager daemon z started 2024-04-26T17:47:05.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:05 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.186:0/1558990095' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T17:47:05.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:05 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.186:0/1558990095' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T17:47:05.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:05 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.186:0/1558990095' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T17:47:05.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:05 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.186:0/1558990095' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T17:47:05.883 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:05.556+0000 7f4992931200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T17:47:05.884 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:05.645+0000 7f4992931200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T17:47:06.185 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:05.883+0000 7f4992931200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T17:47:06.185 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:06 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:06.014+0000 7f4992931200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T17:47:06.185 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:06 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:06.107+0000 7f4992931200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T17:47:06.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:06 smithi186 ceph-mon[32797]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-26T17:47:06.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:06 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T17:47:06.518 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:06 smithi097 ceph-mon[32419]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-26T17:47:06.518 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:06 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T17:47:06.519 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:06 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:06.221+0000 7f4992931200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T17:47:06.519 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:06 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:06.518+0000 7f4992931200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T17:47:06.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:06 smithi031 ceph-mon[29164]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-26T17:47:06.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:06 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T17:47:07.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:07 smithi097 ceph-mon[32419]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:07.384 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:07 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:07.114+0000 7f4992931200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T17:47:07.384 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:47:07 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:47:07.327+0000 7f4992931200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T17:47:07.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:07 smithi186 ceph-mon[32797]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:07.542 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:07 smithi031 ceph-mon[29164]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:08.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:08.200+0000 7f46817fa700 1 -- 172.21.15.31:0/2490882742 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f467c01db10 con 0x7f4684103070 2024-04-26T17:47:08.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:08 smithi186 ceph-mon[32797]: Standby manager daemon y started 2024-04-26T17:47:08.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:08 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.97:0/1172885693' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T17:47:08.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:08 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.97:0/1172885693' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T17:47:08.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:08 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.97:0/1172885693' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T17:47:08.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:08 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.97:0/1172885693' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T17:47:08.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:08 smithi031 ceph-mon[29164]: Standby manager daemon y started 2024-04-26T17:47:08.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:08 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.97:0/1172885693' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T17:47:08.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:08 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.97:0/1172885693' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T17:47:08.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:08 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.97:0/1172885693' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T17:47:08.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:08 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.97:0/1172885693' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T17:47:08.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:08 smithi097 ceph-mon[32419]: Standby manager daemon y started 2024-04-26T17:47:08.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:08 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.97:0/1172885693' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T17:47:08.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:08 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.97:0/1172885693' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T17:47:08.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:08 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.97:0/1172885693' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T17:47:08.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:08 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.97:0/1172885693' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T17:47:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:09 smithi097 ceph-mon[32419]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-26T17:47:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:09 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T17:47:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:09 smithi097 ceph-mon[32419]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:09 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:09 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:09 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:09 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:09 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:09 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:09.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:09 smithi186 ceph-mon[32797]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-26T17:47:09.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:09 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T17:47:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:09 smithi186 ceph-mon[32797]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:09 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:09 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:09 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:09 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:09 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:09 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:09.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:09 smithi031 ceph-mon[29164]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-26T17:47:09.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:09 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T17:47:09.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:09 smithi031 ceph-mon[29164]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:09.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:09 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:09.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:09 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:09.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:09 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:09.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:09 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:09.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:09 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:09.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:09 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:11.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:11 smithi031 ceph-mon[29164]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:11.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T17:47:11.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:11.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:11 smithi031 ceph-mon[29164]: Deploying daemon osd.0 on smithi031 2024-04-26T17:47:11.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:11 smithi097 ceph-mon[32419]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T17:47:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:11 smithi097 ceph-mon[32419]: Deploying daemon osd.0 on smithi031 2024-04-26T17:47:11.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:11 smithi186 ceph-mon[32797]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:11.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T17:47:11.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:11.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:11 smithi186 ceph-mon[32797]: Deploying daemon osd.0 on smithi031 2024-04-26T17:47:13.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:13 smithi031 ceph-mon[29164]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:13.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:13 smithi097 ceph-mon[32419]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:13.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:13 smithi186 ceph-mon[32797]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:15.541 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:15 smithi031 ceph-mon[29164]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:15.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:15 smithi097 ceph-mon[32419]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:15.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:15 smithi186 ceph-mon[32797]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:17.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:17 smithi186 ceph-mon[32797]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:17.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:17.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:17.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:17.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:17 smithi031 ceph-mon[29164]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:17.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:17.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:17.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:17.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:17 smithi097 ceph-mon[32419]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:17.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:17.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:17.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:19.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:19 smithi186 ceph-mon[32797]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:19.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:19.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:19.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:19.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:19.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:19.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:19 smithi031 ceph-mon[29164]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:19.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:19.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:19.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:19.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:19.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:19.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:19 smithi097 ceph-mon[32419]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:19.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:19.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:19.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:19.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:19.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:20.213 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 0 on host 'smithi031' 2024-04-26T17:47:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:20.213+0000 7f46817fa700 1 -- 172.21.15.31:0/2490882742 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4684063860 con 0x7f466c039890 2024-04-26T17:47:20.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:20.215+0000 7f468a8b9700 1 -- 172.21.15.31:0/2490882742 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f466c039890 msgr2=0x7f466c03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:47:20.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:20.215+0000 7f468a8b9700 1 --2- 172.21.15.31:0/2490882742 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f466c039890 0x7f466c03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4674009a10 tx=0x7f4674009190 comp rx=0 tx=0).stop 2024-04-26T17:47:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:20.215+0000 7f468a8b9700 1 -- 172.21.15.31:0/2490882742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4684103070 msgr2=0x7f46841a2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:47:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:20.215+0000 7f468a8b9700 1 --2- 172.21.15.31:0/2490882742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4684103070 0x7f46841a2d80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f467c00b070 tx=0x7f467c004e40 comp rx=0 tx=0).stop 2024-04-26T17:47:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:20.215+0000 7f468a8b9700 1 -- 172.21.15.31:0/2490882742 shutdown_connections 2024-04-26T17:47:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:20.215+0000 7f468a8b9700 1 --2- 172.21.15.31:0/2490882742 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f466c039890 0x7f466c03bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:20.215+0000 7f468a8b9700 1 --2- 172.21.15.31:0/2490882742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4684103070 0x7f46841a2d80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:20.215+0000 7f468a8b9700 1 --2- 172.21.15.31:0/2490882742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4684101d50 0x7f468419b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:20.215+0000 7f468a8b9700 1 --2- 172.21.15.31:0/2490882742 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46841026b0 0x7f468419bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:20.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:20.215+0000 7f468a8b9700 1 -- 172.21.15.31:0/2490882742 >> 172.21.15.31:0/2490882742 conn(0x7f46840fd910 msgr2=0x7f46840ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:47:20.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:20.216+0000 7f468a8b9700 1 -- 172.21.15.31:0/2490882742 shutdown_connections 2024-04-26T17:47:20.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:47:20.216+0000 7f468a8b9700 1 -- 172.21.15.31:0/2490882742 wait complete. 2024-04-26T17:47:20.760 DEBUG:teuthology.orchestra.run.smithi031:osd.0> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.0.service 2024-04-26T17:47:20.762 INFO:tasks.cephadm:Deploying osd.1 on smithi097 with /dev/vg_nvme/lv_4... 2024-04-26T17:47:20.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T17:47:21.000 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T17:47:21.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:21 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:21.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:21 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:21.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:21 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:21.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:21 smithi031 ceph-mon[29164]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:21.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:21 smithi031 ceph-mon[29164]: from='osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T17:47:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:21 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:21 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:21 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:21 smithi097 ceph-mon[32419]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:21 smithi097 ceph-mon[32419]: from='osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T17:47:21.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:21 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:21.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:21 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:21 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:21 smithi186 ceph-mon[32797]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:21 smithi186 ceph-mon[32797]: from='osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T17:47:22.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:22 smithi031 ceph-mon[29164]: from='osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T17:47:22.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:22 smithi031 ceph-mon[29164]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T17:47:22.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:22 smithi031 ceph-mon[29164]: from='osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-26T17:47:22.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:22 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:22.304 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 17:47:22 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-0[41697]: 2024-04-26T17:47:22.196+0000 7f5e5af00700 -1 osd.0 0 waiting for initial osdmap 2024-04-26T17:47:22.304 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 17:47:22 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-0[41697]: 2024-04-26T17:47:22.199+0000 7f5e56cf3700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T17:47:22.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:22 smithi186 ceph-mon[32797]: from='osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T17:47:22.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:22 smithi186 ceph-mon[32797]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T17:47:22.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:22 smithi186 ceph-mon[32797]: from='osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-26T17:47:22.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:22 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:22.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:22 smithi097 ceph-mon[32419]: from='osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T17:47:22.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:22 smithi097 ceph-mon[32419]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T17:47:22.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:22 smithi097 ceph-mon[32419]: from='osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-26T17:47:22.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:22 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:22.740 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T17:47:22.740 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T17:47:22.740 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-04-26T17:47:22.740 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-04-26T17:47:22.740 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114592 s, 915 MB/s 2024-04-26T17:47:22.740 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-04-26T17:47:23.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_4 2024-04-26T17:47:23.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:23 smithi097 ceph-mon[32419]: from='osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-26T17:47:23.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:23 smithi097 ceph-mon[32419]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T17:47:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:23 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:23 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:23 smithi097 ceph-mon[32419]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:23.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:23 smithi031 ceph-mon[29164]: from='osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-26T17:47:23.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:23 smithi031 ceph-mon[29164]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T17:47:23.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:23 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:23.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:23 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:23.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:23 smithi031 ceph-mon[29164]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:23.512 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T17:47:23.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:23 smithi186 ceph-mon[32797]: from='osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-26T17:47:23.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:23 smithi186 ceph-mon[32797]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T17:47:23.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:23 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:23 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:23 smithi186 ceph-mon[32797]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T17:47:24.514 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: purged_snaps scrub starts 2024-04-26T17:47:24.514 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: purged_snaps scrub ok 2024-04-26T17:47:24.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:24.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328] boot 2024-04-26T17:47:24.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T17:47:24.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:24.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: Detected new or changed devices on smithi031 2024-04-26T17:47:24.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:24.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:24.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:47:24.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi031 to 17093M 2024-04-26T17:47:24.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:24.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:24.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:24.516 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:24 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:24.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: purged_snaps scrub starts 2024-04-26T17:47:24.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: purged_snaps scrub ok 2024-04-26T17:47:24.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:24.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328] boot 2024-04-26T17:47:24.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T17:47:24.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:24.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: Detected new or changed devices on smithi031 2024-04-26T17:47:24.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:24.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:24.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:47:24.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi031 to 17093M 2024-04-26T17:47:24.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:24.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:24.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:24.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:24 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:24.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: purged_snaps scrub starts 2024-04-26T17:47:24.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: purged_snaps scrub ok 2024-04-26T17:47:24.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:24.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: osd.0 [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328] boot 2024-04-26T17:47:24.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T17:47:24.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:47:24.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: Detected new or changed devices on smithi031 2024-04-26T17:47:24.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:24.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:24.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:47:24.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi031 to 17093M 2024-04-26T17:47:24.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:24.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:24.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:24.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:24 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:25.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.046+0000 7f034951f700 1 -- 172.21.15.97:0/4180066907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0344103470 msgr2=0x7f0344103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:47:25.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.046+0000 7f034951f700 1 --2- 172.21.15.97:0/4180066907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0344103470 0x7f0344103890 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f03340099c0 tx=0x7f0334009cd0 comp rx=0 tx=0).stop 2024-04-26T17:47:25.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.046+0000 7f034951f700 1 -- 172.21.15.97:0/4180066907 shutdown_connections 2024-04-26T17:47:25.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.046+0000 7f034951f700 1 --2- 172.21.15.97:0/4180066907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0344103470 0x7f0344103890 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:25.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.046+0000 7f034951f700 1 --2- 172.21.15.97:0/4180066907 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0344105010 0x7f034410b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:25.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.046+0000 7f034951f700 1 --2- 172.21.15.97:0/4180066907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0344104670 0x7f0344104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:25.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.046+0000 7f034951f700 1 -- 172.21.15.97:0/4180066907 >> 172.21.15.97:0/4180066907 conn(0x7f03440fea10 msgr2=0x7f0344100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:47:25.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.046+0000 7f034951f700 1 -- 172.21.15.97:0/4180066907 shutdown_connections 2024-04-26T17:47:25.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.046+0000 7f034951f700 1 -- 172.21.15.97:0/4180066907 wait complete. 2024-04-26T17:47:25.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.046+0000 7f034951f700 1 Processor -- start 2024-04-26T17:47:25.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f034951f700 1 -- start start 2024-04-26T17:47:25.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f034951f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0344103470 0x7f0344199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:47:25.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f034951f700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0344104670 0x7f0344199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:47:25.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f034951f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0344105010 0x7f03441a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:47:25.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f034951f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f034410ef10 con 0x7f0344103470 2024-04-26T17:47:25.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f034951f700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f034410ed90 con 0x7f0344104670 2024-04-26T17:47:25.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f034951f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f034410f090 con 0x7f0344105010 2024-04-26T17:47:25.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f03427fc700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0344104670 0x7f0344199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:47:25.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f03427fc700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0344104670 0x7f0344199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38304/0 (socket says 172.21.15.97:38304) 2024-04-26T17:47:25.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f03427fc700 1 -- 172.21.15.97:0/2133024922 learned_addr learned my addr 172.21.15.97:0/2133024922 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-04-26T17:47:25.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f0342ffd700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0344103470 0x7f0344199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:47:25.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f03437fe700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0344105010 0x7f03441a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:47:25.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f03427fc700 1 -- 172.21.15.97:0/2133024922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0344105010 msgr2=0x7f03441a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:47:25.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f03427fc700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0344105010 0x7f03441a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:25.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f03427fc700 1 -- 172.21.15.97:0/2133024922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0344103470 msgr2=0x7f0344199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:47:25.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f03427fc700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0344103470 0x7f0344199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:25.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f03427fc700 1 -- 172.21.15.97:0/2133024922 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03441a1290 con 0x7f0344104670 2024-04-26T17:47:25.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f0342ffd700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0344103470 0x7f0344199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:47:25.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f03427fc700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0344104670 0x7f0344199b70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f033800c920 tx=0x7f033800cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:47:25.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.047+0000 7f032bfff700 1 -- 172.21.15.97:0/2133024922 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0338007b00 con 0x7f0344104670 2024-04-26T17:47:25.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.048+0000 7f034951f700 1 -- 172.21.15.97:0/2133024922 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03440756b0 con 0x7f0344104670 2024-04-26T17:47:25.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.048+0000 7f032bfff700 1 -- 172.21.15.97:0/2133024922 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0338004ce0 con 0x7f0344104670 2024-04-26T17:47:25.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.048+0000 7f032bfff700 1 -- 172.21.15.97:0/2133024922 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03380055f0 con 0x7f0344104670 2024-04-26T17:47:25.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.048+0000 7f034951f700 1 -- 172.21.15.97:0/2133024922 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0344075bf0 con 0x7f0344104670 2024-04-26T17:47:25.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.049+0000 7f034951f700 1 -- 172.21.15.97:0/2133024922 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f034406a110 con 0x7f0344104670 2024-04-26T17:47:25.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.049+0000 7f032bfff700 1 -- 172.21.15.97:0/2133024922 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f0338004750 con 0x7f0344104670 2024-04-26T17:47:25.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.050+0000 7f032bfff700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f032c0a3bb0 0x7f032c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:47:25.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.050+0000 7f032bfff700 1 -- 172.21.15.97:0/2133024922 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f03380ce8e0 con 0x7f0344104670 2024-04-26T17:47:25.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.050+0000 7f0342ffd700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f032c0a3bb0 0x7f032c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:47:25.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.051+0000 7f0342ffd700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f032c0a3bb0 0x7f032c0a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f033400b580 tx=0x7f0334005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:47:25.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.055+0000 7f032bfff700 1 -- 172.21.15.97:0/2133024922 <== mon.2 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0338014030 con 0x7f0344104670 2024-04-26T17:47:25.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:25.264+0000 7f034951f700 1 -- 172.21.15.97:0/2133024922 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0344063860 con 0x7f032c0a3bb0 2024-04-26T17:47:25.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:25 smithi186 ceph-mon[32797]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T17:47:25.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:25 smithi186 ceph-mon[32797]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:25.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:25 smithi031 ceph-mon[29164]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T17:47:25.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:25 smithi031 ceph-mon[29164]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:25.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:25 smithi097 ceph-mon[32419]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T17:47:25.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:25 smithi097 ceph-mon[32419]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:26.497 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:26 smithi097 ceph-mon[32419]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:47:26.498 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:26 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:47:26.498 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:26 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:47:26.498 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:26 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:26.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:26 smithi186 ceph-mon[32797]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:47:26.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:26 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:47:26.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:26 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:47:26.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:26 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:26.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:26 smithi031 ceph-mon[29164]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:47:26.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:47:26.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:47:26.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:27.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:27 smithi097 ceph-mon[32419]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:27.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:27 smithi186 ceph-mon[32797]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:27.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:27 smithi031 ceph-mon[29164]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:28.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:28 smithi186 ceph-mon[32797]: from='client.? 172.21.15.97:0/1663748091' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08be6b26-cb60-41f7-be8b-01a0c93ebafd"}]: dispatch 2024-04-26T17:47:28.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:28 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08be6b26-cb60-41f7-be8b-01a0c93ebafd"}]: dispatch 2024-04-26T17:47:28.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:28 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08be6b26-cb60-41f7-be8b-01a0c93ebafd"}]': finished 2024-04-26T17:47:28.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:28 smithi186 ceph-mon[32797]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T17:47:28.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:28 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:28.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:28 smithi031 ceph-mon[29164]: from='client.? 172.21.15.97:0/1663748091' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08be6b26-cb60-41f7-be8b-01a0c93ebafd"}]: dispatch 2024-04-26T17:47:28.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:28 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08be6b26-cb60-41f7-be8b-01a0c93ebafd"}]: dispatch 2024-04-26T17:47:28.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:28 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08be6b26-cb60-41f7-be8b-01a0c93ebafd"}]': finished 2024-04-26T17:47:28.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:28 smithi031 ceph-mon[29164]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T17:47:28.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:28 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:28.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:28 smithi097 ceph-mon[32419]: from='client.? 172.21.15.97:0/1663748091' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08be6b26-cb60-41f7-be8b-01a0c93ebafd"}]: dispatch 2024-04-26T17:47:28.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:28 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08be6b26-cb60-41f7-be8b-01a0c93ebafd"}]: dispatch 2024-04-26T17:47:28.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:28 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08be6b26-cb60-41f7-be8b-01a0c93ebafd"}]': finished 2024-04-26T17:47:28.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:28 smithi097 ceph-mon[32419]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T17:47:28.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:28 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:29.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:29 smithi186 ceph-mon[32797]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:29.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:29 smithi186 ceph-mon[32797]: from='client.? 172.21.15.97:0/3227068545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:47:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:29 smithi031 ceph-mon[29164]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:29 smithi031 ceph-mon[29164]: from='client.? 172.21.15.97:0/3227068545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:47:29.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:29 smithi097 ceph-mon[32419]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:29 smithi097 ceph-mon[32419]: from='client.? 172.21.15.97:0/3227068545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:47:31.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:31 smithi031 ceph-mon[29164]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:31.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:31 smithi097 ceph-mon[32419]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:31.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:31 smithi186 ceph-mon[32797]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:33.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:33 smithi031 ceph-mon[29164]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:33.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:33 smithi097 ceph-mon[32419]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:33.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:33 smithi186 ceph-mon[32797]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:35.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:35 smithi031 ceph-mon[29164]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:35.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T17:47:35.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:35.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:35 smithi031 ceph-mon[29164]: Deploying daemon osd.1 on smithi097 2024-04-26T17:47:35.561 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:35 smithi097 ceph-mon[32419]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:35.561 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T17:47:35.561 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:35.561 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:35 smithi097 ceph-mon[32419]: Deploying daemon osd.1 on smithi097 2024-04-26T17:47:35.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:35 smithi186 ceph-mon[32797]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:35.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T17:47:35.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:35.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:35 smithi186 ceph-mon[32797]: Deploying daemon osd.1 on smithi097 2024-04-26T17:47:37.549 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:37 smithi097 ceph-mon[32419]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:37.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:37 smithi031 ceph-mon[29164]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:37.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:37 smithi186 ceph-mon[32797]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:39.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:39 smithi031 ceph-mon[29164]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:39.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:39 smithi097 ceph-mon[32419]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:39.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:39 smithi186 ceph-mon[32797]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:41.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:41.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:41.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:41.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:41 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:41.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:41 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:41.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:41 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:41.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:41 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:41 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:41 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:42.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:42 smithi031 ceph-mon[29164]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:42.372 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:42 smithi097 ceph-mon[32419]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:42.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:42 smithi186 ceph-mon[32797]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:43.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:43.577+0000 7f032bfff700 1 -- 172.21.15.97:0/2133024922 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0344063860 con 0x7f032c0a3bb0 2024-04-26T17:47:43.578 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 1 on host 'smithi097' 2024-04-26T17:47:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:43.579+0000 7f034951f700 1 -- 172.21.15.97:0/2133024922 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f032c0a3bb0 msgr2=0x7f032c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:47:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:43.579+0000 7f034951f700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f032c0a3bb0 0x7f032c0a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f033400b580 tx=0x7f0334005fb0 comp rx=0 tx=0).stop 2024-04-26T17:47:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:43.579+0000 7f034951f700 1 -- 172.21.15.97:0/2133024922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0344104670 msgr2=0x7f0344199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:47:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:43.579+0000 7f034951f700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0344104670 0x7f0344199b70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f033800c920 tx=0x7f033800cce0 comp rx=0 tx=0).stop 2024-04-26T17:47:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:43.580+0000 7f034951f700 1 -- 172.21.15.97:0/2133024922 shutdown_connections 2024-04-26T17:47:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:43.580+0000 7f034951f700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f032c0a3bb0 0x7f032c0a6070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:43.580+0000 7f034951f700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0344105010 0x7f03441a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:43.580+0000 7f034951f700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0344103470 0x7f0344199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:43.580+0000 7f034951f700 1 --2- 172.21.15.97:0/2133024922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0344104670 0x7f0344199b70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:43.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:43.580+0000 7f034951f700 1 -- 172.21.15.97:0/2133024922 >> 172.21.15.97:0/2133024922 conn(0x7f03440fea10 msgr2=0x7f0344107a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:47:43.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:43.580+0000 7f034951f700 1 -- 172.21.15.97:0/2133024922 shutdown_connections 2024-04-26T17:47:43.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:47:43.580+0000 7f034951f700 1 -- 172.21.15.97:0/2133024922 wait complete. 2024-04-26T17:47:43.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:43 smithi097 ceph-mon[32419]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:43.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:43 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:43.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:43 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:43.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:43 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:43.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:43 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:43.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:43 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:43.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:43 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:43.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:43 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:43.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:43 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:44.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:43 smithi186 ceph-mon[32797]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:44.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:43 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:44.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:43 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:44.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:43 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:44.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:43 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:44.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:43 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:44.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:43 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:44.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:43 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:44.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:43 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:44.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:43 smithi031 ceph-mon[29164]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:44.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:43 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:44.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:43 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:44.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:43 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:44.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:43 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:44.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:43 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:44.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:43 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:44.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:43 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:44.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:43 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:47:44.191 DEBUG:teuthology.orchestra.run.smithi097:osd.1> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.1.service 2024-04-26T17:47:44.193 INFO:tasks.cephadm:Deploying osd.2 on smithi186 with /dev/vg_nvme/lv_4... 2024-04-26T17:47:44.194 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T17:47:44.437 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T17:47:44.865 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:44 smithi186 ceph-mon[32797]: from='osd.1 [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T17:47:44.865 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:44 smithi186 ceph-mon[32797]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T17:47:45.029 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:44 smithi097 ceph-mon[32419]: from='osd.1 [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T17:47:45.029 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:44 smithi097 ceph-mon[32419]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T17:47:45.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:44 smithi031 ceph-mon[29164]: from='osd.1 [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T17:47:45.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:44 smithi031 ceph-mon[29164]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T17:47:45.830 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:45 smithi186 ceph-mon[32797]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:45.831 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:45 smithi186 ceph-mon[32797]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T17:47:45.831 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:45 smithi186 ceph-mon[32797]: from='osd.1 [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:47:45.831 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:45 smithi186 ceph-mon[32797]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T17:47:45.831 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:45 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:45.831 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:45 smithi186 ceph-mon[32797]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:47:45.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:45 smithi097 ceph-mon[32419]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:45.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:45 smithi097 ceph-mon[32419]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T17:47:45.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:45 smithi097 ceph-mon[32419]: from='osd.1 [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:47:45.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:45 smithi097 ceph-mon[32419]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T17:47:45.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:45 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:45.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:45 smithi097 ceph-mon[32419]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:47:46.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:45 smithi031 ceph-mon[29164]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:46.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:45 smithi031 ceph-mon[29164]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T17:47:46.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:45 smithi031 ceph-mon[29164]: from='osd.1 [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:47:46.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:45 smithi031 ceph-mon[29164]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T17:47:46.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:45 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:46.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:45 smithi031 ceph-mon[29164]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:47:46.207 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T17:47:46.208 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T17:47:46.208 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-26T17:47:46.208 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-26T17:47:46.208 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114125 s, 919 MB/s 2024-04-26T17:47:46.208 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-26T17:47:46.428 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 17:47:46 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-1[37019]: 2024-04-26T17:47:46.168+0000 7f11d5e19700 -1 osd.1 0 waiting for initial osdmap 2024-04-26T17:47:46.428 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 17:47:46 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-1[37019]: 2024-04-26T17:47:46.172+0000 7f11cd403700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T17:47:46.769 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:46 smithi186 ceph-mon[32797]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-26T17:47:46.770 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:46 smithi186 ceph-mon[32797]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T17:47:46.770 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:46.770 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:46.770 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:46 smithi186 ceph-mon[32797]: from='osd.1 ' entity='osd.1' 2024-04-26T17:47:46.770 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:46.871 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_4 2024-04-26T17:47:47.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:46 smithi031 ceph-mon[29164]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-26T17:47:47.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:46 smithi031 ceph-mon[29164]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T17:47:47.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:47.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:47.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:46 smithi031 ceph-mon[29164]: from='osd.1 ' entity='osd.1' 2024-04-26T17:47:47.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:47.062 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:46 smithi097 ceph-mon[32419]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-26T17:47:47.062 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:46 smithi097 ceph-mon[32419]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T17:47:47.062 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:46 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:47.062 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:46 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:47.062 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:46 smithi097 ceph-mon[32419]: from='osd.1 ' entity='osd.1' 2024-04-26T17:47:47.062 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:46 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:47.118 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T17:47:47.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:47 smithi186 ceph-mon[32797]: purged_snaps scrub starts 2024-04-26T17:47:47.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:47 smithi186 ceph-mon[32797]: purged_snaps scrub ok 2024-04-26T17:47:47.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:47 smithi186 ceph-mon[32797]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:47 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:47 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:47 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:47:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:47 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:47 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:47 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:47 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:47 smithi186 ceph-mon[32797]: osd.1 [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427] boot 2024-04-26T17:47:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:47 smithi186 ceph-mon[32797]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T17:47:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:47 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:48.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:47 smithi031 ceph-mon[29164]: purged_snaps scrub starts 2024-04-26T17:47:48.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:47 smithi031 ceph-mon[29164]: purged_snaps scrub ok 2024-04-26T17:47:48.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:47 smithi031 ceph-mon[29164]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:48.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:47 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:48.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:47 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:48.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:47 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:47:48.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:47 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:48.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:47 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:48.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:47 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:48.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:47 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:48.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:47 smithi031 ceph-mon[29164]: osd.1 [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427] boot 2024-04-26T17:47:48.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:47 smithi031 ceph-mon[29164]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T17:47:48.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:47 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:48.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:47 smithi097 ceph-mon[32419]: purged_snaps scrub starts 2024-04-26T17:47:48.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:47 smithi097 ceph-mon[32419]: purged_snaps scrub ok 2024-04-26T17:47:48.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:47 smithi097 ceph-mon[32419]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T17:47:48.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:47 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:48.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:47 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:48.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:47 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:47:48.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:47 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:48.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:47 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:48.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:47 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:47:48.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:47 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:47:48.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:47 smithi097 ceph-mon[32419]: osd.1 [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427] boot 2024-04-26T17:47:48.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:47 smithi097 ceph-mon[32419]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T17:47:48.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:47 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:47:48.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.673+0000 7fb7a047d700 1 -- 172.21.15.186:0/1930765134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798101510 msgr2=0x7fb798101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:47:48.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.673+0000 7fb7a047d700 1 --2- 172.21.15.186:0/1930765134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798101510 0x7fb798101970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb788009990 tx=0x7fb788009ca0 comp rx=0 tx=0).stop 2024-04-26T17:47:48.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.673+0000 7fb7a047d700 1 -- 172.21.15.186:0/1930765134 shutdown_connections 2024-04-26T17:47:48.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.673+0000 7fb7a047d700 1 --2- 172.21.15.186:0/1930765134 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb798104a90 0x7fb798104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:48.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.673+0000 7fb7a047d700 1 --2- 172.21.15.186:0/1930765134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb798101eb0 0x7fb79810ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:48.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.673+0000 7fb7a047d700 1 --2- 172.21.15.186:0/1930765134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798101510 0x7fb798101970 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:48.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.673+0000 7fb7a047d700 1 -- 172.21.15.186:0/1930765134 >> 172.21.15.186:0/1930765134 conn(0x7fb7980fd0b0 msgr2=0x7fb7980ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:47:48.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.673+0000 7fb7a047d700 1 -- 172.21.15.186:0/1930765134 shutdown_connections 2024-04-26T17:47:48.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.673+0000 7fb7a047d700 1 -- 172.21.15.186:0/1930765134 wait complete. 2024-04-26T17:47:48.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb7a047d700 1 Processor -- start 2024-04-26T17:47:48.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb7a047d700 1 -- start start 2024-04-26T17:47:48.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb7a047d700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb798101510 0x7fb7981995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:47:48.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb7a047d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb798101eb0 0x7fb798199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:47:48.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb7a047d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798104a90 0x7fb7981a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:47:48.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb7a047d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb79810ef80 con 0x7fb798101eb0 2024-04-26T17:47:48.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb7a047d700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb79810ee00 con 0x7fb798101510 2024-04-26T17:47:48.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb7a047d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb79810f100 con 0x7fb798104a90 2024-04-26T17:47:48.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb79ea1a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798104a90 0x7fb7981a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:47:48.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb79ea1a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798104a90 0x7fb7981a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:47736/0 (socket says 172.21.15.186:47736) 2024-04-26T17:47:48.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb79ea1a700 1 -- 172.21.15.186:0/1246682970 learned_addr learned my addr 172.21.15.186:0/1246682970 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T17:47:48.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb79da18700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb798101eb0 0x7fb798199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:47:48.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb79e219700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb798101510 0x7fb7981995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:47:48.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb79ea1a700 1 -- 172.21.15.186:0/1246682970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb798101510 msgr2=0x7fb7981995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:47:48.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb79ea1a700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb798101510 0x7fb7981995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:48.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb79ea1a700 1 -- 172.21.15.186:0/1246682970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb798101eb0 msgr2=0x7fb798199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:47:48.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb79ea1a700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb798101eb0 0x7fb798199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:47:48.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb79ea1a700 1 -- 172.21.15.186:0/1246682970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7981a1250 con 0x7fb798104a90 2024-04-26T17:47:48.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb79e219700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb798101510 0x7fb7981995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:47:48.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.674+0000 7fb79da18700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb798101eb0 0x7fb798199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:47:48.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.675+0000 7fb79ea1a700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798104a90 0x7fb7981a0bb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb79400add0 tx=0x7fb79400c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:47:48.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.675+0000 7fb78f7fe700 1 -- 172.21.15.186:0/1246682970 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb794017020 con 0x7fb798104a90 2024-04-26T17:47:48.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.675+0000 7fb78f7fe700 1 -- 172.21.15.186:0/1246682970 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb79400cdf0 con 0x7fb798104a90 2024-04-26T17:47:48.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.675+0000 7fb7a047d700 1 -- 172.21.15.186:0/1246682970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7981a1540 con 0x7fb798104a90 2024-04-26T17:47:48.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.675+0000 7fb78f7fe700 1 -- 172.21.15.186:0/1246682970 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb794005460 con 0x7fb798104a90 2024-04-26T17:47:48.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.675+0000 7fb7a047d700 1 -- 172.21.15.186:0/1246682970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7981a1ab0 con 0x7fb798104a90 2024-04-26T17:47:48.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.676+0000 7fb7a047d700 1 -- 172.21.15.186:0/1246682970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb79806a110 con 0x7fb798104a90 2024-04-26T17:47:48.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.678+0000 7fb78f7fe700 1 -- 172.21.15.186:0/1246682970 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fb794004ad0 con 0x7fb798104a90 2024-04-26T17:47:48.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.678+0000 7fb78f7fe700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb7840a3bb0 0x7fb7840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:47:48.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.678+0000 7fb78f7fe700 1 -- 172.21.15.186:0/1246682970 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7fb7940ce9f0 con 0x7fb798104a90 2024-04-26T17:47:48.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.678+0000 7fb79e219700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb7840a3bb0 0x7fb7840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:47:48.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.679+0000 7fb79e219700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb7840a3bb0 0x7fb7840a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb79806ade0 tx=0x7fb790008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:47:48.681 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.682+0000 7fb78f7fe700 1 -- 172.21.15.186:0/1246682970 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb794021020 con 0x7fb798104a90 2024-04-26T17:47:48.889 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:47:48.890+0000 7fb7a047d700 1 -- 172.21.15.186:0/1246682970 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb798063860 con 0x7fb7840a3bb0 2024-04-26T17:47:49.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:48 smithi186 ceph-mon[32797]: Detected new or changed devices on smithi097 2024-04-26T17:47:49.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:48 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi097 to 17088M 2024-04-26T17:47:49.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:48 smithi031 ceph-mon[29164]: Detected new or changed devices on smithi097 2024-04-26T17:47:49.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:48 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi097 to 17088M 2024-04-26T17:47:49.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:48 smithi097 ceph-mon[32419]: Detected new or changed devices on smithi097 2024-04-26T17:47:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:48 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi097 to 17088M 2024-04-26T17:47:50.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:49 smithi186 ceph-mon[32797]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:49 smithi186 ceph-mon[32797]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T17:47:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:49 smithi186 ceph-mon[32797]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:47:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:47:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:47:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:50.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:49 smithi031 ceph-mon[29164]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:50.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:49 smithi031 ceph-mon[29164]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T17:47:50.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:49 smithi031 ceph-mon[29164]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:47:50.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:47:50.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:47:50.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:49 smithi097 ceph-mon[32419]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:49 smithi097 ceph-mon[32419]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T17:47:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:49 smithi097 ceph-mon[32419]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:47:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:47:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:47:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:52.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:51 smithi186 ceph-mon[32797]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:52.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:51 smithi186 ceph-mon[32797]: from='client.? 172.21.15.186:0/499244855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "498216b1-a0ac-4923-ade8-ce50fb97d472"}]: dispatch 2024-04-26T17:47:52.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:51 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "498216b1-a0ac-4923-ade8-ce50fb97d472"}]: dispatch 2024-04-26T17:47:52.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:51 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "498216b1-a0ac-4923-ade8-ce50fb97d472"}]': finished 2024-04-26T17:47:52.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:51 smithi186 ceph-mon[32797]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T17:47:52.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:51 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:47:52.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:51 smithi031 ceph-mon[29164]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:52.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:51 smithi031 ceph-mon[29164]: from='client.? 172.21.15.186:0/499244855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "498216b1-a0ac-4923-ade8-ce50fb97d472"}]: dispatch 2024-04-26T17:47:52.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:51 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "498216b1-a0ac-4923-ade8-ce50fb97d472"}]: dispatch 2024-04-26T17:47:52.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:51 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "498216b1-a0ac-4923-ade8-ce50fb97d472"}]': finished 2024-04-26T17:47:52.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:51 smithi031 ceph-mon[29164]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T17:47:52.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:51 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:47:52.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:51 smithi097 ceph-mon[32419]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:51 smithi097 ceph-mon[32419]: from='client.? 172.21.15.186:0/499244855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "498216b1-a0ac-4923-ade8-ce50fb97d472"}]: dispatch 2024-04-26T17:47:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:51 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "498216b1-a0ac-4923-ade8-ce50fb97d472"}]: dispatch 2024-04-26T17:47:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:51 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "498216b1-a0ac-4923-ade8-ce50fb97d472"}]': finished 2024-04-26T17:47:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:51 smithi097 ceph-mon[32419]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T17:47:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:51 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:47:53.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:52 smithi031 ceph-mon[29164]: from='client.? 172.21.15.186:0/3265967944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:47:53.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:52 smithi097 ceph-mon[32419]: from='client.? 172.21.15.186:0/3265967944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:47:53.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:52 smithi186 ceph-mon[32797]: from='client.? 172.21.15.186:0/3265967944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:47:54.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:53 smithi031 ceph-mon[29164]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:54.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:53 smithi097 ceph-mon[32419]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:54.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:53 smithi186 ceph-mon[32797]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:56.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:55 smithi031 ceph-mon[29164]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:56.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:55 smithi097 ceph-mon[32419]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:56.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:55 smithi186 ceph-mon[32797]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:57.907 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:57 smithi186 ceph-mon[32797]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:58.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:57 smithi031 ceph-mon[29164]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:58.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:57 smithi097 ceph-mon[32419]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:47:59.035 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T17:47:59.035 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:59.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T17:47:59.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:47:59.178 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T17:47:59.178 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:00.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:59 smithi031 ceph-mon[29164]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:00.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:47:59 smithi031 ceph-mon[29164]: Deploying daemon osd.2 on smithi186 2024-04-26T17:48:00.102 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:59 smithi186 ceph-mon[32797]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:00.102 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:47:59 smithi186 ceph-mon[32797]: Deploying daemon osd.2 on smithi186 2024-04-26T17:48:00.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:59 smithi097 ceph-mon[32419]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:00.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:47:59 smithi097 ceph-mon[32419]: Deploying daemon osd.2 on smithi186 2024-04-26T17:48:02.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:01 smithi186 ceph-mon[32797]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:02.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:01 smithi031 ceph-mon[29164]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:01 smithi097 ceph-mon[32419]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:04.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:03 smithi186 ceph-mon[32797]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:04.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:03 smithi031 ceph-mon[29164]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:04.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:03 smithi097 ceph-mon[32419]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:05.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:04 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:05.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:04 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:05.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:04 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:05.203 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:04 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:05.203 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:04 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:05.203 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:04 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:05.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:04 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:05.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:04 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:05.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:04 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:06.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:05 smithi186 ceph-mon[32797]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:06.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:05 smithi097 ceph-mon[32419]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:06.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:05 smithi031 ceph-mon[29164]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:07.458 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:48:07.458+0000 7fb78f7fe700 1 -- 172.21.15.186:0/1246682970 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb798063860 con 0x7fb7840a3bb0 2024-04-26T17:48:07.459 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 2 on host 'smithi186' 2024-04-26T17:48:07.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:48:07.460+0000 7fb7a047d700 1 -- 172.21.15.186:0/1246682970 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb7840a3bb0 msgr2=0x7fb7840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:48:07.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:48:07.460+0000 7fb7a047d700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb7840a3bb0 0x7fb7840a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb79806ade0 tx=0x7fb790008040 comp rx=0 tx=0).stop 2024-04-26T17:48:07.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:48:07.460+0000 7fb7a047d700 1 -- 172.21.15.186:0/1246682970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798104a90 msgr2=0x7fb7981a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:48:07.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:48:07.460+0000 7fb7a047d700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798104a90 0x7fb7981a0bb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb79400add0 tx=0x7fb79400c330 comp rx=0 tx=0).stop 2024-04-26T17:48:07.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:48:07.460+0000 7fb7a047d700 1 -- 172.21.15.186:0/1246682970 shutdown_connections 2024-04-26T17:48:07.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:48:07.460+0000 7fb7a047d700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb7840a3bb0 0x7fb7840a6070 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:07.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:48:07.460+0000 7fb7a047d700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb798101510 0x7fb7981995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:07.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:48:07.460+0000 7fb7a047d700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb798104a90 0x7fb7981a0bb0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:07.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:48:07.460+0000 7fb7a047d700 1 --2- 172.21.15.186:0/1246682970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb798101eb0 0x7fb798199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:07.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:48:07.460+0000 7fb7a047d700 1 -- 172.21.15.186:0/1246682970 >> 172.21.15.186:0/1246682970 conn(0x7fb7980fd0b0 msgr2=0x7fb798107bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:48:07.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:48:07.461+0000 7fb7a047d700 1 -- 172.21.15.186:0/1246682970 shutdown_connections 2024-04-26T17:48:07.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:48:07.461+0000 7fb7a047d700 1 -- 172.21.15.186:0/1246682970 wait complete. 2024-04-26T17:48:07.740 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:07 smithi186 ceph-mon[32797]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:07.740 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:07 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.740 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:07 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.740 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:07 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:07.740 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:07 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:07.740 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:07 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.740 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:07 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.740 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:07 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.740 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:07 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:07.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:07 smithi031 ceph-mon[29164]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:07.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:07 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:07 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:07 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:07.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:07 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:07.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:07 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:07 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:07 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:07 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:07 smithi097 ceph-mon[32419]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:07 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:07 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:07 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:07 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:07 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:07 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:07 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:07 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:08.025 DEBUG:teuthology.orchestra.run.smithi186:osd.2> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.2.service 2024-04-26T17:48:08.027 INFO:tasks.cephadm:Deploying osd.3 on smithi031 with /dev/vg_nvme/lv_3... 2024-04-26T17:48:08.028 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T17:48:08.284 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:48:08.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:08 smithi031 ceph-mon[29164]: from='osd.2 [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T17:48:08.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:08 smithi031 ceph-mon[29164]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T17:48:08.881 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:08 smithi186 ceph-mon[32797]: from='osd.2 [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T17:48:08.881 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:08 smithi186 ceph-mon[32797]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T17:48:08.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:08 smithi097 ceph-mon[32419]: from='osd.2 [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T17:48:08.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:08 smithi097 ceph-mon[32419]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T17:48:09.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:09 smithi031 ceph-mon[29164]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:09.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:09 smithi031 ceph-mon[29164]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T17:48:09.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:09 smithi031 ceph-mon[29164]: from='osd.2 [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:48:09.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:09 smithi031 ceph-mon[29164]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T17:48:09.706 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:09 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:09.706 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:09 smithi031 ceph-mon[29164]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:48:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:09 smithi186 ceph-mon[32797]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:09 smithi186 ceph-mon[32797]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T17:48:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:09 smithi186 ceph-mon[32797]: from='osd.2 [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:48:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:09 smithi186 ceph-mon[32797]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T17:48:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:09 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:09 smithi186 ceph-mon[32797]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:48:09.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:09 smithi097 ceph-mon[32419]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:09.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:09 smithi097 ceph-mon[32419]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T17:48:09.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:09 smithi097 ceph-mon[32419]: from='osd.2 [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:48:09.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:09 smithi097 ceph-mon[32419]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T17:48:09.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:09 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:09.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:09 smithi097 ceph-mon[32419]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:48:10.267 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 17:48:10 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-2[37964]: 2024-04-26T17:48:10.024+0000 7f161f943700 -1 osd.2 0 waiting for initial osdmap 2024-04-26T17:48:10.267 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 17:48:10 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-2[37964]: 2024-04-26T17:48:10.028+0000 7f161772e700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T17:48:10.273 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T17:48:10.273 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T17:48:10.273 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-04-26T17:48:10.273 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-04-26T17:48:10.273 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115117 s, 911 MB/s 2024-04-26T17:48:10.273 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-04-26T17:48:10.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:10 smithi031 ceph-mon[29164]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T17:48:10.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:10 smithi031 ceph-mon[29164]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T17:48:10.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:10 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:10.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:10 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:10.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:10 smithi031 ceph-mon[29164]: from='osd.2 ' entity='osd.2' 2024-04-26T17:48:10.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:10 smithi097 ceph-mon[32419]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T17:48:10.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:10 smithi097 ceph-mon[32419]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T17:48:10.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:10 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:10.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:10 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:10.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:10 smithi097 ceph-mon[32419]: from='osd.2 ' entity='osd.2' 2024-04-26T17:48:10.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:10 smithi186 ceph-mon[32797]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T17:48:10.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:10 smithi186 ceph-mon[32797]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T17:48:10.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:10 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:10.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:10 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:10.972 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:10 smithi186 ceph-mon[32797]: from='osd.2 ' entity='osd.2' 2024-04-26T17:48:10.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_3 2024-04-26T17:48:11.235 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:48:11.597 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: purged_snaps scrub starts 2024-04-26T17:48:11.597 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: purged_snaps scrub ok 2024-04-26T17:48:11.597 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:11.597 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:11.597 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: osd.2 [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147] boot 2024-04-26T17:48:11.597 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T17:48:11.597 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:11.597 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: Detected new or changed devices on smithi186 2024-04-26T17:48:11.597 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:11.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:11.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:48:11.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi186 to 17088M 2024-04-26T17:48:11.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:11.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:11.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:11.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:11.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: purged_snaps scrub starts 2024-04-26T17:48:11.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: purged_snaps scrub ok 2024-04-26T17:48:11.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:11.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:11.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: osd.2 [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147] boot 2024-04-26T17:48:11.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T17:48:11.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:11.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: Detected new or changed devices on smithi186 2024-04-26T17:48:11.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:11.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:11.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:48:11.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi186 to 17088M 2024-04-26T17:48:11.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:11.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:11.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:11.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:12.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: purged_snaps scrub starts 2024-04-26T17:48:12.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: purged_snaps scrub ok 2024-04-26T17:48:12.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T17:48:12.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:12.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: osd.2 [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147] boot 2024-04-26T17:48:12.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T17:48:12.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:48:12.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: Detected new or changed devices on smithi186 2024-04-26T17:48:12.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:12.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:12.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:48:12.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi186 to 17088M 2024-04-26T17:48:12.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:12.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:12.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:12.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:12.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:12 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T17:48:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.870+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/2091089051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2c0103480 msgr2=0x7fa2c01038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:48:12.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.870+0000 7fa2c5cdd700 1 --2- 172.21.15.31:0/2091089051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2c0103480 0x7fa2c01038a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00099c0 tx=0x7fa2b0009cd0 comp rx=0 tx=0).stop 2024-04-26T17:48:12.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.870+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/2091089051 shutdown_connections 2024-04-26T17:48:12.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.870+0000 7fa2c5cdd700 1 --2- 172.21.15.31:0/2091089051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2c0105020 0x7fa2c010b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:12.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.870+0000 7fa2c5cdd700 1 --2- 172.21.15.31:0/2091089051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2c0103480 0x7fa2c01038a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:12.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.870+0000 7fa2c5cdd700 1 --2- 172.21.15.31:0/2091089051 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c0104680 0x7fa2c0104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:12.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.870+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/2091089051 >> 172.21.15.31:0/2091089051 conn(0x7fa2c00fea20 msgr2=0x7fa2c0100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:48:12.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.870+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/2091089051 shutdown_connections 2024-04-26T17:48:12.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.870+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/2091089051 wait complete. 2024-04-26T17:48:12.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.871+0000 7fa2c5cdd700 1 Processor -- start 2024-04-26T17:48:12.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.871+0000 7fa2c5cdd700 1 -- start start 2024-04-26T17:48:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.871+0000 7fa2c5cdd700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c0103480 0x7fa2c01995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:48:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2c5cdd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2c0104680 0x7fa2c0199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:48:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2c5cdd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2c0105020 0x7fa2c01a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:48:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2c5cdd700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c010ef20 con 0x7fa2c0105020 2024-04-26T17:48:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2c5cdd700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c010eda0 con 0x7fa2c0103480 2024-04-26T17:48:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2c5cdd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c010f0a0 con 0x7fa2c0104680 2024-04-26T17:48:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2bffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2c0105020 0x7fa2c01a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:48:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2bffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2c0105020 0x7fa2c01a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56466/0 (socket says 172.21.15.31:56466) 2024-04-26T17:48:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2bffff700 1 -- 172.21.15.31:0/1194929178 learned_addr learned my addr 172.21.15.31:0/1194929178 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:48:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2beffd700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2c0104680 0x7fa2c0199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:48:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2bf7fe700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c0103480 0x7fa2c01995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:48:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2bffff700 1 -- 172.21.15.31:0/1194929178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2c0104680 msgr2=0x7fa2c0199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:48:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2bffff700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2c0104680 0x7fa2c0199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2bffff700 1 -- 172.21.15.31:0/1194929178 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c0103480 msgr2=0x7fa2c01995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:48:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2bffff700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c0103480 0x7fa2c01995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2bffff700 1 -- 172.21.15.31:0/1194929178 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2c01a1220 con 0x7fa2c0105020 2024-04-26T17:48:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2bf7fe700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c0103480 0x7fa2c01995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:48:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2bffff700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2c0105020 0x7fa2c01a0b80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa2b800ef60 tx=0x7fa2b800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:48:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2bcff9700 1 -- 172.21.15.31:0/1194929178 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b80092e0 con 0x7fa2c0105020 2024-04-26T17:48:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/1194929178 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2c01a14b0 con 0x7fa2c0105020 2024-04-26T17:48:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2bcff9700 1 -- 172.21.15.31:0/1194929178 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2b8004bb0 con 0x7fa2c0105020 2024-04-26T17:48:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2bcff9700 1 -- 172.21.15.31:0/1194929178 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8004280 con 0x7fa2c0105020 2024-04-26T17:48:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.872+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/1194929178 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2c01a1a40 con 0x7fa2c0105020 2024-04-26T17:48:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.873+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/1194929178 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2c006a110 con 0x7fa2c0105020 2024-04-26T17:48:12.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.875+0000 7fa2bcff9700 1 -- 172.21.15.31:0/1194929178 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fa2b801b030 con 0x7fa2c0105020 2024-04-26T17:48:12.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.875+0000 7fa2bcff9700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa2a80a3bb0 0x7fa2a80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:48:12.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.876+0000 7fa2bf7fe700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa2a80a3bb0 0x7fa2a80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:48:12.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.876+0000 7fa2bcff9700 1 -- 172.21.15.31:0/1194929178 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7fa2b80cf3e0 con 0x7fa2c0105020 2024-04-26T17:48:12.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.876+0000 7fa2bf7fe700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa2a80a3bb0 0x7fa2a80a6070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0006010 tx=0x7fa2b0005c20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:48:12.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:12.879+0000 7fa2bcff9700 1 -- 172.21.15.31:0/1194929178 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa2b8022020 con 0x7fa2c0105020 2024-04-26T17:48:12.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:12 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T17:48:13.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:12 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T17:48:13.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:13.097+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/1194929178 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa2c0063860 con 0x7fa2a80a3bb0 2024-04-26T17:48:13.840 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41860]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T17:48:13.840 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41860]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T17:48:13.840 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41860]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T17:48:13.840 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41860]: pam_unix(sudo:session): session closed for user root 2024-04-26T17:48:13.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 ceph-mon[32797]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:13.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T17:48:13.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 ceph-mon[32797]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T17:48:13.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T17:48:13.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:48:13.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:48:13.841 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:13.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 ceph-mon[29164]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:13.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T17:48:13.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 ceph-mon[29164]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T17:48:13.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T17:48:13.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:48:13.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:48:13.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:13.874 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46234]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T17:48:13.874 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46234]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T17:48:13.874 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46234]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T17:48:13.874 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46234]: pam_unix(sudo:session): session closed for user root 2024-04-26T17:48:13.874 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46237]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T17:48:13.874 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46237]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T17:48:13.874 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46237]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T17:48:13.875 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46237]: pam_unix(sudo:session): session closed for user root 2024-04-26T17:48:13.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 ceph-mon[32419]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:13.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T17:48:13.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 ceph-mon[32419]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T17:48:13.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T17:48:13.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:48:13.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:48:13.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:13.884 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41010]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T17:48:13.884 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41010]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T17:48:13.884 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41010]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T17:48:13.884 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41010]: pam_unix(sudo:session): session closed for user root 2024-04-26T17:48:13.884 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41013]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T17:48:13.884 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41013]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T17:48:13.885 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41013]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T17:48:13.885 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41013]: pam_unix(sudo:session): session closed for user root 2024-04-26T17:48:14.266 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41863]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T17:48:14.267 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41863]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T17:48:14.267 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41863]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T17:48:14.267 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41863]: pam_unix(sudo:session): session closed for user root 2024-04-26T17:48:14.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41866]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T17:48:14.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41866]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T17:48:14.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41866]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T17:48:14.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41866]: pam_unix(sudo:session): session closed for user root 2024-04-26T17:48:14.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41869]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T17:48:14.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41869]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T17:48:14.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41869]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T17:48:14.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:13 smithi186 sudo[41869]: pam_unix(sudo:session): session closed for user root 2024-04-26T17:48:14.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46245]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T17:48:14.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46245]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T17:48:14.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46245]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T17:48:14.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46245]: pam_unix(sudo:session): session closed for user root 2024-04-26T17:48:14.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46248]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T17:48:14.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46248]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T17:48:14.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46248]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T17:48:14.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:13 smithi031 sudo[46248]: pam_unix(sudo:session): session closed for user root 2024-04-26T17:48:14.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41016]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T17:48:14.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41016]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T17:48:14.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41016]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T17:48:14.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41016]: pam_unix(sudo:session): session closed for user root 2024-04-26T17:48:14.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41019]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T17:48:14.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41019]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T17:48:14.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41019]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T17:48:14.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:13 smithi097 sudo[41019]: pam_unix(sudo:session): session closed for user root 2024-04-26T17:48:14.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:48:14.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T17:48:14.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T17:48:14.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:14.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:14.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:14.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T17:48:14.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T17:48:14.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:48:14.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:48:14.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:48:14.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T17:48:14.762 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:48:14.762 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:48:14.762 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:48:14.762 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T17:48:14.762 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T17:48:14.762 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:48:14.762 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:48:14.762 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:48:14.762 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:14 smithi031 ceph-mon[29164]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T17:48:14.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:48:14.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T17:48:14.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T17:48:14.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:14.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:14.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:14.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T17:48:14.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T17:48:14.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:48:14.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:48:14.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:48:14.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T17:48:14.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:48:14.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:48:14.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:48:14.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T17:48:14.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T17:48:14.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:48:14.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:48:14.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:48:14.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:14 smithi097 ceph-mon[32419]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T17:48:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:14.889+0000 7fa2bcff9700 1 -- 172.21.15.31:0/1194929178 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa2b8012050 con 0x7fa2c0105020 2024-04-26T17:48:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:48:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T17:48:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T17:48:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T17:48:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T17:48:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:48:15.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:48:15.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:48:15.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T17:48:15.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:48:15.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:48:15.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:48:15.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T17:48:15.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T17:48:15.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:48:15.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:48:15.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:48:15.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:14 smithi186 ceph-mon[32797]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T17:48:15.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:15 smithi097 ceph-mon[32419]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:15.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:15 smithi097 ceph-mon[32419]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T17:48:15.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:15 smithi097 ceph-mon[32419]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T17:48:16.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:15 smithi186 ceph-mon[32797]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:16.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:15 smithi186 ceph-mon[32797]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T17:48:16.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:15 smithi186 ceph-mon[32797]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T17:48:16.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:15 smithi031 ceph-mon[29164]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:16.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:15 smithi031 ceph-mon[29164]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T17:48:16.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:15 smithi031 ceph-mon[29164]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T17:48:16.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:16 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3492203570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "568a352d-a430-4f0f-9a49-6f992e0ceae1"}]: dispatch 2024-04-26T17:48:16.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:16 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3492203570' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "568a352d-a430-4f0f-9a49-6f992e0ceae1"}]': finished 2024-04-26T17:48:16.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:16 smithi097 ceph-mon[32419]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T17:48:16.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:16 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:17.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:16 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3492203570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "568a352d-a430-4f0f-9a49-6f992e0ceae1"}]: dispatch 2024-04-26T17:48:17.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:16 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3492203570' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "568a352d-a430-4f0f-9a49-6f992e0ceae1"}]': finished 2024-04-26T17:48:17.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:16 smithi186 ceph-mon[32797]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T17:48:17.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:16 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:17.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:16 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3492203570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "568a352d-a430-4f0f-9a49-6f992e0ceae1"}]: dispatch 2024-04-26T17:48:17.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:16 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3492203570' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "568a352d-a430-4f0f-9a49-6f992e0ceae1"}]': finished 2024-04-26T17:48:17.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:16 smithi031 ceph-mon[29164]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T17:48:17.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:17.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:17 smithi097 ceph-mon[32419]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:17.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:17 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1596636008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:48:18.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:17 smithi186 ceph-mon[32797]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:18.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:17 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1596636008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:48:18.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:17 smithi031 ceph-mon[29164]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:18.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:17 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1596636008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:48:19.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:19 smithi097 ceph-mon[32419]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:19.967 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:19 smithi031 ceph-mon[29164]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:20.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:19 smithi186 ceph-mon[32797]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:21.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:21 smithi031 ceph-mon[29164]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:21.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:21 smithi097 ceph-mon[32419]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:22.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:21 smithi186 ceph-mon[32797]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:23.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:23 smithi031 ceph-mon[29164]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:23.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:23 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T17:48:23.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:23 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:23.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:23 smithi097 ceph-mon[32419]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:23.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:23 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T17:48:23.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:23 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:24.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:23 smithi186 ceph-mon[32797]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:24.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:23 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T17:48:24.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:23 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:24.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:24 smithi031 ceph-mon[29164]: Deploying daemon osd.3 on smithi031 2024-04-26T17:48:24.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:24 smithi097 ceph-mon[32419]: Deploying daemon osd.3 on smithi031 2024-04-26T17:48:25.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:24 smithi186 ceph-mon[32797]: Deploying daemon osd.3 on smithi031 2024-04-26T17:48:25.696 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:25 smithi031 ceph-mon[29164]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:25.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:25 smithi097 ceph-mon[32419]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:26.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:25 smithi186 ceph-mon[32797]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:27.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:27 smithi097 ceph-mon[32419]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:28.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:27 smithi186 ceph-mon[32797]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:28.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:27 smithi031 ceph-mon[29164]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:29 smithi031 ceph-mon[29164]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:29 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:29 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:29 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:29.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:29 smithi097 ceph-mon[32419]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:29.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:29 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:29.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:29 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:29.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:29 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:30.015 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:29 smithi186 ceph-mon[32797]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:30.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:29 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:30.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:29 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:30.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:29 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:31.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:31 smithi031 ceph-mon[29164]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:31 smithi097 ceph-mon[32419]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:32.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:31 smithi186 ceph-mon[32797]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:33.411 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 3 on host 'smithi031' 2024-04-26T17:48:33.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:33.410+0000 7fa2bcff9700 1 -- 172.21.15.31:0/1194929178 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa2c0063860 con 0x7fa2a80a3bb0 2024-04-26T17:48:33.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:33.413+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/1194929178 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa2a80a3bb0 msgr2=0x7fa2a80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:48:33.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:33.413+0000 7fa2c5cdd700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa2a80a3bb0 0x7fa2a80a6070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0006010 tx=0x7fa2b0005c20 comp rx=0 tx=0).stop 2024-04-26T17:48:33.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:33.413+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/1194929178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2c0105020 msgr2=0x7fa2c01a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:48:33.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:33.413+0000 7fa2c5cdd700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2c0105020 0x7fa2c01a0b80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa2b800ef60 tx=0x7fa2b800ef90 comp rx=0 tx=0).stop 2024-04-26T17:48:33.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:33.413+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/1194929178 shutdown_connections 2024-04-26T17:48:33.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:33.413+0000 7fa2c5cdd700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa2a80a3bb0 0x7fa2a80a6070 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:33.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:33.413+0000 7fa2c5cdd700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2c0103480 0x7fa2c01995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:33.413+0000 7fa2c5cdd700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2c0105020 0x7fa2c01a0b80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:33.413+0000 7fa2c5cdd700 1 --2- 172.21.15.31:0/1194929178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2c0104680 0x7fa2c0199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:33.413+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/1194929178 >> 172.21.15.31:0/1194929178 conn(0x7fa2c00fea20 msgr2=0x7fa2c0107aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:48:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:33.413+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/1194929178 shutdown_connections 2024-04-26T17:48:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:48:33.413+0000 7fa2c5cdd700 1 -- 172.21.15.31:0/1194929178 wait complete. 2024-04-26T17:48:33.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:33 smithi186 ceph-mon[32797]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:33 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:33 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:33 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:33 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:33 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:33 smithi186 ceph-mon[32797]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T17:48:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:33 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:33 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:33 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:33.780 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:33 smithi031 ceph-mon[29164]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:33.780 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:33 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.780 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:33 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.780 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:33 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:33.780 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:33 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:33.780 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:33 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.780 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:33 smithi031 ceph-mon[29164]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T17:48:33.780 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:33 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.780 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:33 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.780 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:33 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:33 smithi097 ceph-mon[32419]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:33 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:33 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:33 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:33 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:33 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:33 smithi097 ceph-mon[32419]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T17:48:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:33 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:33 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:33 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:34.101 DEBUG:teuthology.orchestra.run.smithi031:osd.3> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.3.service 2024-04-26T17:48:34.104 INFO:tasks.cephadm:Deploying osd.4 on smithi097 with /dev/vg_nvme/lv_3... 2024-04-26T17:48:34.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T17:48:34.354 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T17:48:34.772 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:34 smithi097 ceph-mon[32419]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T17:48:34.772 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:34 smithi097 ceph-mon[32419]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T17:48:34.772 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:34 smithi097 ceph-mon[32419]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-26T17:48:34.772 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:34 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:34.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:34 smithi031 ceph-mon[29164]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T17:48:34.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:34 smithi031 ceph-mon[29164]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T17:48:34.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:34 smithi031 ceph-mon[29164]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-26T17:48:34.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:34 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:35.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:34 smithi186 ceph-mon[32797]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T17:48:35.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:34 smithi186 ceph-mon[32797]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T17:48:35.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:34 smithi186 ceph-mon[32797]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-26T17:48:35.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:34 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:35.304 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 17:48:35 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-3[48391]: 2024-04-26T17:48:35.086+0000 7f31040ba700 -1 osd.3 0 waiting for initial osdmap 2024-04-26T17:48:35.304 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 17:48:35 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-3[48391]: 2024-04-26T17:48:35.093+0000 7f30fb6a4700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T17:48:35.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:35 smithi097 ceph-mon[32419]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:35.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:35 smithi097 ceph-mon[32419]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-26T17:48:35.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:35 smithi097 ceph-mon[32419]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T17:48:35.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:35.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:35.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:35 smithi097 ceph-mon[32419]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' 2024-04-26T17:48:36.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:35 smithi186 ceph-mon[32797]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:36.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:35 smithi186 ceph-mon[32797]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-26T17:48:36.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:35 smithi186 ceph-mon[32797]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T17:48:36.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:36.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:36.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:35 smithi186 ceph-mon[32797]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' 2024-04-26T17:48:36.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:35 smithi031 ceph-mon[29164]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T17:48:36.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:35 smithi031 ceph-mon[29164]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-26T17:48:36.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:35 smithi031 ceph-mon[29164]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T17:48:36.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:35 smithi031 ceph-mon[29164]: from='osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341]' entity='osd.3' 2024-04-26T17:48:36.200 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T17:48:36.200 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T17:48:36.200 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-04-26T17:48:36.201 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-04-26T17:48:36.201 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115432 s, 908 MB/s 2024-04-26T17:48:36.201 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-04-26T17:48:36.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:36 smithi031 ceph-mon[29164]: purged_snaps scrub starts 2024-04-26T17:48:36.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:36 smithi031 ceph-mon[29164]: purged_snaps scrub ok 2024-04-26T17:48:36.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:36 smithi031 ceph-mon[29164]: osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341] boot 2024-04-26T17:48:36.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:36 smithi031 ceph-mon[29164]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T17:48:36.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:36 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:36.840 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:36 smithi097 ceph-mon[32419]: purged_snaps scrub starts 2024-04-26T17:48:36.841 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:36 smithi097 ceph-mon[32419]: purged_snaps scrub ok 2024-04-26T17:48:36.841 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:36 smithi097 ceph-mon[32419]: osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341] boot 2024-04-26T17:48:36.841 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:36 smithi097 ceph-mon[32419]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T17:48:36.841 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:36 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:36.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_3 2024-04-26T17:48:37.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:36 smithi186 ceph-mon[32797]: purged_snaps scrub starts 2024-04-26T17:48:37.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:36 smithi186 ceph-mon[32797]: purged_snaps scrub ok 2024-04-26T17:48:37.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:36 smithi186 ceph-mon[32797]: osd.3 [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341] boot 2024-04-26T17:48:37.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:36 smithi186 ceph-mon[32797]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T17:48:37.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:36 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:48:37.120 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T17:48:37.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:37 smithi097 ceph-mon[32419]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-26T17:48:37.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:37 smithi097 ceph-mon[32419]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T17:48:37.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:37.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:37.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:48:37.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:48:37.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:37.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:37.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:37.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:38.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:37 smithi186 ceph-mon[32797]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-26T17:48:38.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:37 smithi186 ceph-mon[32797]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T17:48:38.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:38.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:38.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:48:38.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:48:38.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:38.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:38.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:38.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:38.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:37 smithi031 ceph-mon[29164]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-26T17:48:38.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:37 smithi031 ceph-mon[29164]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T17:48:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:48:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:48:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:48:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:38.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.739+0000 7fa2478bb700 1 -- 172.21.15.97:0/3736838927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa240103f30 msgr2=0x7fa24010a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:48:38.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.739+0000 7fa2478bb700 1 --2- 172.21.15.97:0/3736838927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa240103f30 0x7fa24010a7e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa23800b0d0 tx=0x7fa23800b3e0 comp rx=0 tx=0).stop 2024-04-26T17:48:38.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.739+0000 7fa2478bb700 1 -- 172.21.15.97:0/3736838927 shutdown_connections 2024-04-26T17:48:38.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.739+0000 7fa2478bb700 1 --2- 172.21.15.97:0/3736838927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa240103f30 0x7fa24010a7e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:38.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.739+0000 7fa2478bb700 1 --2- 172.21.15.97:0/3736838927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa240102390 0x7fa2401027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:38.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.739+0000 7fa2478bb700 1 --2- 172.21.15.97:0/3736838927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa240103590 0x7fa2401039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:38.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.739+0000 7fa2478bb700 1 -- 172.21.15.97:0/3736838927 >> 172.21.15.97:0/3736838927 conn(0x7fa2400fd950 msgr2=0x7fa2400ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:48:38.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.739+0000 7fa2478bb700 1 -- 172.21.15.97:0/3736838927 shutdown_connections 2024-04-26T17:48:38.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.739+0000 7fa2478bb700 1 -- 172.21.15.97:0/3736838927 wait complete. 2024-04-26T17:48:38.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.740+0000 7fa2478bb700 1 Processor -- start 2024-04-26T17:48:38.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.740+0000 7fa2478bb700 1 -- start start 2024-04-26T17:48:38.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.740+0000 7fa2478bb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa240102390 0x7fa240199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:48:38.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.740+0000 7fa2478bb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa240103590 0x7fa240199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:48:38.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.740+0000 7fa2478bb700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa240103f30 0x7fa2401a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:48:38.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.740+0000 7fa2478bb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa24010de80 con 0x7fa240102390 2024-04-26T17:48:38.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.740+0000 7fa2478bb700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa24010dd00 con 0x7fa240103f30 2024-04-26T17:48:38.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.740+0000 7fa2478bb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa24010e000 con 0x7fa240103590 2024-04-26T17:48:38.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.740+0000 7fa245e58700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa240103f30 0x7fa2401a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:48:38.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.740+0000 7fa245e58700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa240103f30 0x7fa2401a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45156/0 (socket says 172.21.15.97:45156) 2024-04-26T17:48:38.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.740+0000 7fa245657700 1 --2- 172.21.15.97:0/2394699190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa240102390 0x7fa240199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:48:38.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.740+0000 7fa245e58700 1 -- 172.21.15.97:0/2394699190 learned_addr learned my addr 172.21.15.97:0/2394699190 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-04-26T17:48:38.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.740+0000 7fa244e56700 1 --2- 172.21.15.97:0/2394699190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa240103590 0x7fa240199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:48:38.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.741+0000 7fa245e58700 1 -- 172.21.15.97:0/2394699190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa240103590 msgr2=0x7fa240199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:48:38.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.741+0000 7fa245e58700 1 --2- 172.21.15.97:0/2394699190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa240103590 0x7fa240199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:38.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.741+0000 7fa245e58700 1 -- 172.21.15.97:0/2394699190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa240102390 msgr2=0x7fa240199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:48:38.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.741+0000 7fa245e58700 1 --2- 172.21.15.97:0/2394699190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa240102390 0x7fa240199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:38.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.741+0000 7fa245e58700 1 -- 172.21.15.97:0/2394699190 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2401a1280 con 0x7fa240103f30 2024-04-26T17:48:38.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.741+0000 7fa245e58700 1 --2- 172.21.15.97:0/2394699190 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa240103f30 0x7fa2401a0be0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa23800d010 tx=0x7fa238007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:48:38.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.741+0000 7fa2367fc700 1 -- 172.21.15.97:0/2394699190 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238021070 con 0x7fa240103f30 2024-04-26T17:48:38.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.741+0000 7fa2478bb700 1 -- 172.21.15.97:0/2394699190 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2401a1510 con 0x7fa240103f30 2024-04-26T17:48:38.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.741+0000 7fa2367fc700 1 -- 172.21.15.97:0/2394699190 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa23801db00 con 0x7fa240103f30 2024-04-26T17:48:38.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.741+0000 7fa2367fc700 1 -- 172.21.15.97:0/2394699190 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23801bbf0 con 0x7fa240103f30 2024-04-26T17:48:38.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.741+0000 7fa2478bb700 1 -- 172.21.15.97:0/2394699190 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2401a1a40 con 0x7fa240103f30 2024-04-26T17:48:38.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.742+0000 7fa2478bb700 1 -- 172.21.15.97:0/2394699190 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa24006a110 con 0x7fa240103f30 2024-04-26T17:48:38.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.743+0000 7fa2367fc700 1 -- 172.21.15.97:0/2394699190 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa238024070 con 0x7fa240103f30 2024-04-26T17:48:38.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.744+0000 7fa2367fc700 1 --2- 172.21.15.97:0/2394699190 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa22c0a3c40 0x7fa22c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:48:38.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.744+0000 7fa2367fc700 1 -- 172.21.15.97:0/2394699190 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7fa23800f020 con 0x7fa240103f30 2024-04-26T17:48:38.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.744+0000 7fa245657700 1 --2- 172.21.15.97:0/2394699190 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa22c0a3c40 0x7fa22c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:48:38.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.744+0000 7fa245657700 1 --2- 172.21.15.97:0/2394699190 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa22c0a3c40 0x7fa22c0a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa2401033f0 tx=0x7fa23000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:48:38.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.747+0000 7fa2367fc700 1 -- 172.21.15.97:0/2394699190 <== mon.2 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa23802b020 con 0x7fa240103f30 2024-04-26T17:48:38.855 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:38 smithi031 ceph-mon[29164]: Detected new or changed devices on smithi031 2024-04-26T17:48:38.855 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:38 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi031 to 8546M 2024-04-26T17:48:38.855 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:38 smithi031 ceph-mon[29164]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T17:48:38.855 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:38 smithi031 ceph-mon[29164]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T17:48:38.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:38.953+0000 7fa2478bb700 1 -- 172.21.15.97:0/2394699190 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa2400ffb40 con 0x7fa22c0a3c40 2024-04-26T17:48:38.952 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:38 smithi097 ceph-mon[32419]: Detected new or changed devices on smithi031 2024-04-26T17:48:38.952 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:38 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi031 to 8546M 2024-04-26T17:48:38.952 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:38 smithi097 ceph-mon[32419]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T17:48:38.952 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:38 smithi097 ceph-mon[32419]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T17:48:39.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:38 smithi186 ceph-mon[32797]: Detected new or changed devices on smithi031 2024-04-26T17:48:39.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:38 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi031 to 8546M 2024-04-26T17:48:39.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:38 smithi186 ceph-mon[32797]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T17:48:39.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:38 smithi186 ceph-mon[32797]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T17:48:39.855 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:39 smithi097 ceph-mon[32419]: pgmap v102: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T17:48:39.855 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:39 smithi097 ceph-mon[32419]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T17:48:39.855 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:39 smithi097 ceph-mon[32419]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:48:39.855 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:39 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:48:39.855 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:39 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:48:39.855 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:39 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:39.855 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:39 smithi097 ceph-mon[32419]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T17:48:40.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:39 smithi186 ceph-mon[32797]: pgmap v102: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T17:48:40.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:39 smithi186 ceph-mon[32797]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T17:48:40.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:39 smithi186 ceph-mon[32797]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:48:40.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:39 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:48:40.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:39 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:48:40.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:39 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:40.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:39 smithi186 ceph-mon[32797]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T17:48:40.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:39 smithi031 ceph-mon[29164]: pgmap v102: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T17:48:40.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:39 smithi031 ceph-mon[29164]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T17:48:40.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:39 smithi031 ceph-mon[29164]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:48:40.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:39 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:48:40.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:39 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:48:40.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:39 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:40.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:39 smithi031 ceph-mon[29164]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T17:48:41.969 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:41 smithi097 ceph-mon[32419]: pgmap v105: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T17:48:41.969 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:41 smithi097 ceph-mon[32419]: from='client.? 172.21.15.97:0/1456328274' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "112ccbf7-24e8-473c-9852-2e7167dd3108"}]: dispatch 2024-04-26T17:48:41.970 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:41 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "112ccbf7-24e8-473c-9852-2e7167dd3108"}]: dispatch 2024-04-26T17:48:41.970 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:41 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "112ccbf7-24e8-473c-9852-2e7167dd3108"}]': finished 2024-04-26T17:48:41.970 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:41 smithi097 ceph-mon[32419]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T17:48:41.970 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:48:42.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:41 smithi186 ceph-mon[32797]: pgmap v105: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T17:48:42.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:41 smithi186 ceph-mon[32797]: from='client.? 172.21.15.97:0/1456328274' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "112ccbf7-24e8-473c-9852-2e7167dd3108"}]: dispatch 2024-04-26T17:48:42.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:41 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "112ccbf7-24e8-473c-9852-2e7167dd3108"}]: dispatch 2024-04-26T17:48:42.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:41 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "112ccbf7-24e8-473c-9852-2e7167dd3108"}]': finished 2024-04-26T17:48:42.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:41 smithi186 ceph-mon[32797]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T17:48:42.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:41 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:48:42.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:41 smithi031 ceph-mon[29164]: pgmap v105: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T17:48:42.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:41 smithi031 ceph-mon[29164]: from='client.? 172.21.15.97:0/1456328274' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "112ccbf7-24e8-473c-9852-2e7167dd3108"}]: dispatch 2024-04-26T17:48:42.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:41 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "112ccbf7-24e8-473c-9852-2e7167dd3108"}]: dispatch 2024-04-26T17:48:42.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:41 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "112ccbf7-24e8-473c-9852-2e7167dd3108"}]': finished 2024-04-26T17:48:42.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:41 smithi031 ceph-mon[29164]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T17:48:42.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:41 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:48:43.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:42 smithi186 ceph-mon[32797]: from='client.? 172.21.15.97:0/3977282386' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:48:43.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:42 smithi031 ceph-mon[29164]: from='client.? 172.21.15.97:0/3977282386' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:48:43.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:42 smithi097 ceph-mon[32419]: from='client.? 172.21.15.97:0/3977282386' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:48:44.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:43 smithi186 ceph-mon[32797]: pgmap v107: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T17:48:44.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:43 smithi031 ceph-mon[29164]: pgmap v107: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T17:48:44.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:43 smithi097 ceph-mon[32419]: pgmap v107: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T17:48:45.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:44 smithi186 ceph-mon[32797]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T17:48:45.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:44 smithi186 ceph-mon[32797]: Cluster is now healthy 2024-04-26T17:48:45.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:44 smithi031 ceph-mon[29164]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T17:48:45.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:44 smithi031 ceph-mon[29164]: Cluster is now healthy 2024-04-26T17:48:45.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:44 smithi097 ceph-mon[32419]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T17:48:45.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:44 smithi097 ceph-mon[32419]: Cluster is now healthy 2024-04-26T17:48:45.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:45 smithi097 ceph-mon[32419]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T17:48:46.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:45 smithi186 ceph-mon[32797]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T17:48:46.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:45 smithi031 ceph-mon[29164]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T17:48:48.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:47 smithi186 ceph-mon[32797]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T17:48:48.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:47 smithi031 ceph-mon[29164]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T17:48:48.124 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:47 smithi097 ceph-mon[32419]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T17:48:50.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:49 smithi186 ceph-mon[32797]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-26T17:48:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T17:48:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:49 smithi186 ceph-mon[32797]: Deploying daemon osd.4 on smithi097 2024-04-26T17:48:50.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:49 smithi031 ceph-mon[29164]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-26T17:48:50.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T17:48:50.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:50.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:49 smithi031 ceph-mon[29164]: Deploying daemon osd.4 on smithi097 2024-04-26T17:48:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:49 smithi097 ceph-mon[32419]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-26T17:48:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T17:48:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:48:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:49 smithi097 ceph-mon[32419]: Deploying daemon osd.4 on smithi097 2024-04-26T17:48:52.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:51 smithi186 ceph-mon[32797]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T17:48:52.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:51 smithi031 ceph-mon[29164]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T17:48:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:51 smithi097 ceph-mon[32419]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T17:48:53.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:53 smithi097 ceph-mon[32419]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-26T17:48:54.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:53 smithi186 ceph-mon[32797]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-26T17:48:54.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:53 smithi031 ceph-mon[29164]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-26T17:48:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:55 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:55 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:55 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:55 smithi097 ceph-mon[32419]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T17:48:55.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:55 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:55 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:55 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:55 smithi186 ceph-mon[32797]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T17:48:55.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:55 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:55.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:55 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:55.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:55 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:48:55.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:55 smithi031 ceph-mon[29164]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T17:48:57.517 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 4 on host 'smithi097' 2024-04-26T17:48:57.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:57.517+0000 7fa2367fc700 1 -- 172.21.15.97:0/2394699190 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa2400ffb40 con 0x7fa22c0a3c40 2024-04-26T17:48:57.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:57.519+0000 7fa2478bb700 1 -- 172.21.15.97:0/2394699190 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa22c0a3c40 msgr2=0x7fa22c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:48:57.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:57.519+0000 7fa2478bb700 1 --2- 172.21.15.97:0/2394699190 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa22c0a3c40 0x7fa22c0a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa2401033f0 tx=0x7fa23000b410 comp rx=0 tx=0).stop 2024-04-26T17:48:57.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:57.519+0000 7fa2478bb700 1 -- 172.21.15.97:0/2394699190 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa240103f30 msgr2=0x7fa2401a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:48:57.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:57.519+0000 7fa2478bb700 1 --2- 172.21.15.97:0/2394699190 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa240103f30 0x7fa2401a0be0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa23800d010 tx=0x7fa238007a20 comp rx=0 tx=0).stop 2024-04-26T17:48:57.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:57.519+0000 7fa2478bb700 1 -- 172.21.15.97:0/2394699190 shutdown_connections 2024-04-26T17:48:57.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:57.519+0000 7fa2478bb700 1 --2- 172.21.15.97:0/2394699190 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa22c0a3c40 0x7fa22c0a6100 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:57.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:57.519+0000 7fa2478bb700 1 --2- 172.21.15.97:0/2394699190 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa240103f30 0x7fa2401a0be0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:57.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:57.519+0000 7fa2478bb700 1 --2- 172.21.15.97:0/2394699190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa240102390 0x7fa240199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:57.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:57.519+0000 7fa2478bb700 1 --2- 172.21.15.97:0/2394699190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa240103590 0x7fa240199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:48:57.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:57.519+0000 7fa2478bb700 1 -- 172.21.15.97:0/2394699190 >> 172.21.15.97:0/2394699190 conn(0x7fa2400fd950 msgr2=0x7fa2400ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:48:57.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:57.519+0000 7fa2478bb700 1 -- 172.21.15.97:0/2394699190 shutdown_connections 2024-04-26T17:48:57.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:48:57.519+0000 7fa2478bb700 1 -- 172.21.15.97:0/2394699190 wait complete. 2024-04-26T17:48:57.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:57 smithi031 ceph-mon[29164]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-26T17:48:57.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:57 smithi097 ceph-mon[32419]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-26T17:48:57.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:57 smithi186 ceph-mon[32797]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-26T17:48:58.157 DEBUG:teuthology.orchestra.run.smithi097:osd.4> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.4.service 2024-04-26T17:48:58.159 INFO:tasks.cephadm:Deploying osd.5 on smithi186 with /dev/vg_nvme/lv_3... 2024-04-26T17:48:58.160 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T17:48:58.407 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T17:48:58.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:58.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:58.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:58 smithi186 ceph-mon[32797]: from='osd.4 [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T17:48:58.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:58 smithi186 ceph-mon[32797]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T17:48:58.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:58.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:58.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:58.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:58.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:58 smithi031 ceph-mon[29164]: from='osd.4 [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T17:48:58.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:58 smithi031 ceph-mon[29164]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T17:48:58.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:58.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:58.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:58.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:58.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:58 smithi097 ceph-mon[32419]: from='osd.4 [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T17:48:58.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:58 smithi097 ceph-mon[32419]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T17:48:58.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:58.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:48:59.718 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:59 smithi186 ceph-mon[32797]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-26T17:48:59.718 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:59 smithi186 ceph-mon[32797]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T17:48:59.718 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:59 smithi186 ceph-mon[32797]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T17:48:59.718 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:59 smithi186 ceph-mon[32797]: from='osd.4 [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:48:59.719 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:48:59.719 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:48:59 smithi186 ceph-mon[32797]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:48:59.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:59 smithi031 ceph-mon[29164]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-26T17:48:59.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:59 smithi031 ceph-mon[29164]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T17:48:59.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:59 smithi031 ceph-mon[29164]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T17:48:59.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:59 smithi031 ceph-mon[29164]: from='osd.4 [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:48:59.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:48:59.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:48:59 smithi031 ceph-mon[29164]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:48:59.871 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:59 smithi097 ceph-mon[32419]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-26T17:48:59.871 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:59 smithi097 ceph-mon[32419]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T17:48:59.872 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:59 smithi097 ceph-mon[32419]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T17:48:59.872 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:59 smithi097 ceph-mon[32419]: from='osd.4 [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:48:59.872 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:48:59.872 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:48:59 smithi097 ceph-mon[32419]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:48:59.872 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 17:48:59 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4[43823]: 2024-04-26T17:48:59.516+0000 7f905592a700 -1 osd.4 0 waiting for initial osdmap 2024-04-26T17:48:59.872 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 17:48:59 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4[43823]: 2024-04-26T17:48:59.523+0000 7f905171d700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T17:49:00.393 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T17:49:00.394 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T17:49:00.394 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-26T17:49:00.394 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-26T17:49:00.394 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129401 s, 810 MB/s 2024-04-26T17:49:00.394 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-26T17:49:00.682 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:00 smithi186 ceph-mon[32797]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-26T17:49:00.683 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:00 smithi186 ceph-mon[32797]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T17:49:00.683 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:00 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:49:00.683 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:00 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:49:00.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:00 smithi031 ceph-mon[29164]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-26T17:49:00.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:00 smithi031 ceph-mon[29164]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T17:49:00.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:00 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:49:00.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:00 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:49:00.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:00 smithi097 ceph-mon[32419]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-26T17:49:00.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:00 smithi097 ceph-mon[32419]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T17:49:00.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:00 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:49:00.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:00 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:49:01.044 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_3 2024-04-26T17:49:01.279 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T17:49:01.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:01 smithi186 ceph-mon[32797]: purged_snaps scrub starts 2024-04-26T17:49:01.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:01 smithi186 ceph-mon[32797]: purged_snaps scrub ok 2024-04-26T17:49:01.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:01 smithi186 ceph-mon[32797]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-26T17:49:01.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:01 smithi186 ceph-mon[32797]: osd.4 [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143] boot 2024-04-26T17:49:01.728 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:01 smithi186 ceph-mon[32797]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T17:49:01.729 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:49:01.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:01 smithi031 ceph-mon[29164]: purged_snaps scrub starts 2024-04-26T17:49:01.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:01 smithi031 ceph-mon[29164]: purged_snaps scrub ok 2024-04-26T17:49:01.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:01 smithi031 ceph-mon[29164]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-26T17:49:01.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:01 smithi031 ceph-mon[29164]: osd.4 [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143] boot 2024-04-26T17:49:01.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:01 smithi031 ceph-mon[29164]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T17:49:01.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:49:01.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:01 smithi097 ceph-mon[32419]: purged_snaps scrub starts 2024-04-26T17:49:01.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:01 smithi097 ceph-mon[32419]: purged_snaps scrub ok 2024-04-26T17:49:01.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:01 smithi097 ceph-mon[32419]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-26T17:49:01.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:01 smithi097 ceph-mon[32419]: osd.4 [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143] boot 2024-04-26T17:49:01.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:01 smithi097 ceph-mon[32419]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T17:49:01.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:49:02.634 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T17:49:02.634 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: Detected new or changed devices on smithi097 2024-04-26T17:49:02.634 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.635 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.635 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:02.635 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:02.635 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi097 to 8544M 2024-04-26T17:49:02.635 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.635 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:02.635 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:02.635 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.635 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:02.635 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:02.635 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:02.635 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T17:49:02.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: Detected new or changed devices on smithi097 2024-04-26T17:49:02.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:02.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:02.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi097 to 8544M 2024-04-26T17:49:02.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:02.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:02.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:02.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:02.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:02.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.875+0000 7f458d16b700 1 -- 172.21.15.186:0/3097872860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4588103940 msgr2=0x7f4588103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:49:02.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.875+0000 7f458d16b700 1 --2- 172.21.15.186:0/3097872860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4588103940 0x7f4588103d60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f45780099c0 tx=0x7f4578009cd0 comp rx=0 tx=0).stop 2024-04-26T17:49:02.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.876+0000 7f458d16b700 1 -- 172.21.15.186:0/3097872860 shutdown_connections 2024-04-26T17:49:02.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.876+0000 7f458d16b700 1 --2- 172.21.15.186:0/3097872860 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4588103020 0x7f458810a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:02.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.876+0000 7f458d16b700 1 --2- 172.21.15.186:0/3097872860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4588103940 0x7f4588103d60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:02.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.876+0000 7f458d16b700 1 --2- 172.21.15.186:0/3097872860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4588102680 0x7f4588102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:02.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.876+0000 7f458d16b700 1 -- 172.21.15.186:0/3097872860 >> 172.21.15.186:0/3097872860 conn(0x7f45880fd0d0 msgr2=0x7f45880ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:49:02.876 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.876+0000 7f458d16b700 1 -- 172.21.15.186:0/3097872860 shutdown_connections 2024-04-26T17:49:02.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.876+0000 7f458d16b700 1 -- 172.21.15.186:0/3097872860 wait complete. 2024-04-26T17:49:02.877 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.876+0000 7f458d16b700 1 Processor -- start 2024-04-26T17:49:02.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.876+0000 7f458d16b700 1 -- start start 2024-04-26T17:49:02.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f458d16b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4588102680 0x7f458819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:49:02.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f458d16b700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4588103020 0x7f458819bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:49:02.879 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f458d16b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4588103940 0x7f45881a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:49:02.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f458d16b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f458810dea0 con 0x7f4588103940 2024-04-26T17:49:02.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f458d16b700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f458810dd20 con 0x7f4588103020 2024-04-26T17:49:02.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f458d16b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f458810e020 con 0x7f4588102680 2024-04-26T17:49:02.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f4586d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4588102680 0x7f458819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:49:02.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f4586d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4588102680 0x7f458819b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:50526/0 (socket says 172.21.15.186:50526) 2024-04-26T17:49:02.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f4586d9d700 1 -- 172.21.15.186:0/3620406774 learned_addr learned my addr 172.21.15.186:0/3620406774 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T17:49:02.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f458759e700 1 --2- 172.21.15.186:0/3620406774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4588103940 0x7f45881a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:49:02.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f458659c700 1 --2- 172.21.15.186:0/3620406774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4588103020 0x7f458819bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:49:02.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f4586d9d700 1 -- 172.21.15.186:0/3620406774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4588103020 msgr2=0x7f458819bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:49:02.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f4586d9d700 1 --2- 172.21.15.186:0/3620406774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4588103020 0x7f458819bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:02.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f4586d9d700 1 -- 172.21.15.186:0/3620406774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4588103940 msgr2=0x7f45881a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:49:02.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f4586d9d700 1 --2- 172.21.15.186:0/3620406774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4588103940 0x7f45881a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:02.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f4586d9d700 1 -- 172.21.15.186:0/3620406774 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45881a3500 con 0x7f4588102680 2024-04-26T17:49:02.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f4586d9d700 1 --2- 172.21.15.186:0/3620406774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4588102680 0x7f458819b8a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4578009f50 tx=0x7f45780047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:49:02.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f456ffff700 1 -- 172.21.15.186:0/3620406774 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4578027070 con 0x7f4588102680 2024-04-26T17:49:02.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.877+0000 7f458d16b700 1 -- 172.21.15.186:0/3620406774 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45881a3790 con 0x7f4588102680 2024-04-26T17:49:02.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.878+0000 7f458d16b700 1 -- 172.21.15.186:0/3620406774 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45881a3cc0 con 0x7f4588102680 2024-04-26T17:49:02.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.878+0000 7f456ffff700 1 -- 172.21.15.186:0/3620406774 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45780040a0 con 0x7f4588102680 2024-04-26T17:49:02.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.878+0000 7f456ffff700 1 -- 172.21.15.186:0/3620406774 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4578022ce0 con 0x7f4588102680 2024-04-26T17:49:02.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.878+0000 7f458d16b700 1 -- 172.21.15.186:0/3620406774 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4574005330 con 0x7f4588102680 2024-04-26T17:49:02.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.879+0000 7f456ffff700 1 -- 172.21.15.186:0/3620406774 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f457802a070 con 0x7f4588102680 2024-04-26T17:49:02.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.880+0000 7f456ffff700 1 --2- 172.21.15.186:0/3620406774 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f45700a3c40 0x7f45700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:49:02.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.880+0000 7f456ffff700 1 -- 172.21.15.186:0/3620406774 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f45780cfec0 con 0x7f4588102680 2024-04-26T17:49:02.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.880+0000 7f458659c700 1 --2- 172.21.15.186:0/3620406774 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f45700a3c40 0x7f45700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:49:02.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.881+0000 7f458659c700 1 --2- 172.21.15.186:0/3620406774 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f45700a3c40 0x7f45700a6100 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f457c006fd0 tx=0x7f457c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:49:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T17:49:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: Detected new or changed devices on smithi097 2024-04-26T17:49:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi097 to 8544M 2024-04-26T17:49:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:02.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:02.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:02.885+0000 7f456ffff700 1 -- 172.21.15.186:0/3620406774 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4578013070 con 0x7f4588102680 2024-04-26T17:49:03.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:03.090+0000 7f458d16b700 1 -- 172.21.15.186:0/3620406774 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f4574000bc0 con 0x7f45700a3c40 2024-04-26T17:49:03.957 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:03 smithi186 ceph-mon[32797]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:03.957 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:03 smithi186 ceph-mon[32797]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:49:03.957 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:49:03.957 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:49:03.957 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:04.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:03 smithi097 ceph-mon[32419]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:04.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:03 smithi097 ceph-mon[32419]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:49:04.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:49:04.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:49:04.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:04.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:03 smithi031 ceph-mon[29164]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:04.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:03 smithi031 ceph-mon[29164]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:49:04.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:49:04.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:49:04.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:06.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:05 smithi097 ceph-mon[32419]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:06.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:05 smithi186 ceph-mon[32797]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:06.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:05 smithi031 ceph-mon[29164]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:07.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:06 smithi097 ceph-mon[32419]: from='client.? 172.21.15.186:0/3551577323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb17759e-1ad8-4bda-9f61-54d2a9781535"}]: dispatch 2024-04-26T17:49:07.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:06 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb17759e-1ad8-4bda-9f61-54d2a9781535"}]: dispatch 2024-04-26T17:49:07.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:06 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb17759e-1ad8-4bda-9f61-54d2a9781535"}]': finished 2024-04-26T17:49:07.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:06 smithi097 ceph-mon[32419]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T17:49:07.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:06 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:07.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:06 smithi097 ceph-mon[32419]: from='client.? 172.21.15.186:0/835311135' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:49:07.210 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:06 smithi031 ceph-mon[29164]: from='client.? 172.21.15.186:0/3551577323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb17759e-1ad8-4bda-9f61-54d2a9781535"}]: dispatch 2024-04-26T17:49:07.210 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:06 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb17759e-1ad8-4bda-9f61-54d2a9781535"}]: dispatch 2024-04-26T17:49:07.210 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:06 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb17759e-1ad8-4bda-9f61-54d2a9781535"}]': finished 2024-04-26T17:49:07.210 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:06 smithi031 ceph-mon[29164]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T17:49:07.210 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:06 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:07.211 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:06 smithi031 ceph-mon[29164]: from='client.? 172.21.15.186:0/835311135' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:49:07.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:06 smithi186 ceph-mon[32797]: from='client.? 172.21.15.186:0/3551577323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb17759e-1ad8-4bda-9f61-54d2a9781535"}]: dispatch 2024-04-26T17:49:07.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:06 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb17759e-1ad8-4bda-9f61-54d2a9781535"}]: dispatch 2024-04-26T17:49:07.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:06 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb17759e-1ad8-4bda-9f61-54d2a9781535"}]': finished 2024-04-26T17:49:07.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:06 smithi186 ceph-mon[32797]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T17:49:07.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:06 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:07.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:06 smithi186 ceph-mon[32797]: from='client.? 172.21.15.186:0/835311135' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:49:08.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:07 smithi186 ceph-mon[32797]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:08.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:07 smithi031 ceph-mon[29164]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:08.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:07 smithi097 ceph-mon[32419]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:10.244 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:09 smithi186 ceph-mon[32797]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:10.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:09 smithi031 ceph-mon[29164]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:10.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:09 smithi097 ceph-mon[32419]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:12.071 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:11 smithi186 ceph-mon[32797]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:12.255 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:11 smithi097 ceph-mon[32419]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:12.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:11 smithi031 ceph-mon[29164]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:14.060 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:13 smithi186 ceph-mon[32797]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:14.060 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:13 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T17:49:14.061 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:13 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:14.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:13 smithi031 ceph-mon[29164]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:14.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:13 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T17:49:14.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:13 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:14.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:13 smithi097 ceph-mon[32419]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:14.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:13 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T17:49:14.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:13 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:15.126 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:14 smithi186 ceph-mon[32797]: Deploying daemon osd.5 on smithi186 2024-04-26T17:49:15.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:14 smithi031 ceph-mon[29164]: Deploying daemon osd.5 on smithi186 2024-04-26T17:49:15.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:14 smithi097 ceph-mon[32419]: Deploying daemon osd.5 on smithi186 2024-04-26T17:49:16.211 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:15 smithi186 ceph-mon[32797]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:16.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:15 smithi031 ceph-mon[29164]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:16.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:15 smithi097 ceph-mon[32419]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:18.172 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:17 smithi186 ceph-mon[32797]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:18.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:17 smithi031 ceph-mon[29164]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:18.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:17 smithi097 ceph-mon[32419]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:19 smithi186 ceph-mon[32797]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:20.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:19 smithi031 ceph-mon[29164]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:20.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:20.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:20.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:20.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:19 smithi097 ceph-mon[32419]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:20.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:20.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:20.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:22.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:21 smithi186 ceph-mon[32797]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:22.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:21 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:22.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:21 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:22.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:21 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:22.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:21 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:22.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:21 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:22.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:21 smithi031 ceph-mon[29164]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:22.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:21 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:22.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:21 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:22.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:21 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:22.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:21 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:22.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:21 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:22.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:21 smithi097 ceph-mon[32419]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:22.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:21 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:22.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:21 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:22.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:21 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:22.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:21 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:22.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:21 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:22.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:22.461+0000 7f456ffff700 1 -- 172.21.15.186:0/3620406774 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4574000bc0 con 0x7f45700a3c40 2024-04-26T17:49:22.461 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 5 on host 'smithi186' 2024-04-26T17:49:22.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:22.463+0000 7f458d16b700 1 -- 172.21.15.186:0/3620406774 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f45700a3c40 msgr2=0x7f45700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:49:22.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:22.463+0000 7f458d16b700 1 --2- 172.21.15.186:0/3620406774 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f45700a3c40 0x7f45700a6100 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f457c006fd0 tx=0x7f457c008040 comp rx=0 tx=0).stop 2024-04-26T17:49:22.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:22.463+0000 7f458d16b700 1 -- 172.21.15.186:0/3620406774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4588102680 msgr2=0x7f458819b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:49:22.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:22.463+0000 7f458d16b700 1 --2- 172.21.15.186:0/3620406774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4588102680 0x7f458819b8a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4578009f50 tx=0x7f45780047b0 comp rx=0 tx=0).stop 2024-04-26T17:49:22.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:22.463+0000 7f458d16b700 1 -- 172.21.15.186:0/3620406774 shutdown_connections 2024-04-26T17:49:22.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:22.463+0000 7f458d16b700 1 --2- 172.21.15.186:0/3620406774 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f45700a3c40 0x7f45700a6100 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:22.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:22.463+0000 7f458d16b700 1 --2- 172.21.15.186:0/3620406774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4588103940 0x7f45881a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:22.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:22.463+0000 7f458d16b700 1 --2- 172.21.15.186:0/3620406774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4588102680 0x7f458819b8a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:22.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:22.463+0000 7f458d16b700 1 --2- 172.21.15.186:0/3620406774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4588103020 0x7f458819bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:22.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:22.463+0000 7f458d16b700 1 -- 172.21.15.186:0/3620406774 >> 172.21.15.186:0/3620406774 conn(0x7f45880fd0d0 msgr2=0x7f45880fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:49:22.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:22.464+0000 7f458d16b700 1 -- 172.21.15.186:0/3620406774 shutdown_connections 2024-04-26T17:49:22.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:49:22.464+0000 7f458d16b700 1 -- 172.21.15.186:0/3620406774 wait complete. 2024-04-26T17:49:23.180 DEBUG:teuthology.orchestra.run.smithi186:osd.5> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.5.service 2024-04-26T17:49:23.183 INFO:tasks.cephadm:Deploying osd.6 on smithi031 with /dev/vg_nvme/lv_2... 2024-04-26T17:49:23.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T17:49:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:49:23.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:23 smithi031 ceph-mon[29164]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:23.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:23 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:23.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:23 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:23.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:23 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:23.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:23 smithi031 ceph-mon[29164]: from='osd.5 [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T17:49:23.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:23 smithi031 ceph-mon[29164]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T17:49:23.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:23 smithi186 ceph-mon[32797]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:23.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:23 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:23.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:23 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:23.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:23 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:23.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:23 smithi186 ceph-mon[32797]: from='osd.5 [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T17:49:23.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:23 smithi186 ceph-mon[32797]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T17:49:23.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:23 smithi097 ceph-mon[32419]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:23.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:23 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:23.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:23 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:23.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:23 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:23.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:23 smithi097 ceph-mon[32419]: from='osd.5 [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T17:49:23.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:23 smithi097 ceph-mon[32419]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T17:49:24.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:24 smithi031 ceph-mon[29164]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T17:49:24.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:24 smithi031 ceph-mon[29164]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T17:49:24.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:24 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:24.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:24 smithi031 ceph-mon[29164]: from='osd.5 [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:49:24.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:24 smithi031 ceph-mon[29164]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:49:24.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:24 smithi186 ceph-mon[32797]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T17:49:24.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:24 smithi186 ceph-mon[32797]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T17:49:24.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:24 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:24.725 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:24 smithi186 ceph-mon[32797]: from='osd.5 [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:49:24.726 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:24 smithi186 ceph-mon[32797]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:49:24.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:24 smithi097 ceph-mon[32419]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T17:49:24.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:24 smithi097 ceph-mon[32419]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T17:49:24.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:24 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:24.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:24 smithi097 ceph-mon[32419]: from='osd.5 [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:49:24.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:24 smithi097 ceph-mon[32419]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:49:25.017 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 17:49:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T17:49:24.917+0000 7f4f89eb2700 -1 osd.5 0 waiting for initial osdmap 2024-04-26T17:49:25.017 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 17:49:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T17:49:24.923+0000 7f4f81c9d700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T17:49:25.500 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T17:49:25.500 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T17:49:25.501 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-04-26T17:49:25.501 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-04-26T17:49:25.501 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0129793 s, 808 MB/s 2024-04-26T17:49:25.501 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-04-26T17:49:25.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:25 smithi031 ceph-mon[29164]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:25.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:25 smithi031 ceph-mon[29164]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T17:49:25.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:25 smithi031 ceph-mon[29164]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T17:49:25.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:25 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:25.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:25 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:25.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:25 smithi031 ceph-mon[29164]: from='osd.5 ' entity='osd.5' 2024-04-26T17:49:25.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:25 smithi186 ceph-mon[32797]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:25.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:25 smithi186 ceph-mon[32797]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T17:49:25.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:25 smithi186 ceph-mon[32797]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T17:49:25.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:25 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:25.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:25 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:25.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:25 smithi186 ceph-mon[32797]: from='osd.5 ' entity='osd.5' 2024-04-26T17:49:25.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:25 smithi097 ceph-mon[32419]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-26T17:49:25.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:25 smithi097 ceph-mon[32419]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T17:49:25.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:25 smithi097 ceph-mon[32419]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T17:49:25.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:25 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:25.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:25 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:25.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:25 smithi097 ceph-mon[32419]: from='osd.5 ' entity='osd.5' 2024-04-26T17:49:26.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_2 2024-04-26T17:49:26.359 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:49:26.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:26 smithi186 ceph-mon[32797]: purged_snaps scrub starts 2024-04-26T17:49:26.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:26 smithi186 ceph-mon[32797]: purged_snaps scrub ok 2024-04-26T17:49:26.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:26 smithi186 ceph-mon[32797]: osd.5 [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626] boot 2024-04-26T17:49:26.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:26 smithi186 ceph-mon[32797]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T17:49:26.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:26 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:26.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:26 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:26.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:26 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:26.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:26 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:26.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:26 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:26.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:26 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:26.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:26 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:26.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:26 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:26.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:26 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:26.775 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:26 smithi031 ceph-mon[29164]: purged_snaps scrub starts 2024-04-26T17:49:26.775 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:26 smithi031 ceph-mon[29164]: purged_snaps scrub ok 2024-04-26T17:49:26.775 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:26 smithi031 ceph-mon[29164]: osd.5 [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626] boot 2024-04-26T17:49:26.775 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:26 smithi031 ceph-mon[29164]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T17:49:26.775 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:26.775 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:26.775 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:26.775 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:26.776 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:26.776 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:26.776 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:26.776 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:26.776 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:26.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:26 smithi097 ceph-mon[32419]: purged_snaps scrub starts 2024-04-26T17:49:26.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:26 smithi097 ceph-mon[32419]: purged_snaps scrub ok 2024-04-26T17:49:26.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:26 smithi097 ceph-mon[32419]: osd.5 [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626] boot 2024-04-26T17:49:26.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:26 smithi097 ceph-mon[32419]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T17:49:26.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:26 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:49:26.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:26 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:26.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:26 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:26.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:26 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:26.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:26 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:26.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:26 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:26.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:26 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:26.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:26 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:26.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:26 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:27.537 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:27 smithi031 ceph-mon[29164]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:27.537 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:27 smithi031 ceph-mon[29164]: Detected new or changed devices on smithi186 2024-04-26T17:49:27.537 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:27 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi186 to 8544M 2024-04-26T17:49:27.537 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:27 smithi031 ceph-mon[29164]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T17:49:27.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:27 smithi186 ceph-mon[32797]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:27.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:27 smithi186 ceph-mon[32797]: Detected new or changed devices on smithi186 2024-04-26T17:49:27.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:27 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi186 to 8544M 2024-04-26T17:49:27.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:27 smithi186 ceph-mon[32797]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T17:49:27.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:27 smithi097 ceph-mon[32419]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:27.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:27 smithi097 ceph-mon[32419]: Detected new or changed devices on smithi186 2024-04-26T17:49:27.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:27 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi186 to 8544M 2024-04-26T17:49:27.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:27 smithi097 ceph-mon[32419]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T17:49:28.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.084+0000 7fcdca7aa700 1 -- 172.21.15.31:0/4206450379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdc4105020 msgr2=0x7fcdc410b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:49:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.084+0000 7fcdca7aa700 1 --2- 172.21.15.31:0/4206450379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdc4105020 0x7fcdc410b8d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc00b0d0 tx=0x7fcdbc00b3e0 comp rx=0 tx=0).stop 2024-04-26T17:49:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.084+0000 7fcdca7aa700 1 -- 172.21.15.31:0/4206450379 shutdown_connections 2024-04-26T17:49:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.084+0000 7fcdca7aa700 1 --2- 172.21.15.31:0/4206450379 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdc4103480 0x7fcdc41038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.084+0000 7fcdca7aa700 1 --2- 172.21.15.31:0/4206450379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdc4105020 0x7fcdc410b8d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:28.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.084+0000 7fcdca7aa700 1 --2- 172.21.15.31:0/4206450379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdc4104680 0x7fcdc4104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:28.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.084+0000 7fcdca7aa700 1 -- 172.21.15.31:0/4206450379 >> 172.21.15.31:0/4206450379 conn(0x7fcdc40fea20 msgr2=0x7fcdc4100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:49:28.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.084+0000 7fcdca7aa700 1 -- 172.21.15.31:0/4206450379 shutdown_connections 2024-04-26T17:49:28.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.084+0000 7fcdca7aa700 1 -- 172.21.15.31:0/4206450379 wait complete. 2024-04-26T17:49:28.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.085+0000 7fcdca7aa700 1 Processor -- start 2024-04-26T17:49:28.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.085+0000 7fcdca7aa700 1 -- start start 2024-04-26T17:49:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.085+0000 7fcdca7aa700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdc4103480 0x7fcdc40771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:49:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.085+0000 7fcdca7aa700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdc4104680 0x7fcdc4075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:49:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.085+0000 7fcdca7aa700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdc4105020 0x7fcdc4075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:49:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.085+0000 7fcdca7aa700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc410ef00 con 0x7fcdc4105020 2024-04-26T17:49:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.085+0000 7fcdca7aa700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc410ed80 con 0x7fcdc4104680 2024-04-26T17:49:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.085+0000 7fcdca7aa700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc410f080 con 0x7fcdc4103480 2024-04-26T17:49:28.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.085+0000 7fcdc8d47700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdc4105020 0x7fcdc4075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:49:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.085+0000 7fcdc8d47700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdc4105020 0x7fcdc4075d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56702/0 (socket says 172.21.15.31:56702) 2024-04-26T17:49:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.085+0000 7fcdc8d47700 1 -- 172.21.15.31:0/2873323142 learned_addr learned my addr 172.21.15.31:0/2873323142 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:49:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.085+0000 7fcdc3fff700 1 --2- 172.21.15.31:0/2873323142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdc4103480 0x7fcdc40771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:49:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.085+0000 7fcdc37fe700 1 --2- 172.21.15.31:0/2873323142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdc4104680 0x7fcdc4075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:49:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.086+0000 7fcdc8d47700 1 -- 172.21.15.31:0/2873323142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdc4103480 msgr2=0x7fcdc40771c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:49:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.086+0000 7fcdc8d47700 1 --2- 172.21.15.31:0/2873323142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdc4103480 0x7fcdc40771c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.086+0000 7fcdc8d47700 1 -- 172.21.15.31:0/2873323142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdc4104680 msgr2=0x7fcdc4075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:49:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.086+0000 7fcdc8d47700 1 --2- 172.21.15.31:0/2873323142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdc4104680 0x7fcdc4075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.086+0000 7fcdc8d47700 1 -- 172.21.15.31:0/2873323142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc4076600 con 0x7fcdc4105020 2024-04-26T17:49:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.086+0000 7fcdc8d47700 1 --2- 172.21.15.31:0/2873323142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdc4105020 0x7fcdc4075d50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009040 tx=0x7fcdbc004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:49:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.086+0000 7fcdc17fa700 1 -- 172.21.15.31:0/2873323142 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc01f030 con 0x7fcdc4105020 2024-04-26T17:49:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.086+0000 7fcdc17fa700 1 -- 172.21.15.31:0/2873323142 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdbc003a00 con 0x7fcdc4105020 2024-04-26T17:49:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.086+0000 7fcdca7aa700 1 -- 172.21.15.31:0/2873323142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc41a3640 con 0x7fcdc4105020 2024-04-26T17:49:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.086+0000 7fcdc17fa700 1 -- 172.21.15.31:0/2873323142 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc0046e0 con 0x7fcdc4105020 2024-04-26T17:49:28.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.086+0000 7fcdca7aa700 1 -- 172.21.15.31:0/2873323142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc41a3a90 con 0x7fcdc4105020 2024-04-26T17:49:28.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.087+0000 7fcdca7aa700 1 -- 172.21.15.31:0/2873323142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc406a110 con 0x7fcdc4105020 2024-04-26T17:49:28.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.088+0000 7fcdc17fa700 1 -- 172.21.15.31:0/2873323142 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcdbc003bb0 con 0x7fcdc4105020 2024-04-26T17:49:28.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.089+0000 7fcdc17fa700 1 --2- 172.21.15.31:0/2873323142 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fcdac0a3cc0 0x7fcdac0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:49:28.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.089+0000 7fcdc17fa700 1 -- 172.21.15.31:0/2873323142 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7fcdbc00f020 con 0x7fcdc4105020 2024-04-26T17:49:28.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.089+0000 7fcdc3fff700 1 --2- 172.21.15.31:0/2873323142 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fcdac0a3cc0 0x7fcdac0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:49:28.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.089+0000 7fcdc3fff700 1 --2- 172.21.15.31:0/2873323142 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fcdac0a3cc0 0x7fcdac0a6180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcdc41044e0 tx=0x7fcdb400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:49:28.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.094+0000 7fcdc17fa700 1 -- 172.21.15.31:0/2873323142 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcdbc029020 con 0x7fcdc4105020 2024-04-26T17:49:28.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:28.312+0000 7fcdca7aa700 1 -- 172.21.15.31:0/2873323142 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fcdc4063860 con 0x7fcdac0a3cc0 2024-04-26T17:49:28.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:28 smithi186 ceph-mon[32797]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T17:49:28.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:28 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:49:28.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:28 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:49:28.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:28 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:28.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:28 smithi031 ceph-mon[29164]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T17:49:28.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:28 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:49:28.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:28 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:49:28.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:28 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:28.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:28 smithi097 ceph-mon[32419]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T17:49:28.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:28 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:49:28.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:28 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:49:28.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:28 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:29 smithi031 ceph-mon[29164]: pgmap v140: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T17:49:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:29 smithi031 ceph-mon[29164]: from='client.14340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:49:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:29 smithi031 ceph-mon[29164]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T17:49:29.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:29 smithi186 ceph-mon[32797]: pgmap v140: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T17:49:29.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:29 smithi186 ceph-mon[32797]: from='client.14340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:49:29.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:29 smithi186 ceph-mon[32797]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T17:49:29.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:29 smithi097 ceph-mon[32419]: pgmap v140: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T17:49:29.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:29 smithi097 ceph-mon[32419]: from='client.14340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:49:29.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:29 smithi097 ceph-mon[32419]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T17:49:30.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:30 smithi031 ceph-mon[29164]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T17:49:30.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:30 smithi186 ceph-mon[32797]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T17:49:30.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:30 smithi097 ceph-mon[32419]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T17:49:31.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:31 smithi186 ceph-mon[32797]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T17:49:31.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:31 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/124764559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1d11e38-40e9-4db4-868f-145f1ffc0d30"}]: dispatch 2024-04-26T17:49:31.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:31 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/124764559' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1d11e38-40e9-4db4-868f-145f1ffc0d30"}]': finished 2024-04-26T17:49:31.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:31 smithi186 ceph-mon[32797]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T17:49:31.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:31 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:31.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:31 smithi031 ceph-mon[29164]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T17:49:31.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:31 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/124764559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1d11e38-40e9-4db4-868f-145f1ffc0d30"}]: dispatch 2024-04-26T17:49:31.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:31 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/124764559' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1d11e38-40e9-4db4-868f-145f1ffc0d30"}]': finished 2024-04-26T17:49:31.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:31 smithi031 ceph-mon[29164]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T17:49:31.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:31 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:31 smithi097 ceph-mon[32419]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 203 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T17:49:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:31 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/124764559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1d11e38-40e9-4db4-868f-145f1ffc0d30"}]: dispatch 2024-04-26T17:49:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:31 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/124764559' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1d11e38-40e9-4db4-868f-145f1ffc0d30"}]': finished 2024-04-26T17:49:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:31 smithi097 ceph-mon[32419]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T17:49:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:31 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:32.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:32 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1459346046' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:49:32.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:32 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1459346046' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:49:32.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:32 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1459346046' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:49:33.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:33 smithi186 ceph-mon[32797]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T17:49:33.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:33 smithi031 ceph-mon[29164]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T17:49:33.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:33 smithi097 ceph-mon[32419]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T17:49:35.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:35 smithi186 ceph-mon[32797]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:35.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:35 smithi031 ceph-mon[29164]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:35.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:35 smithi097 ceph-mon[32419]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:37.650 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:37 smithi031 ceph-mon[29164]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:37.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:37 smithi186 ceph-mon[32797]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:37.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:37 smithi097 ceph-mon[32419]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:39.710 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:39 smithi031 ceph-mon[29164]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:39.710 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:39 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T17:49:39.710 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:39 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:39.710 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:39 smithi031 ceph-mon[29164]: Deploying daemon osd.6 on smithi031 2024-04-26T17:49:39.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:39 smithi186 ceph-mon[32797]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:39.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:39 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T17:49:39.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:39 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:39.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:39 smithi186 ceph-mon[32797]: Deploying daemon osd.6 on smithi031 2024-04-26T17:49:39.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:39 smithi097 ceph-mon[32419]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:39.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:39 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T17:49:39.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:39 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:39.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:39 smithi097 ceph-mon[32419]: Deploying daemon osd.6 on smithi031 2024-04-26T17:49:41.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:41 smithi031 ceph-mon[29164]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:41.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:41 smithi186 ceph-mon[32797]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:41.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:41 smithi097 ceph-mon[32419]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:43.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:43 smithi186 ceph-mon[32797]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:43.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:43 smithi031 ceph-mon[29164]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:43.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:43 smithi097 ceph-mon[32419]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:46.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:46 smithi031 ceph-mon[29164]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:46.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:46.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:46.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:46.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:46 smithi097 ceph-mon[32419]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:46.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:46 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:46.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:46 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:46.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:46 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:46.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:46 smithi186 ceph-mon[32797]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:46.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:46.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:46.519 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:48.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:48 smithi097 ceph-mon[32419]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:48.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:48 smithi186 ceph-mon[32797]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:48.542 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:48 smithi031 ceph-mon[29164]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:48.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:48.931+0000 7fcdc17fa700 1 -- 172.21.15.31:0/2873323142 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcdc4063860 con 0x7fcdac0a3cc0 2024-04-26T17:49:48.931 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 6 on host 'smithi031' 2024-04-26T17:49:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:48.934+0000 7fcdca7aa700 1 -- 172.21.15.31:0/2873323142 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fcdac0a3cc0 msgr2=0x7fcdac0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:49:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:48.934+0000 7fcdca7aa700 1 --2- 172.21.15.31:0/2873323142 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fcdac0a3cc0 0x7fcdac0a6180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcdc41044e0 tx=0x7fcdb400b410 comp rx=0 tx=0).stop 2024-04-26T17:49:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:48.934+0000 7fcdca7aa700 1 -- 172.21.15.31:0/2873323142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdc4105020 msgr2=0x7fcdc4075d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:49:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:48.934+0000 7fcdca7aa700 1 --2- 172.21.15.31:0/2873323142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdc4105020 0x7fcdc4075d50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009040 tx=0x7fcdbc004d50 comp rx=0 tx=0).stop 2024-04-26T17:49:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:48.934+0000 7fcdca7aa700 1 -- 172.21.15.31:0/2873323142 shutdown_connections 2024-04-26T17:49:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:48.934+0000 7fcdca7aa700 1 --2- 172.21.15.31:0/2873323142 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fcdac0a3cc0 0x7fcdac0a6180 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:48.934+0000 7fcdca7aa700 1 --2- 172.21.15.31:0/2873323142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdc4105020 0x7fcdc4075d50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:48.934+0000 7fcdca7aa700 1 --2- 172.21.15.31:0/2873323142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdc4103480 0x7fcdc40771c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:48.934+0000 7fcdca7aa700 1 --2- 172.21.15.31:0/2873323142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdc4104680 0x7fcdc4075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:48.934+0000 7fcdca7aa700 1 -- 172.21.15.31:0/2873323142 >> 172.21.15.31:0/2873323142 conn(0x7fcdc40fea20 msgr2=0x7fcdc4107980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:49:48.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:48.934+0000 7fcdca7aa700 1 -- 172.21.15.31:0/2873323142 shutdown_connections 2024-04-26T17:49:48.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:49:48.934+0000 7fcdca7aa700 1 -- 172.21.15.31:0/2873323142 wait complete. 2024-04-26T17:49:49.767 DEBUG:teuthology.orchestra.run.smithi031:osd.6> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T17:49:49.770 INFO:tasks.cephadm:Deploying osd.7 on smithi097 with /dev/vg_nvme/lv_2... 2024-04-26T17:49:49.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T17:49:50.016 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T17:49:50.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:49 smithi031 ceph-mon[29164]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:50.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:50.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:50.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:50.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:50.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:49 smithi031 ceph-mon[29164]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T17:49:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:49 smithi097 ceph-mon[32419]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:49 smithi097 ceph-mon[32419]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T17:49:50.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:49 smithi186 ceph-mon[32797]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:50.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:50.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:50.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:50.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:50.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:49 smithi186 ceph-mon[32797]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T17:49:51.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:50 smithi031 ceph-mon[29164]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T17:49:51.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:50 smithi031 ceph-mon[29164]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T17:49:51.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:50 smithi031 ceph-mon[29164]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-26T17:49:51.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:50 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:51.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:50 smithi186 ceph-mon[32797]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T17:49:51.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:50 smithi186 ceph-mon[32797]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T17:49:51.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:50 smithi186 ceph-mon[32797]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-26T17:49:51.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:50 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:51.268 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:50 smithi097 ceph-mon[32419]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T17:49:51.268 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:50 smithi097 ceph-mon[32419]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T17:49:51.268 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:50 smithi097 ceph-mon[32419]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-26T17:49:51.268 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:50 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:51.809 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 17:49:51 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T17:49:51.494+0000 7f144ce91700 -1 osd.6 0 waiting for initial osdmap 2024-04-26T17:49:51.809 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 17:49:51 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T17:49:51.506+0000 7f1444c7c700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T17:49:52.169 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:51 smithi097 ceph-mon[32419]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:52.169 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:51 smithi097 ceph-mon[32419]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-26T17:49:52.169 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:51 smithi097 ceph-mon[32419]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T17:49:52.169 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:51 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:52.169 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:51 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:52.169 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:51 smithi097 ceph-mon[32419]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' 2024-04-26T17:49:52.169 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T17:49:52.170 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T17:49:52.170 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-04-26T17:49:52.170 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-04-26T17:49:52.170 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128368 s, 817 MB/s 2024-04-26T17:49:52.170 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-04-26T17:49:52.250 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:51 smithi031 ceph-mon[29164]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:52.250 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:51 smithi031 ceph-mon[29164]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-26T17:49:52.250 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:51 smithi031 ceph-mon[29164]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T17:49:52.250 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:51 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:52.251 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:51 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:52.251 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:51 smithi031 ceph-mon[29164]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' 2024-04-26T17:49:52.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:51 smithi186 ceph-mon[32797]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:52.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:51 smithi186 ceph-mon[32797]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-26T17:49:52.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:51 smithi186 ceph-mon[32797]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T17:49:52.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:51 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:52.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:51 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:52.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:51 smithi186 ceph-mon[32797]: from='osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395]' entity='osd.6' 2024-04-26T17:49:52.796 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_2 2024-04-26T17:49:53.035 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T17:49:53.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:53 smithi097 ceph-mon[32419]: purged_snaps scrub starts 2024-04-26T17:49:53.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:53 smithi097 ceph-mon[32419]: purged_snaps scrub ok 2024-04-26T17:49:53.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:53 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:53.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:53 smithi097 ceph-mon[32419]: osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395] boot 2024-04-26T17:49:53.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:53 smithi097 ceph-mon[32419]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T17:49:53.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:53 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:53.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:53 smithi186 ceph-mon[32797]: purged_snaps scrub starts 2024-04-26T17:49:53.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:53 smithi186 ceph-mon[32797]: purged_snaps scrub ok 2024-04-26T17:49:53.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:53 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:53.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:53 smithi186 ceph-mon[32797]: osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395] boot 2024-04-26T17:49:53.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:53 smithi186 ceph-mon[32797]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T17:49:53.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:53 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:53.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:53 smithi031 ceph-mon[29164]: purged_snaps scrub starts 2024-04-26T17:49:53.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:53 smithi031 ceph-mon[29164]: purged_snaps scrub ok 2024-04-26T17:49:53.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:53 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:53.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:53 smithi031 ceph-mon[29164]: osd.6 [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395] boot 2024-04-26T17:49:53.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:53 smithi031 ceph-mon[29164]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T17:49:53.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:53 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:49:54.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:54 smithi097 ceph-mon[32419]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:54.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:54.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:54.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:54.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:54.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:54.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:54.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:54.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:54.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:54 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:54 smithi186 ceph-mon[32797]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:54.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:54 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:54 smithi031 ceph-mon[29164]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-26T17:49:54.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:54.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:54.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:49:54.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:54.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:54.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:49:54.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:54.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:49:54.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:54 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:49:54.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.673+0000 7f2a2d514700 1 -- 172.21.15.97:0/3644700048 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a28106840 msgr2=0x7f2a28108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:49:54.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.673+0000 7f2a2d514700 1 --2- 172.21.15.97:0/3644700048 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a28106840 0x7f2a28108c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009990 tx=0x7f2a10009ca0 comp rx=0 tx=0).stop 2024-04-26T17:49:54.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.673+0000 7f2a2d514700 1 -- 172.21.15.97:0/3644700048 shutdown_connections 2024-04-26T17:49:54.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.673+0000 7f2a2d514700 1 --2- 172.21.15.97:0/3644700048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a28109170 0x7f2a2810b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:54.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.673+0000 7f2a2d514700 1 --2- 172.21.15.97:0/3644700048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a28103f10 0x7f2a28106300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:54.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.673+0000 7f2a2d514700 1 --2- 172.21.15.97:0/3644700048 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a28106840 0x7f2a28108c30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:54.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.673+0000 7f2a2d514700 1 -- 172.21.15.97:0/3644700048 >> 172.21.15.97:0/3644700048 conn(0x7f2a280fd920 msgr2=0x7f2a280ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:49:54.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.673+0000 7f2a2d514700 1 -- 172.21.15.97:0/3644700048 shutdown_connections 2024-04-26T17:49:54.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.673+0000 7f2a2d514700 1 -- 172.21.15.97:0/3644700048 wait complete. 2024-04-26T17:49:54.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.673+0000 7f2a2d514700 1 Processor -- start 2024-04-26T17:49:54.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a2d514700 1 -- start start 2024-04-26T17:49:54.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a2d514700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a28103f10 0x7f2a28199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:49:54.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a2d514700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a28106840 0x7f2a28199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:49:54.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a2d514700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a28109170 0x7f2a281a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:49:54.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a2d514700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2810f000 con 0x7f2a28103f10 2024-04-26T17:49:54.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a2d514700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2810ee80 con 0x7f2a28109170 2024-04-26T17:49:54.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a2d514700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2810f180 con 0x7f2a28106840 2024-04-26T17:49:54.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a277fe700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a28109170 0x7f2a281a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:49:54.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a277fe700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a28109170 0x7f2a281a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57302/0 (socket says 172.21.15.97:57302) 2024-04-26T17:49:54.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a277fe700 1 -- 172.21.15.97:0/1635597688 learned_addr learned my addr 172.21.15.97:0/1635597688 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-04-26T17:49:54.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a267fc700 1 --2- 172.21.15.97:0/1635597688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a28106840 0x7f2a28199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:49:54.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a26ffd700 1 --2- 172.21.15.97:0/1635597688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a28103f10 0x7f2a28199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:49:54.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a277fe700 1 -- 172.21.15.97:0/1635597688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a28106840 msgr2=0x7f2a28199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:49:54.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a277fe700 1 --2- 172.21.15.97:0/1635597688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a28106840 0x7f2a28199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:54.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a277fe700 1 -- 172.21.15.97:0/1635597688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a28103f10 msgr2=0x7f2a28199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:49:54.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a277fe700 1 --2- 172.21.15.97:0/1635597688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a28103f10 0x7f2a28199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:49:54.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a277fe700 1 -- 172.21.15.97:0/1635597688 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a281a13b0 con 0x7f2a28109170 2024-04-26T17:49:54.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.674+0000 7f2a277fe700 1 --2- 172.21.15.97:0/1635597688 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a28109170 0x7f2a281a0d10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2a2000b480 tx=0x7f2a2000b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:49:54.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.675+0000 7f2a1ffff700 1 -- 172.21.15.97:0/1635597688 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a20015020 con 0x7f2a28109170 2024-04-26T17:49:54.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.675+0000 7f2a2d514700 1 -- 172.21.15.97:0/1635597688 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a281a16a0 con 0x7f2a28109170 2024-04-26T17:49:54.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.675+0000 7f2a1ffff700 1 -- 172.21.15.97:0/1635597688 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a20011ce0 con 0x7f2a28109170 2024-04-26T17:49:54.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.675+0000 7f2a1ffff700 1 -- 172.21.15.97:0/1635597688 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a20010d70 con 0x7f2a28109170 2024-04-26T17:49:54.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.675+0000 7f2a2d514700 1 -- 172.21.15.97:0/1635597688 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a281a1c60 con 0x7f2a28109170 2024-04-26T17:49:54.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.676+0000 7f2a2d514700 1 -- 172.21.15.97:0/1635597688 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a2806a110 con 0x7f2a28109170 2024-04-26T17:49:54.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.677+0000 7f2a1ffff700 1 -- 172.21.15.97:0/1635597688 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2a20018020 con 0x7f2a28109170 2024-04-26T17:49:54.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.677+0000 7f2a1ffff700 1 --2- 172.21.15.97:0/1635597688 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2a140a3c40 0x7f2a140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:49:54.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.677+0000 7f2a1ffff700 1 -- 172.21.15.97:0/1635597688 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f2a200cf2d0 con 0x7f2a28109170 2024-04-26T17:49:54.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.677+0000 7f2a26ffd700 1 --2- 172.21.15.97:0/1635597688 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2a140a3c40 0x7f2a140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:49:54.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.678+0000 7f2a26ffd700 1 --2- 172.21.15.97:0/1635597688 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2a140a3c40 0x7f2a140a6100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009f20 tx=0x7f2a180094f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:49:54.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.681+0000 7f2a1ffff700 1 -- 172.21.15.97:0/1635597688 <== mon.2 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2a2001f020 con 0x7f2a28109170 2024-04-26T17:49:54.889 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:49:54.889+0000 7f2a2d514700 1 -- 172.21.15.97:0/1635597688 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f2a28063860 con 0x7f2a140a3c40 2024-04-26T17:49:55.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:55 smithi031 ceph-mon[29164]: Detected new or changed devices on smithi031 2024-04-26T17:49:55.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:55 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi031 to 5697M 2024-04-26T17:49:55.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:55 smithi031 ceph-mon[29164]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T17:49:55.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:55 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:49:55.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:55 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:49:55.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:55 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:55.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:55 smithi097 ceph-mon[32419]: Detected new or changed devices on smithi031 2024-04-26T17:49:55.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:55 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi031 to 5697M 2024-04-26T17:49:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:55 smithi097 ceph-mon[32419]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T17:49:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:55 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:49:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:55 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:49:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:55 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:55.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:55 smithi186 ceph-mon[32797]: Detected new or changed devices on smithi031 2024-04-26T17:49:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:55 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi031 to 5697M 2024-04-26T17:49:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:55 smithi186 ceph-mon[32797]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T17:49:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:55 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:49:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:55 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:49:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:55 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:49:56.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:56 smithi031 ceph-mon[29164]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:49:56.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:56 smithi031 ceph-mon[29164]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:49:56.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:56 smithi097 ceph-mon[32419]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:49:56.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:56 smithi097 ceph-mon[32419]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:49:56.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:56 smithi186 ceph-mon[32797]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:49:56.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:56 smithi186 ceph-mon[32797]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:49:58.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:58 smithi031 ceph-mon[29164]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:49:58.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:58 smithi031 ceph-mon[29164]: from='client.? 172.21.15.97:0/3702069566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e95f728-23ff-4bca-9b94-afde2662a5bf"}]: dispatch 2024-04-26T17:49:58.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:58 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e95f728-23ff-4bca-9b94-afde2662a5bf"}]: dispatch 2024-04-26T17:49:58.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:58 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e95f728-23ff-4bca-9b94-afde2662a5bf"}]': finished 2024-04-26T17:49:58.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:58 smithi031 ceph-mon[29164]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T17:49:58.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:49:58.319 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:58 smithi186 ceph-mon[32797]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:49:58.319 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:58 smithi186 ceph-mon[32797]: from='client.? 172.21.15.97:0/3702069566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e95f728-23ff-4bca-9b94-afde2662a5bf"}]: dispatch 2024-04-26T17:49:58.319 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:58 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e95f728-23ff-4bca-9b94-afde2662a5bf"}]: dispatch 2024-04-26T17:49:58.319 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:58 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e95f728-23ff-4bca-9b94-afde2662a5bf"}]': finished 2024-04-26T17:49:58.319 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:58 smithi186 ceph-mon[32797]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T17:49:58.320 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:49:58.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:58 smithi097 ceph-mon[32419]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:49:58.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:58 smithi097 ceph-mon[32419]: from='client.? 172.21.15.97:0/3702069566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e95f728-23ff-4bca-9b94-afde2662a5bf"}]: dispatch 2024-04-26T17:49:58.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:58 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e95f728-23ff-4bca-9b94-afde2662a5bf"}]: dispatch 2024-04-26T17:49:58.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:58 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e95f728-23ff-4bca-9b94-afde2662a5bf"}]': finished 2024-04-26T17:49:58.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:58 smithi097 ceph-mon[32419]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T17:49:58.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:49:59.285 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:49:59 smithi097 ceph-mon[32419]: from='client.? 172.21.15.97:0/568932913' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:49:59.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:49:59 smithi031 ceph-mon[29164]: from='client.? 172.21.15.97:0/568932913' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:49:59.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:49:59 smithi186 ceph-mon[32797]: from='client.? 172.21.15.97:0/568932913' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:50:00.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:00 smithi031 ceph-mon[29164]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:00.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:00 smithi031 ceph-mon[29164]: overall HEALTH_OK 2024-04-26T17:50:00.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:00 smithi097 ceph-mon[32419]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:00.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:00 smithi097 ceph-mon[32419]: overall HEALTH_OK 2024-04-26T17:50:00.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:00 smithi186 ceph-mon[32797]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:00.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:00 smithi186 ceph-mon[32797]: overall HEALTH_OK 2024-04-26T17:50:02.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:02 smithi031 ceph-mon[29164]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:02.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:02 smithi097 ceph-mon[32419]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:02.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:02 smithi186 ceph-mon[32797]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:04.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:04 smithi097 ceph-mon[32419]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:04.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:04 smithi031 ceph-mon[29164]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:04.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:04 smithi186 ceph-mon[32797]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:05.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:05 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T17:50:05.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:05 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:05.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:05 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T17:50:05.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:05 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:05 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T17:50:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:05 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:06.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:06 smithi097 ceph-mon[32419]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:06.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:06 smithi097 ceph-mon[32419]: Deploying daemon osd.7 on smithi097 2024-04-26T17:50:06.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:06 smithi031 ceph-mon[29164]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:06.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:06 smithi031 ceph-mon[29164]: Deploying daemon osd.7 on smithi097 2024-04-26T17:50:06.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:06 smithi186 ceph-mon[32797]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:06.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:06 smithi186 ceph-mon[32797]: Deploying daemon osd.7 on smithi097 2024-04-26T17:50:08.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:08 smithi031 ceph-mon[29164]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:08.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:08 smithi097 ceph-mon[32419]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:08.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:08 smithi186 ceph-mon[32797]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:10.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:10 smithi097 ceph-mon[32419]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:10.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:10 smithi031 ceph-mon[29164]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:10.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:10 smithi186 ceph-mon[32797]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:11 smithi097 ceph-mon[32419]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:11.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:11 smithi186 ceph-mon[32797]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:11.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:11.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:11.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:11.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:11 smithi031 ceph-mon[29164]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:11.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:11.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:11.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:13.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:13 smithi186 ceph-mon[32797]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:13.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:13 smithi031 ceph-mon[29164]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:13.865 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:13 smithi097 ceph-mon[32419]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:13.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:50:13.912+0000 7f2a1ffff700 1 -- 172.21.15.97:0/1635597688 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2a28063860 con 0x7f2a140a3c40 2024-04-26T17:50:13.912 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 7 on host 'smithi097' 2024-04-26T17:50:13.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:50:13.915+0000 7f2a2d514700 1 -- 172.21.15.97:0/1635597688 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2a140a3c40 msgr2=0x7f2a140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:50:13.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:50:13.915+0000 7f2a2d514700 1 --2- 172.21.15.97:0/1635597688 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2a140a3c40 0x7f2a140a6100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009f20 tx=0x7f2a180094f0 comp rx=0 tx=0).stop 2024-04-26T17:50:13.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:50:13.915+0000 7f2a2d514700 1 -- 172.21.15.97:0/1635597688 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a28109170 msgr2=0x7f2a281a0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:50:13.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:50:13.915+0000 7f2a2d514700 1 --2- 172.21.15.97:0/1635597688 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a28109170 0x7f2a281a0d10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2a2000b480 tx=0x7f2a2000b790 comp rx=0 tx=0).stop 2024-04-26T17:50:13.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:50:13.915+0000 7f2a2d514700 1 -- 172.21.15.97:0/1635597688 shutdown_connections 2024-04-26T17:50:13.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:50:13.915+0000 7f2a2d514700 1 --2- 172.21.15.97:0/1635597688 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f2a140a3c40 0x7f2a140a6100 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:13.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:50:13.915+0000 7f2a2d514700 1 --2- 172.21.15.97:0/1635597688 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2a28109170 0x7f2a281a0d10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:13.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:50:13.915+0000 7f2a2d514700 1 --2- 172.21.15.97:0/1635597688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a28103f10 0x7f2a28199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:13.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:50:13.915+0000 7f2a2d514700 1 --2- 172.21.15.97:0/1635597688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a28106840 0x7f2a28199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:13.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:50:13.915+0000 7f2a2d514700 1 -- 172.21.15.97:0/1635597688 >> 172.21.15.97:0/1635597688 conn(0x7f2a280fd920 msgr2=0x7f2a28104930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:50:13.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:50:13.915+0000 7f2a2d514700 1 -- 172.21.15.97:0/1635597688 shutdown_connections 2024-04-26T17:50:13.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:50:13.915+0000 7f2a2d514700 1 -- 172.21.15.97:0/1635597688 wait complete. 2024-04-26T17:50:14.597 DEBUG:teuthology.orchestra.run.smithi097:osd.7> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.7.service 2024-04-26T17:50:14.600 INFO:tasks.cephadm:Deploying osd.8 on smithi186 with /dev/vg_nvme/lv_2... 2024-04-26T17:50:14.600 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T17:50:14.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:14.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:14.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:14.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:14 smithi186 ceph-mon[32797]: from='osd.7 [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T17:50:14.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:14 smithi186 ceph-mon[32797]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T17:50:14.784 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.784 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.784 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:14.784 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:14.784 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:14.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:14 smithi097 ceph-mon[32419]: from='osd.7 [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T17:50:14.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:14 smithi097 ceph-mon[32419]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T17:50:14.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:14.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:14.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:14.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:14.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:14 smithi031 ceph-mon[29164]: from='osd.7 [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T17:50:14.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:14 smithi031 ceph-mon[29164]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T17:50:14.848 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T17:50:16.041 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:15 smithi097 ceph-mon[32419]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:16.042 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:15 smithi097 ceph-mon[32419]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T17:50:16.042 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:15 smithi097 ceph-mon[32419]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T17:50:16.042 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:15 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:50:16.042 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:15 smithi097 ceph-mon[32419]: from='osd.7 [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:50:16.042 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:15 smithi097 ceph-mon[32419]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:50:16.042 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 17:50:15 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T17:50:15.916+0000 7f61bdedb700 -1 osd.7 0 waiting for initial osdmap 2024-04-26T17:50:16.042 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 17:50:15 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T17:50:15.926+0000 7f61b9cce700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T17:50:16.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:15 smithi186 ceph-mon[32797]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:16.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:15 smithi186 ceph-mon[32797]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T17:50:16.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:15 smithi186 ceph-mon[32797]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T17:50:16.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:15 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:50:16.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:15 smithi186 ceph-mon[32797]: from='osd.7 [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:50:16.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:15 smithi186 ceph-mon[32797]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:50:16.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:15 smithi031 ceph-mon[29164]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:16.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:15 smithi031 ceph-mon[29164]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T17:50:16.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:15 smithi031 ceph-mon[29164]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T17:50:16.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:15 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:50:16.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:15 smithi031 ceph-mon[29164]: from='osd.7 [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:50:16.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:15 smithi031 ceph-mon[29164]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:50:16.962 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T17:50:16.963 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T17:50:16.963 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-26T17:50:16.963 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-26T17:50:16.963 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114959 s, 912 MB/s 2024-04-26T17:50:16.963 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-26T17:50:17.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:16 smithi097 ceph-mon[32419]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-26T17:50:17.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:16 smithi097 ceph-mon[32419]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T17:50:17.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:16 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:50:17.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:16 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:50:17.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:16 smithi097 ceph-mon[32419]: osd.7 [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075] boot 2024-04-26T17:50:17.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:16 smithi097 ceph-mon[32419]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T17:50:17.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:16 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:50:17.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:16 smithi186 ceph-mon[32797]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-26T17:50:17.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:16 smithi186 ceph-mon[32797]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T17:50:17.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:16 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:50:17.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:16 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:50:17.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:16 smithi186 ceph-mon[32797]: osd.7 [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075] boot 2024-04-26T17:50:17.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:16 smithi186 ceph-mon[32797]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T17:50:17.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:16 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:50:17.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:16 smithi031 ceph-mon[29164]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-26T17:50:17.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:16 smithi031 ceph-mon[29164]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T17:50:17.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:50:17.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:50:17.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:16 smithi031 ceph-mon[29164]: osd.7 [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075] boot 2024-04-26T17:50:17.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:16 smithi031 ceph-mon[29164]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T17:50:17.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:50:17.575 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_2 2024-04-26T17:50:17.813 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T17:50:18.200 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:17 smithi186 ceph-mon[32797]: purged_snaps scrub starts 2024-04-26T17:50:18.200 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:17 smithi186 ceph-mon[32797]: purged_snaps scrub ok 2024-04-26T17:50:18.200 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:17 smithi186 ceph-mon[32797]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:18.201 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:18.201 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:18.201 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:18.201 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:18.201 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:18.201 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:18.201 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:18.201 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:18.201 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:18.201 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:17 smithi186 ceph-mon[32797]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T17:50:18.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:17 smithi031 ceph-mon[29164]: purged_snaps scrub starts 2024-04-26T17:50:18.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:17 smithi031 ceph-mon[29164]: purged_snaps scrub ok 2024-04-26T17:50:18.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:17 smithi031 ceph-mon[29164]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:18.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:18.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:18.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:18.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:18.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:18.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:18.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:18.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:18.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:18.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:17 smithi031 ceph-mon[29164]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T17:50:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:17 smithi097 ceph-mon[32419]: purged_snaps scrub starts 2024-04-26T17:50:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:17 smithi097 ceph-mon[32419]: purged_snaps scrub ok 2024-04-26T17:50:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:17 smithi097 ceph-mon[32419]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-26T17:50:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:18.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:18.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:18.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:18.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:17 smithi097 ceph-mon[32419]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T17:50:19.240 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:18 smithi186 ceph-mon[32797]: Detected new or changed devices on smithi097 2024-04-26T17:50:19.240 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:18 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi097 to 5696M 2024-04-26T17:50:19.240 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:18 smithi186 ceph-mon[32797]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T17:50:19.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:18 smithi031 ceph-mon[29164]: Detected new or changed devices on smithi097 2024-04-26T17:50:19.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:18 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi097 to 5696M 2024-04-26T17:50:19.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:18 smithi031 ceph-mon[29164]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T17:50:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:18 smithi097 ceph-mon[32419]: Detected new or changed devices on smithi097 2024-04-26T17:50:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:18 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi097 to 5696M 2024-04-26T17:50:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:18 smithi097 ceph-mon[32419]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T17:50:19.483 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.479+0000 7f094226d700 1 -- 172.21.15.186:0/3470141042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c1034b0 msgr2=0x7f093c1038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:50:19.483 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.479+0000 7f094226d700 1 --2- 172.21.15.186:0/3470141042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c1034b0 0x7f093c1038d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0930009990 tx=0x7f0930009ca0 comp rx=0 tx=0).stop 2024-04-26T17:50:19.483 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.479+0000 7f094226d700 1 -- 172.21.15.186:0/3470141042 shutdown_connections 2024-04-26T17:50:19.484 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.479+0000 7f094226d700 1 --2- 172.21.15.186:0/3470141042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c105050 0x7f093c10b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:19.484 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.479+0000 7f094226d700 1 --2- 172.21.15.186:0/3470141042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c1034b0 0x7f093c1038d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:19.484 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.479+0000 7f094226d700 1 --2- 172.21.15.186:0/3470141042 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f093c1046b0 0x7f093c104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:19.484 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.479+0000 7f094226d700 1 -- 172.21.15.186:0/3470141042 >> 172.21.15.186:0/3470141042 conn(0x7f093c0fea70 msgr2=0x7f093c100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:50:19.484 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.479+0000 7f094226d700 1 -- 172.21.15.186:0/3470141042 shutdown_connections 2024-04-26T17:50:19.484 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.479+0000 7f094226d700 1 -- 172.21.15.186:0/3470141042 wait complete. 2024-04-26T17:50:19.484 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.480+0000 7f094226d700 1 Processor -- start 2024-04-26T17:50:19.484 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.480+0000 7f094226d700 1 -- start start 2024-04-26T17:50:19.485 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.480+0000 7f094226d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c1034b0 0x7f093c199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:50:19.485 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.480+0000 7f094226d700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f093c1046b0 0x7f093c199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:50:19.485 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.480+0000 7f094226d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c105050 0x7f093c1a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:50:19.485 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.480+0000 7f094226d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f093c10efb0 con 0x7f093c105050 2024-04-26T17:50:19.485 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.480+0000 7f094226d700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f093c10ee30 con 0x7f093c1046b0 2024-04-26T17:50:19.485 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.480+0000 7f094226d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f093c10f130 con 0x7f093c1034b0 2024-04-26T17:50:19.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.480+0000 7f093bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c1034b0 0x7f093c199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:50:19.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f093bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c1034b0 0x7f093c199650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:60188/0 (socket says 172.21.15.186:60188) 2024-04-26T17:50:19.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f093bfff700 1 -- 172.21.15.186:0/3942355567 learned_addr learned my addr 172.21.15.186:0/3942355567 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T17:50:19.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f094080a700 1 --2- 172.21.15.186:0/3942355567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c105050 0x7f093c1a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:50:19.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f093b7fe700 1 --2- 172.21.15.186:0/3942355567 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f093c1046b0 0x7f093c199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:50:19.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f093bfff700 1 -- 172.21.15.186:0/3942355567 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f093c1046b0 msgr2=0x7f093c199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:50:19.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f093bfff700 1 --2- 172.21.15.186:0/3942355567 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f093c1046b0 0x7f093c199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:19.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f093bfff700 1 -- 172.21.15.186:0/3942355567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c105050 msgr2=0x7f093c1a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:50:19.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f093bfff700 1 --2- 172.21.15.186:0/3942355567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c105050 0x7f093c1a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:19.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f093bfff700 1 -- 172.21.15.186:0/3942355567 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f093c1a12b0 con 0x7f093c1034b0 2024-04-26T17:50:19.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f093bfff700 1 --2- 172.21.15.186:0/3942355567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c1034b0 0x7f093c199650 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f09300047b0 tx=0x7f0930005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:50:19.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f09397fa700 1 -- 172.21.15.186:0/3942355567 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093000fbc0 con 0x7f093c1034b0 2024-04-26T17:50:19.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f094226d700 1 -- 172.21.15.186:0/3942355567 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f093c1a1540 con 0x7f093c1034b0 2024-04-26T17:50:19.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f09397fa700 1 -- 172.21.15.186:0/3942355567 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f093000fd60 con 0x7f093c1034b0 2024-04-26T17:50:19.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f09397fa700 1 -- 172.21.15.186:0/3942355567 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0930017650 con 0x7f093c1034b0 2024-04-26T17:50:19.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.481+0000 7f094226d700 1 -- 172.21.15.186:0/3942355567 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f093c1a1930 con 0x7f093c1034b0 2024-04-26T17:50:19.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.482+0000 7f094226d700 1 -- 172.21.15.186:0/3942355567 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f093c06a110 con 0x7f093c1034b0 2024-04-26T17:50:19.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.483+0000 7f09397fa700 1 -- 172.21.15.186:0/3942355567 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0930009d10 con 0x7f093c1034b0 2024-04-26T17:50:19.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.484+0000 7f09397fa700 1 --2- 172.21.15.186:0/3942355567 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f09240a3c40 0x7f09240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:50:19.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.484+0000 7f09397fa700 1 -- 172.21.15.186:0/3942355567 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f09300d0e30 con 0x7f093c1034b0 2024-04-26T17:50:19.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.484+0000 7f093b7fe700 1 --2- 172.21.15.186:0/3942355567 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f09240a3c40 0x7f09240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:50:19.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.484+0000 7f093b7fe700 1 --2- 172.21.15.186:0/3942355567 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f09240a3c40 0x7f09240a6100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f092c009c60 tx=0x7f092c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:50:19.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.487+0000 7f09397fa700 1 -- 172.21.15.186:0/3942355567 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0930013070 con 0x7f093c1034b0 2024-04-26T17:50:19.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:19.694+0000 7f094226d700 1 -- 172.21.15.186:0/3942355567 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f093c063860 con 0x7f09240a3c40 2024-04-26T17:50:20.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:19 smithi186 ceph-mon[32797]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:50:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:50:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:20.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:19 smithi031 ceph-mon[29164]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:20.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:50:20.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:50:20.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:20.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:19 smithi097 ceph-mon[32419]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:20.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:50:20.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:50:20.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:21.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:20 smithi186 ceph-mon[32797]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:50:21.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:20 smithi186 ceph-mon[32797]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T17:50:21.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:20 smithi031 ceph-mon[29164]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:50:21.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:20 smithi031 ceph-mon[29164]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T17:50:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:20 smithi097 ceph-mon[32419]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:50:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:20 smithi097 ceph-mon[32419]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T17:50:22.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:21 smithi186 ceph-mon[32797]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:22.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:21 smithi186 ceph-mon[32797]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T17:50:22.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:21 smithi031 ceph-mon[29164]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:22.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:21 smithi031 ceph-mon[29164]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T17:50:22.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:21 smithi097 ceph-mon[32419]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:22.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:21 smithi097 ceph-mon[32419]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T17:50:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:22 smithi186 ceph-mon[32797]: from='client.? 172.21.15.186:0/2978652540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f91c915-2568-4d20-b645-00dad2efe952"}]: dispatch 2024-04-26T17:50:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:22 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f91c915-2568-4d20-b645-00dad2efe952"}]: dispatch 2024-04-26T17:50:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:22 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f91c915-2568-4d20-b645-00dad2efe952"}]': finished 2024-04-26T17:50:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:22 smithi186 ceph-mon[32797]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T17:50:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:22 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:23.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:22 smithi031 ceph-mon[29164]: from='client.? 172.21.15.186:0/2978652540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f91c915-2568-4d20-b645-00dad2efe952"}]: dispatch 2024-04-26T17:50:23.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:22 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f91c915-2568-4d20-b645-00dad2efe952"}]: dispatch 2024-04-26T17:50:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:22 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f91c915-2568-4d20-b645-00dad2efe952"}]': finished 2024-04-26T17:50:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:22 smithi031 ceph-mon[29164]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T17:50:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:22 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:23.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:22 smithi097 ceph-mon[32419]: from='client.? 172.21.15.186:0/2978652540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f91c915-2568-4d20-b645-00dad2efe952"}]: dispatch 2024-04-26T17:50:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:22 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f91c915-2568-4d20-b645-00dad2efe952"}]: dispatch 2024-04-26T17:50:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:22 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f91c915-2568-4d20-b645-00dad2efe952"}]': finished 2024-04-26T17:50:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:22 smithi097 ceph-mon[32419]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T17:50:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:22 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:24.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:23 smithi186 ceph-mon[32797]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:24.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:23 smithi186 ceph-mon[32797]: from='client.? 172.21.15.186:0/3149567900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:50:24.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:23 smithi031 ceph-mon[29164]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:24.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:23 smithi031 ceph-mon[29164]: from='client.? 172.21.15.186:0/3149567900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:50:24.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:23 smithi097 ceph-mon[32419]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:24.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:23 smithi097 ceph-mon[32419]: from='client.? 172.21.15.186:0/3149567900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:50:26.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:25 smithi186 ceph-mon[32797]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:26.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:25 smithi031 ceph-mon[29164]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:26.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:25 smithi097 ceph-mon[32419]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:28.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:27 smithi186 ceph-mon[32797]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:28.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:27 smithi031 ceph-mon[29164]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:28.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:27 smithi097 ceph-mon[32419]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:30.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:29 smithi186 ceph-mon[32797]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:30.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:29 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T17:50:30.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:29 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:30.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:29 smithi031 ceph-mon[29164]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:30.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:29 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T17:50:30.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:29 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:30.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:29 smithi097 ceph-mon[32419]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:30.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:29 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T17:50:30.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:29 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:31.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:30 smithi186 ceph-mon[32797]: Deploying daemon osd.8 on smithi186 2024-04-26T17:50:31.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:30 smithi031 ceph-mon[29164]: Deploying daemon osd.8 on smithi186 2024-04-26T17:50:31.380 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:30 smithi097 ceph-mon[32419]: Deploying daemon osd.8 on smithi186 2024-04-26T17:50:32.033 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:31 smithi186 ceph-mon[32797]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:32.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:31 smithi031 ceph-mon[29164]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:32.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:31 smithi097 ceph-mon[32419]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:34.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:33 smithi186 ceph-mon[32797]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:34.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:33 smithi031 ceph-mon[29164]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:34.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:33 smithi097 ceph-mon[32419]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:35 smithi186 ceph-mon[32797]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:36.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:35 smithi031 ceph-mon[29164]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:36.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:36.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:36.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:36.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:35 smithi097 ceph-mon[32419]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:38.230 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:37 smithi186 ceph-mon[32797]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:38.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:37 smithi031 ceph-mon[29164]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:38.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:37 smithi097 ceph-mon[32419]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:39.263 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 8 on host 'smithi186' 2024-04-26T17:50:39.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:39.264+0000 7f09397fa700 1 -- 172.21.15.186:0/3942355567 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f093c063860 con 0x7f09240a3c40 2024-04-26T17:50:39.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:39.266+0000 7f094226d700 1 -- 172.21.15.186:0/3942355567 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f09240a3c40 msgr2=0x7f09240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:50:39.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:39.266+0000 7f094226d700 1 --2- 172.21.15.186:0/3942355567 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f09240a3c40 0x7f09240a6100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f092c009c60 tx=0x7f092c009260 comp rx=0 tx=0).stop 2024-04-26T17:50:39.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:39.266+0000 7f094226d700 1 -- 172.21.15.186:0/3942355567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c1034b0 msgr2=0x7f093c199650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:50:39.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:39.266+0000 7f094226d700 1 --2- 172.21.15.186:0/3942355567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c1034b0 0x7f093c199650 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f09300047b0 tx=0x7f0930005e80 comp rx=0 tx=0).stop 2024-04-26T17:50:39.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:39.266+0000 7f094226d700 1 -- 172.21.15.186:0/3942355567 shutdown_connections 2024-04-26T17:50:39.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:39.266+0000 7f094226d700 1 --2- 172.21.15.186:0/3942355567 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f09240a3c40 0x7f09240a6100 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:39.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:39.266+0000 7f094226d700 1 --2- 172.21.15.186:0/3942355567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c105050 0x7f093c1a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:39.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:39.266+0000 7f094226d700 1 --2- 172.21.15.186:0/3942355567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f093c1034b0 0x7f093c199650 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:39.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:39.266+0000 7f094226d700 1 --2- 172.21.15.186:0/3942355567 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f093c1046b0 0x7f093c199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:39.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:39.266+0000 7f094226d700 1 -- 172.21.15.186:0/3942355567 >> 172.21.15.186:0/3942355567 conn(0x7f093c0fea70 msgr2=0x7f093c107a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:50:39.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:39.266+0000 7f094226d700 1 -- 172.21.15.186:0/3942355567 shutdown_connections 2024-04-26T17:50:39.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:50:39.266+0000 7f094226d700 1 -- 172.21.15.186:0/3942355567 wait complete. 2024-04-26T17:50:39.942 DEBUG:teuthology.orchestra.run.smithi186:osd.8> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.8.service 2024-04-26T17:50:39.946 INFO:tasks.cephadm:Deploying osd.9 on smithi031 with /dev/vg_nvme/lv_1... 2024-04-26T17:50:39.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T17:50:40.204 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:50:40.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:39 smithi186 ceph-mon[32797]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:40.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:39 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:40.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:39 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:40.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:39 smithi186 ceph-mon[32797]: from='osd.8 [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T17:50:40.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:39 smithi186 ceph-mon[32797]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T17:50:40.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:39 smithi031 ceph-mon[29164]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:40.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:39 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:40.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:39 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:40.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:39 smithi031 ceph-mon[29164]: from='osd.8 [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T17:50:40.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:39 smithi031 ceph-mon[29164]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T17:50:40.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:39 smithi097 ceph-mon[32419]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:40.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:39 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:40.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:39 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:40.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:39 smithi097 ceph-mon[32419]: from='osd.8 [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T17:50:40.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:39 smithi097 ceph-mon[32419]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T17:50:41.255 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:41 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:41.255 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:41 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:41.255 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:41 smithi186 ceph-mon[32797]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T17:50:41.255 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:41 smithi186 ceph-mon[32797]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T17:50:41.255 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:41 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:41.255 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:41 smithi186 ceph-mon[32797]: from='osd.8 [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:50:41.255 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:41 smithi186 ceph-mon[32797]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:50:41.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:41 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:41.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:41 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:41.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:41 smithi031 ceph-mon[29164]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T17:50:41.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:41 smithi031 ceph-mon[29164]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T17:50:41.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:41 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:41.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:41 smithi031 ceph-mon[29164]: from='osd.8 [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:50:41.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:41 smithi031 ceph-mon[29164]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:50:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:41 smithi097 ceph-mon[32419]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T17:50:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:41 smithi097 ceph-mon[32419]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T17:50:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:41 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:41 smithi097 ceph-mon[32419]: from='osd.8 [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:50:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:41 smithi097 ceph-mon[32419]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:50:41.517 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 17:50:41 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T17:50:41.255+0000 7fb881b82700 -1 osd.8 0 waiting for initial osdmap 2024-04-26T17:50:41.517 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 17:50:41 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T17:50:41.265+0000 7fb87d975700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T17:50:42.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:42 smithi186 ceph-mon[32797]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:42.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:42 smithi186 ceph-mon[32797]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T17:50:42.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:42 smithi186 ceph-mon[32797]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T17:50:42.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:42 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:42.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:42 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:42.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:42 smithi031 ceph-mon[29164]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:42.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:42 smithi031 ceph-mon[29164]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T17:50:42.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:42 smithi031 ceph-mon[29164]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T17:50:42.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:42 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:42.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:42 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:42.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:42 smithi097 ceph-mon[32419]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-26T17:50:42.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:42 smithi097 ceph-mon[32419]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T17:50:42.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:42 smithi097 ceph-mon[32419]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T17:50:42.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:42 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:42.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:42 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:42.411 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T17:50:42.412 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T17:50:42.412 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-04-26T17:50:42.412 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-04-26T17:50:42.412 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111315 s, 942 MB/s 2024-04-26T17:50:42.412 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-04-26T17:50:43.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_1 2024-04-26T17:50:43.441 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:43 smithi186 ceph-mon[32797]: purged_snaps scrub starts 2024-04-26T17:50:43.441 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:43 smithi186 ceph-mon[32797]: purged_snaps scrub ok 2024-04-26T17:50:43.441 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:43 smithi186 ceph-mon[32797]: osd.8 [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170] boot 2024-04-26T17:50:43.441 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:43 smithi186 ceph-mon[32797]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T17:50:43.441 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:43 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:43.441 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:43 smithi186 ceph-mon[32797]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:50:43.540 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:50:43.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:43 smithi031 ceph-mon[29164]: purged_snaps scrub starts 2024-04-26T17:50:43.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:43 smithi031 ceph-mon[29164]: purged_snaps scrub ok 2024-04-26T17:50:43.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:43 smithi031 ceph-mon[29164]: osd.8 [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170] boot 2024-04-26T17:50:43.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:43 smithi031 ceph-mon[29164]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T17:50:43.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:43 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:43.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:43 smithi031 ceph-mon[29164]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:50:43.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:43 smithi097 ceph-mon[32419]: purged_snaps scrub starts 2024-04-26T17:50:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:43 smithi097 ceph-mon[32419]: purged_snaps scrub ok 2024-04-26T17:50:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:43 smithi097 ceph-mon[32419]: osd.8 [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170] boot 2024-04-26T17:50:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:43 smithi097 ceph-mon[32419]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T17:50:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:43 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:50:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:43 smithi097 ceph-mon[32419]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:50:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T17:50:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: Detected new or changed devices on smithi186 2024-04-26T17:50:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi186 to 5696M 2024-04-26T17:50:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:44.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:44.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:44.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:44.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:44.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:44.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T17:50:44.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: Detected new or changed devices on smithi186 2024-04-26T17:50:44.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi186 to 5696M 2024-04-26T17:50:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:44.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T17:50:44.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: Detected new or changed devices on smithi186 2024-04-26T17:50:44.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:44.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:44.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:44.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:44.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:50:44.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi186 to 5696M 2024-04-26T17:50:44.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:44.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:44.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:44.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:44.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:50:44.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:44.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:50:44.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:50:45.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.203+0000 7f20223e6700 1 -- 172.21.15.31:0/4257365383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f201c103490 msgr2=0x7f201c1038b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:50:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.203+0000 7f20223e6700 1 --2- 172.21.15.31:0/4257365383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f201c103490 0x7f201c1038b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f200c0099c0 tx=0x7f200c009cd0 comp rx=0 tx=0).stop 2024-04-26T17:50:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.203+0000 7f20223e6700 1 -- 172.21.15.31:0/4257365383 shutdown_connections 2024-04-26T17:50:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.203+0000 7f20223e6700 1 --2- 172.21.15.31:0/4257365383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f201c105030 0x7f201c10b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.203+0000 7f20223e6700 1 --2- 172.21.15.31:0/4257365383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f201c103490 0x7f201c1038b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.203+0000 7f20223e6700 1 --2- 172.21.15.31:0/4257365383 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f201c104690 0x7f201c104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.203+0000 7f20223e6700 1 -- 172.21.15.31:0/4257365383 >> 172.21.15.31:0/4257365383 conn(0x7f201c0fea30 msgr2=0x7f201c100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:50:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.203+0000 7f20223e6700 1 -- 172.21.15.31:0/4257365383 shutdown_connections 2024-04-26T17:50:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.203+0000 7f20223e6700 1 -- 172.21.15.31:0/4257365383 wait complete. 2024-04-26T17:50:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.204+0000 7f20223e6700 1 Processor -- start 2024-04-26T17:50:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.204+0000 7f20223e6700 1 -- start start 2024-04-26T17:50:45.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.204+0000 7f20223e6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f201c103490 0x7f201c1995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:50:45.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.204+0000 7f20223e6700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f201c104690 0x7f201c199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:50:45.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.204+0000 7f20223e6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f201c105030 0x7f201c1a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:50:45.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.204+0000 7f20223e6700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f201c10ef20 con 0x7f201c105030 2024-04-26T17:50:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.204+0000 7f20223e6700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f201c10eda0 con 0x7f201c104690 2024-04-26T17:50:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.204+0000 7f20223e6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f201c10f0a0 con 0x7f201c103490 2024-04-26T17:50:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.204+0000 7f2020983700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f201c105030 0x7f201c1a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:50:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.204+0000 7f2020983700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f201c105030 0x7f201c1a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35662/0 (socket says 172.21.15.31:35662) 2024-04-26T17:50:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.204+0000 7f2020983700 1 -- 172.21.15.31:0/1839174086 learned_addr learned my addr 172.21.15.31:0/1839174086 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:50:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f201bfff700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f201c103490 0x7f201c1995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:50:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f2020983700 1 -- 172.21.15.31:0/1839174086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f201c103490 msgr2=0x7f201c1995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:50:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f201b7fe700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f201c104690 0x7f201c199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:50:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f2020983700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f201c103490 0x7f201c1995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f2020983700 1 -- 172.21.15.31:0/1839174086 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f201c104690 msgr2=0x7f201c199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:50:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f2020983700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f201c104690 0x7f201c199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:50:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f2020983700 1 -- 172.21.15.31:0/1839174086 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201c1a1210 con 0x7f201c105030 2024-04-26T17:50:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f201bfff700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f201c103490 0x7f201c1995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:50:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f201b7fe700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f201c104690 0x7f201c199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:50:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f2020983700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f201c105030 0x7f201c1a0b70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2014007f10 tx=0x7f201400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:50:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f20197fa700 1 -- 172.21.15.31:0/1839174086 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014004bb0 con 0x7f201c105030 2024-04-26T17:50:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f20197fa700 1 -- 172.21.15.31:0/1839174086 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2014004d50 con 0x7f201c105030 2024-04-26T17:50:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f20223e6700 1 -- 172.21.15.31:0/1839174086 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201c1a1500 con 0x7f201c105030 2024-04-26T17:50:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f20197fa700 1 -- 172.21.15.31:0/1839174086 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014005a30 con 0x7f201c105030 2024-04-26T17:50:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.205+0000 7f20223e6700 1 -- 172.21.15.31:0/1839174086 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f201c1a19f0 con 0x7f201c105030 2024-04-26T17:50:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.206+0000 7f20223e6700 1 -- 172.21.15.31:0/1839174086 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f201c06a110 con 0x7f201c105030 2024-04-26T17:50:45.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.207+0000 7f20197fa700 1 -- 172.21.15.31:0/1839174086 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f201401a020 con 0x7f201c105030 2024-04-26T17:50:45.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.208+0000 7f20197fa700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f20040a3c40 0x7f20040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:50:45.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.208+0000 7f201bfff700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f20040a3c40 0x7f20040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:50:45.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.208+0000 7f20197fa700 1 -- 172.21.15.31:0/1839174086 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5858+0+0 (secure 0 0 0) 0x7f20140cf870 con 0x7f201c105030 2024-04-26T17:50:45.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.208+0000 7f201bfff700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f20040a3c40 0x7f20040a6100 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f200c000c00 tx=0x7f200c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:50:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.211+0000 7f20197fa700 1 -- 172.21.15.31:0/1839174086 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2014010070 con 0x7f201c105030 2024-04-26T17:50:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:50:45.432+0000 7f20223e6700 1 -- 172.21.15.31:0/1839174086 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f201c063860 con 0x7f20040a3c40 2024-04-26T17:50:45.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:45 smithi031 ceph-mon[29164]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T17:50:45.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:45 smithi031 ceph-mon[29164]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:50:45.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:45 smithi097 ceph-mon[32419]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T17:50:45.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:45 smithi097 ceph-mon[32419]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:50:45.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:45 smithi186 ceph-mon[32797]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T17:50:45.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:45 smithi186 ceph-mon[32797]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:50:46.324 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:46 smithi031 ceph-mon[29164]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T17:50:46.324 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:46 smithi031 ceph-mon[29164]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:50:46.324 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:50:46.324 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:50:46.324 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:46 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:46.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:46 smithi097 ceph-mon[32419]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T17:50:46.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:46 smithi097 ceph-mon[32419]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:50:46.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:46 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:50:46.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:46 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:50:46.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:46 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:46.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:46 smithi186 ceph-mon[32797]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T17:50:46.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:46 smithi186 ceph-mon[32797]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:50:46.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:50:46.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:50:46.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:46 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:47.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:47 smithi031 ceph-mon[29164]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T17:50:47.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:47 smithi031 ceph-mon[29164]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:50:47.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:47 smithi097 ceph-mon[32419]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T17:50:47.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:47 smithi097 ceph-mon[32419]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:50:47.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:47 smithi186 ceph-mon[32797]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T17:50:47.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:47 smithi186 ceph-mon[32797]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:50:49.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:49 smithi031 ceph-mon[29164]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-26T17:50:49.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:49 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1441829478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12fda87d-830a-461f-a6cb-4699c52dc5e5"}]: dispatch 2024-04-26T17:50:49.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:49 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1441829478' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12fda87d-830a-461f-a6cb-4699c52dc5e5"}]': finished 2024-04-26T17:50:49.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:49 smithi031 ceph-mon[29164]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T17:50:49.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:50:49.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:49 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3416167667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:50:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:49 smithi097 ceph-mon[32419]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-26T17:50:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:49 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1441829478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12fda87d-830a-461f-a6cb-4699c52dc5e5"}]: dispatch 2024-04-26T17:50:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:49 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1441829478' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12fda87d-830a-461f-a6cb-4699c52dc5e5"}]': finished 2024-04-26T17:50:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:49 smithi097 ceph-mon[32419]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T17:50:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:50:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:49 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3416167667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:50:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:49 smithi186 ceph-mon[32797]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-26T17:50:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:49 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1441829478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12fda87d-830a-461f-a6cb-4699c52dc5e5"}]: dispatch 2024-04-26T17:50:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:49 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1441829478' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12fda87d-830a-461f-a6cb-4699c52dc5e5"}]': finished 2024-04-26T17:50:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:49 smithi186 ceph-mon[32797]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T17:50:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:50:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:49 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3416167667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:50:51.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:51 smithi031 ceph-mon[29164]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T17:50:51.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:51 smithi097 ceph-mon[32419]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T17:50:51.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:51 smithi186 ceph-mon[32797]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T17:50:53.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:53 smithi031 ceph-mon[29164]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T17:50:53.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:53 smithi097 ceph-mon[32419]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T17:50:53.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:53 smithi186 ceph-mon[32797]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T17:50:55.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:55 smithi031 ceph-mon[29164]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T17:50:55.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:55 smithi097 ceph-mon[32419]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T17:50:55.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:55 smithi186 ceph-mon[32797]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T17:50:56.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T17:50:56.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:56.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T17:50:56.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:56.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T17:50:56.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:50:57.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:57 smithi031 ceph-mon[29164]: Deploying daemon osd.9 on smithi031 2024-04-26T17:50:57.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:57 smithi031 ceph-mon[29164]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T17:50:57.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:57 smithi097 ceph-mon[32419]: Deploying daemon osd.9 on smithi031 2024-04-26T17:50:57.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:57 smithi097 ceph-mon[32419]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T17:50:57.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:57 smithi186 ceph-mon[32797]: Deploying daemon osd.9 on smithi031 2024-04-26T17:50:57.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:57 smithi186 ceph-mon[32797]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T17:50:59.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:50:59 smithi031 ceph-mon[29164]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:50:59.569 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:50:59 smithi186 ceph-mon[32797]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:50:59.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:50:59 smithi097 ceph-mon[32419]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:01.412 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:01 smithi031 ceph-mon[29164]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:01.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:01 smithi097 ceph-mon[32419]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:01.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:01 smithi186 ceph-mon[32797]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:03.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:03 smithi186 ceph-mon[32797]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:03.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:03.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:03.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:03.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:03 smithi031 ceph-mon[29164]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:03.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:03.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:03.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:03.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:03 smithi097 ceph-mon[32419]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:03.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:03.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:03.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:05.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:05 smithi186 ceph-mon[32797]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:05.798 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:05 smithi031 ceph-mon[29164]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:05.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:05 smithi097 ceph-mon[32419]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:06.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:51:06.507+0000 7f20197fa700 1 -- 172.21.15.31:0/1839174086 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f201c063860 con 0x7f20040a3c40 2024-04-26T17:51:06.508 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 9 on host 'smithi031' 2024-04-26T17:51:06.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:51:06.509+0000 7f20223e6700 1 -- 172.21.15.31:0/1839174086 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f20040a3c40 msgr2=0x7f20040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:51:06.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:51:06.509+0000 7f20223e6700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f20040a3c40 0x7f20040a6100 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f200c000c00 tx=0x7f200c005fb0 comp rx=0 tx=0).stop 2024-04-26T17:51:06.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:51:06.509+0000 7f20223e6700 1 -- 172.21.15.31:0/1839174086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f201c105030 msgr2=0x7f201c1a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:51:06.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:51:06.509+0000 7f20223e6700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f201c105030 0x7f201c1a0b70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2014007f10 tx=0x7f201400d3b0 comp rx=0 tx=0).stop 2024-04-26T17:51:06.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:51:06.509+0000 7f20223e6700 1 -- 172.21.15.31:0/1839174086 shutdown_connections 2024-04-26T17:51:06.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:51:06.509+0000 7f20223e6700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f20040a3c40 0x7f20040a6100 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:06.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:51:06.509+0000 7f20223e6700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f201c105030 0x7f201c1a0b70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:06.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:51:06.509+0000 7f20223e6700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f201c103490 0x7f201c1995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:06.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:51:06.510+0000 7f20223e6700 1 --2- 172.21.15.31:0/1839174086 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f201c104690 0x7f201c199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:06.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:51:06.510+0000 7f20223e6700 1 -- 172.21.15.31:0/1839174086 >> 172.21.15.31:0/1839174086 conn(0x7f201c0fea30 msgr2=0x7f201c107a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:51:06.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:51:06.510+0000 7f20223e6700 1 -- 172.21.15.31:0/1839174086 shutdown_connections 2024-04-26T17:51:06.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:51:06.510+0000 7f20223e6700 1 -- 172.21.15.31:0/1839174086 wait complete. 2024-04-26T17:51:07.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:06 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:06 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:06 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:07.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:06 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:07.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:06 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:06 smithi186 ceph-mon[32797]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T17:51:07.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:06 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:06 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:07.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:06 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:06 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:06 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:06 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:07.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:06 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:07.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:06 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:06 smithi031 ceph-mon[29164]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T17:51:07.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:06 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:06 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:07.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:06 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:06 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:06 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:06 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:07.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:06 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:07.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:06 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:06 smithi097 ceph-mon[32419]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T17:51:07.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:06 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:06 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:07.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:06 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:07.200 DEBUG:teuthology.orchestra.run.smithi031:osd.9> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.9.service 2024-04-26T17:51:07.203 INFO:tasks.cephadm:Deploying osd.10 on smithi097 with /dev/vg_nvme/lv_1... 2024-04-26T17:51:07.203 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T17:51:07.447 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T17:51:07.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:07 smithi097 ceph-mon[32419]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:07.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:07 smithi097 ceph-mon[32419]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T17:51:07.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:07 smithi097 ceph-mon[32419]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T17:51:07.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:07 smithi097 ceph-mon[32419]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-26T17:51:07.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:07 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:08.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:07 smithi186 ceph-mon[32797]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:08.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:07 smithi186 ceph-mon[32797]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T17:51:08.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:07 smithi186 ceph-mon[32797]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T17:51:08.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:07 smithi186 ceph-mon[32797]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-26T17:51:08.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:07 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:08.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:07 smithi031 ceph-mon[29164]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:08.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:07 smithi031 ceph-mon[29164]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T17:51:08.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:07 smithi031 ceph-mon[29164]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T17:51:08.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:07 smithi031 ceph-mon[29164]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-26T17:51:08.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:07 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:08.399 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 17:51:08 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T17:51:08.159+0000 7f5d5cc9f700 -1 osd.9 0 waiting for initial osdmap 2024-04-26T17:51:08.399 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 17:51:08 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T17:51:08.172+0000 7f5d54a8a700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T17:51:08.710 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:08 smithi031 ceph-mon[29164]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-26T17:51:08.710 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:08 smithi031 ceph-mon[29164]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T17:51:08.710 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:08 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:08.710 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:08 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:08.710 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:08 smithi031 ceph-mon[29164]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' 2024-04-26T17:51:08.855 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:08 smithi097 ceph-mon[32419]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-26T17:51:08.856 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:08 smithi097 ceph-mon[32419]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T17:51:08.856 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:08 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:08.856 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:08 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:08.856 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:08 smithi097 ceph-mon[32419]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' 2024-04-26T17:51:09.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:08 smithi186 ceph-mon[32797]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-26T17:51:09.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:08 smithi186 ceph-mon[32797]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T17:51:09.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:08 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:09.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:08 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:09.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:08 smithi186 ceph-mon[32797]: from='osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905]' entity='osd.9' 2024-04-26T17:51:09.447 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T17:51:09.447 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T17:51:09.447 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-04-26T17:51:09.447 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-04-26T17:51:09.447 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131036 s, 800 MB/s 2024-04-26T17:51:09.447 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-04-26T17:51:09.885 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:09 smithi031 ceph-mon[29164]: purged_snaps scrub starts 2024-04-26T17:51:09.885 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:09 smithi031 ceph-mon[29164]: purged_snaps scrub ok 2024-04-26T17:51:09.885 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:09 smithi031 ceph-mon[29164]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:09.886 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:09 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:09.886 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:09 smithi031 ceph-mon[29164]: osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905] boot 2024-04-26T17:51:09.886 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:09 smithi031 ceph-mon[29164]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T17:51:09.886 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:09 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:10.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:09 smithi186 ceph-mon[32797]: purged_snaps scrub starts 2024-04-26T17:51:10.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:09 smithi186 ceph-mon[32797]: purged_snaps scrub ok 2024-04-26T17:51:10.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:09 smithi186 ceph-mon[32797]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:10.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:09 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:10.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:09 smithi186 ceph-mon[32797]: osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905] boot 2024-04-26T17:51:10.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:09 smithi186 ceph-mon[32797]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T17:51:10.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:09 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:10.067 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:09 smithi097 ceph-mon[32419]: purged_snaps scrub starts 2024-04-26T17:51:10.067 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:09 smithi097 ceph-mon[32419]: purged_snaps scrub ok 2024-04-26T17:51:10.067 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:09 smithi097 ceph-mon[32419]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-26T17:51:10.067 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:09 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:10.067 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:09 smithi097 ceph-mon[32419]: osd.9 [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905] boot 2024-04-26T17:51:10.067 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:09 smithi097 ceph-mon[32419]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T17:51:10.068 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:09 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:51:10.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_1 2024-04-26T17:51:10.328 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T17:51:11.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:11 smithi097 ceph-mon[32419]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T17:51:11.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:11 smithi097 ceph-mon[32419]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:11.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:11 smithi097 ceph-mon[32419]: Detected new or changed devices on smithi031 2024-04-26T17:51:11.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:11.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:11.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:11.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:11.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:11.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:11.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:11 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi031 to 4273M 2024-04-26T17:51:11.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:11.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:11.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:11.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:11.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:11 smithi186 ceph-mon[32797]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T17:51:11.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:11 smithi186 ceph-mon[32797]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:11.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:11 smithi186 ceph-mon[32797]: Detected new or changed devices on smithi031 2024-04-26T17:51:11.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:11.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:11.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:11.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:11.519 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:11.519 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:11.519 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:11 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi031 to 4273M 2024-04-26T17:51:11.519 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:11.519 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:11.519 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:11.519 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:11.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:11 smithi031 ceph-mon[29164]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T17:51:11.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:11 smithi031 ceph-mon[29164]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:11.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:11 smithi031 ceph-mon[29164]: Detected new or changed devices on smithi031 2024-04-26T17:51:11.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:11.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:11.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:11.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:11.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:11.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:11.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:11 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi031 to 4273M 2024-04-26T17:51:11.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:11.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:11.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:11.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:11.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.913+0000 7f58a02d5700 1 -- 172.21.15.97:0/1494820548 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5898104680 msgr2=0x7f5898104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:51:11.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.913+0000 7f58a02d5700 1 --2- 172.21.15.97:0/1494820548 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5898104680 0x7f5898104ae0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f58940099c0 tx=0x7f5894009cd0 comp rx=0 tx=0).stop 2024-04-26T17:51:11.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.913+0000 7f58a02d5700 1 -- 172.21.15.97:0/1494820548 shutdown_connections 2024-04-26T17:51:11.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.913+0000 7f58a02d5700 1 --2- 172.21.15.97:0/1494820548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5898105020 0x7f589810b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:11.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.913+0000 7f58a02d5700 1 --2- 172.21.15.97:0/1494820548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5898103480 0x7f58981038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:11.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.913+0000 7f58a02d5700 1 --2- 172.21.15.97:0/1494820548 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5898104680 0x7f5898104ae0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:11.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.913+0000 7f58a02d5700 1 -- 172.21.15.97:0/1494820548 >> 172.21.15.97:0/1494820548 conn(0x7f58980fea20 msgr2=0x7f5898100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:51:11.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.913+0000 7f58a02d5700 1 -- 172.21.15.97:0/1494820548 shutdown_connections 2024-04-26T17:51:11.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.913+0000 7f58a02d5700 1 -- 172.21.15.97:0/1494820548 wait complete. 2024-04-26T17:51:11.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.914+0000 7f58a02d5700 1 Processor -- start 2024-04-26T17:51:11.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.914+0000 7f58a02d5700 1 -- start start 2024-04-26T17:51:11.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.914+0000 7f58a02d5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5898103480 0x7f5898199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:51:11.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.914+0000 7f58a02d5700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5898104680 0x7f5898199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:51:11.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.914+0000 7f58a02d5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5898105020 0x7f58981a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:51:11.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.914+0000 7f58a02d5700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f589810ef90 con 0x7f5898103480 2024-04-26T17:51:11.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.914+0000 7f58a02d5700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f589810ee10 con 0x7f5898104680 2024-04-26T17:51:11.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.914+0000 7f58a02d5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f589810f110 con 0x7f5898105020 2024-04-26T17:51:11.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f589d870700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5898104680 0x7f5898199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:51:11.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f589d870700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5898104680 0x7f5898199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33214/0 (socket says 172.21.15.97:33214) 2024-04-26T17:51:11.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f589d870700 1 -- 172.21.15.97:0/2136420324 learned_addr learned my addr 172.21.15.97:0/2136420324 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-04-26T17:51:11.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f589e071700 1 --2- 172.21.15.97:0/2136420324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5898103480 0x7f5898199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:51:11.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f589e872700 1 --2- 172.21.15.97:0/2136420324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5898105020 0x7f58981a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:51:11.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f589d870700 1 -- 172.21.15.97:0/2136420324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5898105020 msgr2=0x7f58981a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:51:11.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f589d870700 1 --2- 172.21.15.97:0/2136420324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5898105020 0x7f58981a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:11.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f589d870700 1 -- 172.21.15.97:0/2136420324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5898103480 msgr2=0x7f5898199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:51:11.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f589d870700 1 --2- 172.21.15.97:0/2136420324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5898103480 0x7f5898199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:11.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f589d870700 1 -- 172.21.15.97:0/2136420324 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58981a12c0 con 0x7f5898104680 2024-04-26T17:51:11.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f589d870700 1 --2- 172.21.15.97:0/2136420324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5898104680 0x7f5898199ba0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5894003790 tx=0x7f5894005b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:51:11.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f588f7fe700 1 -- 172.21.15.97:0/2136420324 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5894027070 con 0x7f5898104680 2024-04-26T17:51:11.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f588f7fe700 1 -- 172.21.15.97:0/2136420324 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5894004160 con 0x7f5898104680 2024-04-26T17:51:11.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f58a02d5700 1 -- 172.21.15.97:0/2136420324 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58981a1550 con 0x7f5898104680 2024-04-26T17:51:11.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f588f7fe700 1 -- 172.21.15.97:0/2136420324 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5894022dc0 con 0x7f5898104680 2024-04-26T17:51:11.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.915+0000 7f58a02d5700 1 -- 172.21.15.97:0/2136420324 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58981a1a30 con 0x7f5898104680 2024-04-26T17:51:11.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.916+0000 7f58a02d5700 1 -- 172.21.15.97:0/2136420324 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f589806a110 con 0x7f5898104680 2024-04-26T17:51:11.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.917+0000 7f588f7fe700 1 -- 172.21.15.97:0/2136420324 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f589402a070 con 0x7f5898104680 2024-04-26T17:51:11.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.918+0000 7f588f7fe700 1 --2- 172.21.15.97:0/2136420324 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f58840a3c40 0x7f58840a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:51:11.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.918+0000 7f588f7fe700 1 -- 172.21.15.97:0/2136420324 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f58940d0900 con 0x7f5898104680 2024-04-26T17:51:11.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.918+0000 7f589e071700 1 --2- 172.21.15.97:0/2136420324 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f58840a3c40 0x7f58840a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:51:11.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.919+0000 7f589e071700 1 --2- 172.21.15.97:0/2136420324 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f58840a3c40 0x7f58840a6100 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f58981044e0 tx=0x7f5888009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:51:11.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:11.922+0000 7f588f7fe700 1 -- 172.21.15.97:0/2136420324 <== mon.2 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5894013070 con 0x7f5898104680 2024-04-26T17:51:12.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:12.125+0000 7f58a02d5700 1 -- 172.21.15.97:0/2136420324 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f5898063860 con 0x7f58840a3c40 2024-04-26T17:51:12.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:12 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:51:12.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:12 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:51:12.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:12 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:12.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:12 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:51:12.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:12 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:51:12.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:12 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:12.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:12 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:51:12.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:12 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:51:12.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:12 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:13.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:13 smithi097 ceph-mon[32419]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:51:13.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:13 smithi097 ceph-mon[32419]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:13.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:13 smithi186 ceph-mon[32797]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:51:13.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:13 smithi186 ceph-mon[32797]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:13.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:13 smithi031 ceph-mon[29164]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:51:13.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:13 smithi031 ceph-mon[29164]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:15.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:15 smithi097 ceph-mon[32419]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:15.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:15 smithi097 ceph-mon[32419]: from='client.? 172.21.15.97:0/886087191' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92a639b3-918b-485c-bd30-258b4224993e"}]: dispatch 2024-04-26T17:51:15.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:15 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92a639b3-918b-485c-bd30-258b4224993e"}]: dispatch 2024-04-26T17:51:15.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:15 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92a639b3-918b-485c-bd30-258b4224993e"}]': finished 2024-04-26T17:51:15.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:15 smithi097 ceph-mon[32419]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T17:51:15.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:15 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:15.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:15 smithi186 ceph-mon[32797]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:15.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:15 smithi186 ceph-mon[32797]: from='client.? 172.21.15.97:0/886087191' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92a639b3-918b-485c-bd30-258b4224993e"}]: dispatch 2024-04-26T17:51:15.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:15 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92a639b3-918b-485c-bd30-258b4224993e"}]: dispatch 2024-04-26T17:51:15.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:15 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92a639b3-918b-485c-bd30-258b4224993e"}]': finished 2024-04-26T17:51:15.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:15 smithi186 ceph-mon[32797]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T17:51:15.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:15 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:15.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:15 smithi031 ceph-mon[29164]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:15.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:15 smithi031 ceph-mon[29164]: from='client.? 172.21.15.97:0/886087191' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92a639b3-918b-485c-bd30-258b4224993e"}]: dispatch 2024-04-26T17:51:15.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:15 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92a639b3-918b-485c-bd30-258b4224993e"}]: dispatch 2024-04-26T17:51:15.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:15 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92a639b3-918b-485c-bd30-258b4224993e"}]': finished 2024-04-26T17:51:15.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:15 smithi031 ceph-mon[29164]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T17:51:15.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:15 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:16.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:16 smithi097 ceph-mon[32419]: from='client.? 172.21.15.97:0/1775697139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:51:16.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:16 smithi186 ceph-mon[32797]: from='client.? 172.21.15.97:0/1775697139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:51:16.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:16 smithi031 ceph-mon[29164]: from='client.? 172.21.15.97:0/1775697139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:51:17.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:17 smithi097 ceph-mon[32419]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:17.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:17 smithi186 ceph-mon[32797]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:17.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:17 smithi031 ceph-mon[29164]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:19.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:19 smithi097 ceph-mon[32419]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:19.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:19 smithi186 ceph-mon[32797]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:19.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:19 smithi031 ceph-mon[29164]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:21.622 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:21 smithi097 ceph-mon[32419]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:21.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:21 smithi186 ceph-mon[32797]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:21.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:21 smithi031 ceph-mon[29164]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:22.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:22 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T17:51:22.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:22 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:22.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:22 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T17:51:22.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:22 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:22.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:22 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T17:51:22.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:22 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:23.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:23 smithi097 ceph-mon[32419]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:23.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:23 smithi097 ceph-mon[32419]: Deploying daemon osd.10 on smithi097 2024-04-26T17:51:23.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:23 smithi186 ceph-mon[32797]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:23.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:23 smithi186 ceph-mon[32797]: Deploying daemon osd.10 on smithi097 2024-04-26T17:51:23.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:23 smithi031 ceph-mon[29164]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:23.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:23 smithi031 ceph-mon[29164]: Deploying daemon osd.10 on smithi097 2024-04-26T17:51:25.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:25 smithi097 ceph-mon[32419]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:25.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:25 smithi186 ceph-mon[32797]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:25.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:25 smithi031 ceph-mon[29164]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:27.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:27 smithi097 ceph-mon[32419]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:27.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:27 smithi186 ceph-mon[32797]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:27.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:27 smithi031 ceph-mon[29164]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:29.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:28 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:29.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:28 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:29.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:28 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:29.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:28 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:29.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:28 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:29.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:28 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:29.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:28 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:29.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:28 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:29.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:28 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:30.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:29 smithi186 ceph-mon[32797]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:30.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:29 smithi031 ceph-mon[29164]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:30.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:29 smithi097 ceph-mon[32419]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:31.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:31.796+0000 7f588f7fe700 1 -- 172.21.15.97:0/2136420324 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f5898063860 con 0x7f58840a3c40 2024-04-26T17:51:31.796 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 10 on host 'smithi097' 2024-04-26T17:51:31.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:31.798+0000 7f58a02d5700 1 -- 172.21.15.97:0/2136420324 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f58840a3c40 msgr2=0x7f58840a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:51:31.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:31.798+0000 7f58a02d5700 1 --2- 172.21.15.97:0/2136420324 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f58840a3c40 0x7f58840a6100 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f58981044e0 tx=0x7f5888009190 comp rx=0 tx=0).stop 2024-04-26T17:51:31.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:31.798+0000 7f58a02d5700 1 -- 172.21.15.97:0/2136420324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5898104680 msgr2=0x7f5898199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:51:31.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:31.798+0000 7f58a02d5700 1 --2- 172.21.15.97:0/2136420324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5898104680 0x7f5898199ba0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5894003790 tx=0x7f5894005b00 comp rx=0 tx=0).stop 2024-04-26T17:51:31.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:31.798+0000 7f58a02d5700 1 -- 172.21.15.97:0/2136420324 shutdown_connections 2024-04-26T17:51:31.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:31.798+0000 7f58a02d5700 1 --2- 172.21.15.97:0/2136420324 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f58840a3c40 0x7f58840a6100 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:31.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:31.798+0000 7f58a02d5700 1 --2- 172.21.15.97:0/2136420324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5898105020 0x7f58981a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:31.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:31.798+0000 7f58a02d5700 1 --2- 172.21.15.97:0/2136420324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5898103480 0x7f5898199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:31.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:31.798+0000 7f58a02d5700 1 --2- 172.21.15.97:0/2136420324 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5898104680 0x7f5898199ba0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:31.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:31.798+0000 7f58a02d5700 1 -- 172.21.15.97:0/2136420324 >> 172.21.15.97:0/2136420324 conn(0x7f58980fea20 msgr2=0x7f5898107a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:51:31.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:31.798+0000 7f58a02d5700 1 -- 172.21.15.97:0/2136420324 shutdown_connections 2024-04-26T17:51:31.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-26T17:51:31.798+0000 7f58a02d5700 1 -- 172.21.15.97:0/2136420324 wait complete. 2024-04-26T17:51:32.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:32 smithi186 ceph-mon[32797]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:32.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:32.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:32.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:32.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:32 smithi186 ceph-mon[32797]: from='osd.10 [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T17:51:32.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:32 smithi186 ceph-mon[32797]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T17:51:32.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:32 smithi031 ceph-mon[29164]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:32.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:32 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:32 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:32 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:32.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:32 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:32.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:32 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:32 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:32 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:32.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:32 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:32 smithi031 ceph-mon[29164]: from='osd.10 [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T17:51:32.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:32 smithi031 ceph-mon[29164]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T17:51:32.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:32 smithi097 ceph-mon[32419]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:32.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:32 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:32 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:32 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:32.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:32 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:32.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:32 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:32 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:32 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:32.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:32 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:32.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:32 smithi097 ceph-mon[32419]: from='osd.10 [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T17:51:32.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:32 smithi097 ceph-mon[32419]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T17:51:32.472 DEBUG:teuthology.orchestra.run.smithi097:osd.10> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.10.service 2024-04-26T17:51:32.474 INFO:tasks.cephadm:Deploying osd.11 on smithi186 with /dev/vg_nvme/lv_1... 2024-04-26T17:51:32.474 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T17:51:32.732 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T17:51:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:33 smithi097 ceph-mon[32419]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:33 smithi097 ceph-mon[32419]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T17:51:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:33 smithi097 ceph-mon[32419]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T17:51:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:33 smithi097 ceph-mon[32419]: from='osd.10 [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:51:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:33 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:33 smithi097 ceph-mon[32419]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:51:33.884 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 17:51:33 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T17:51:33.795+0000 7f4cd6819700 -1 osd.10 0 waiting for initial osdmap 2024-04-26T17:51:33.884 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 17:51:33 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T17:51:33.810+0000 7f4cd260c700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T17:51:33.988 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:33 smithi186 ceph-mon[32797]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:33.988 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:33 smithi186 ceph-mon[32797]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T17:51:33.988 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:33 smithi186 ceph-mon[32797]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T17:51:33.988 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:33 smithi186 ceph-mon[32797]: from='osd.10 [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:51:33.988 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:33 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:33.988 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:33 smithi186 ceph-mon[32797]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:51:34.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:33 smithi031 ceph-mon[29164]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:34.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:33 smithi031 ceph-mon[29164]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T17:51:34.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:33 smithi031 ceph-mon[29164]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T17:51:34.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:33 smithi031 ceph-mon[29164]: from='osd.10 [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:51:34.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:33 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:34.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:33 smithi031 ceph-mon[29164]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-26T17:51:34.812 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T17:51:34.812 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T17:51:34.812 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-26T17:51:34.813 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-26T17:51:34.813 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0117998 s, 889 MB/s 2024-04-26T17:51:34.813 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 2024-04-26T17:51:34.813 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-26T17:51:35.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:34 smithi031 ceph-mon[29164]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-26T17:51:35.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:34 smithi031 ceph-mon[29164]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T17:51:35.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:34 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:35.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:34 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:35.116 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:34 smithi097 ceph-mon[32419]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-26T17:51:35.116 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:34 smithi097 ceph-mon[32419]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T17:51:35.116 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:34 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:35.117 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:34 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:35.195 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:34 smithi186 ceph-mon[32797]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-26T17:51:35.195 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:34 smithi186 ceph-mon[32797]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T17:51:35.195 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:34 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:35.195 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:34 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:35.649 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_1 2024-04-26T17:51:35.899 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T17:51:35.913 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: purged_snaps scrub starts 2024-04-26T17:51:35.913 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: purged_snaps scrub ok 2024-04-26T17:51:35.913 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:35.913 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:35.913 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: osd.10 [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903] boot 2024-04-26T17:51:35.913 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T17:51:35.913 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:35.913 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:35.913 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:35.913 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:35.913 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:35.913 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:35.914 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:35.914 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:35.914 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:35.914 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:35.914 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:36.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: purged_snaps scrub starts 2024-04-26T17:51:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: purged_snaps scrub ok 2024-04-26T17:51:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: osd.10 [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903] boot 2024-04-26T17:51:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T17:51:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:36.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:36.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:36.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:36.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:36.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:36.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:36.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:36.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: purged_snaps scrub starts 2024-04-26T17:51:36.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: purged_snaps scrub ok 2024-04-26T17:51:36.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-26T17:51:36.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:36.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: osd.10 [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903] boot 2024-04-26T17:51:36.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T17:51:36.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:51:36.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:36.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:36.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:36.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:36.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:36.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:51:36.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:36.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:36.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:36.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:37.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:36 smithi097 ceph-mon[32419]: Detected new or changed devices on smithi097 2024-04-26T17:51:37.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:36 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi097 to 4272M 2024-04-26T17:51:37.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:36 smithi097 ceph-mon[32419]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T17:51:37.175 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:36 smithi186 ceph-mon[32797]: Detected new or changed devices on smithi097 2024-04-26T17:51:37.175 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:36 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi097 to 4272M 2024-04-26T17:51:37.175 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:36 smithi186 ceph-mon[32797]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T17:51:37.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:36 smithi031 ceph-mon[29164]: Detected new or changed devices on smithi097 2024-04-26T17:51:37.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:36 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi097 to 4272M 2024-04-26T17:51:37.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:36 smithi031 ceph-mon[29164]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T17:51:37.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.488+0000 7f1db3c08700 1 -- 172.21.15.186:0/2969375356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1dac103570 msgr2=0x7f1dac1039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:51:37.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.488+0000 7f1db3c08700 1 --2- 172.21.15.186:0/2969375356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1dac103570 0x7f1dac1039d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1da000b0a0 tx=0x7f1da000b3b0 comp rx=0 tx=0).stop 2024-04-26T17:51:37.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.488+0000 7f1db3c08700 1 -- 172.21.15.186:0/2969375356 shutdown_connections 2024-04-26T17:51:37.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.488+0000 7f1db3c08700 1 --2- 172.21.15.186:0/2969375356 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1dac103f10 0x7f1dac10a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:37.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.488+0000 7f1db3c08700 1 --2- 172.21.15.186:0/2969375356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dac102370 0x7f1dac102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:37.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.488+0000 7f1db3c08700 1 --2- 172.21.15.186:0/2969375356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1dac103570 0x7f1dac1039d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:37.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.488+0000 7f1db3c08700 1 -- 172.21.15.186:0/2969375356 >> 172.21.15.186:0/2969375356 conn(0x7f1dac0fd910 msgr2=0x7f1dac0ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:51:37.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.488+0000 7f1db3c08700 1 -- 172.21.15.186:0/2969375356 shutdown_connections 2024-04-26T17:51:37.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.488+0000 7f1db3c08700 1 -- 172.21.15.186:0/2969375356 wait complete. 2024-04-26T17:51:37.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.488+0000 7f1db3c08700 1 Processor -- start 2024-04-26T17:51:37.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.488+0000 7f1db3c08700 1 -- start start 2024-04-26T17:51:37.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db3c08700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1dac102370 0x7f1dac199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:51:37.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db3c08700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1dac103570 0x7f1dac199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:51:37.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db3c08700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dac103f10 0x7f1dac1a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:51:37.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db3c08700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac10df50 con 0x7f1dac103f10 2024-04-26T17:51:37.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db3c08700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac10ddd0 con 0x7f1dac102370 2024-04-26T17:51:37.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db3c08700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac10e0d0 con 0x7f1dac103570 2024-04-26T17:51:37.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db11a3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1dac103570 0x7f1dac199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:51:37.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db11a3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1dac103570 0x7f1dac199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34420/0 (socket says 172.21.15.186:34420) 2024-04-26T17:51:37.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db11a3700 1 -- 172.21.15.186:0/4000095564 learned_addr learned my addr 172.21.15.186:0/4000095564 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T17:51:37.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db19a4700 1 --2- 172.21.15.186:0/4000095564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1dac102370 0x7f1dac199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:51:37.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db21a5700 1 --2- 172.21.15.186:0/4000095564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dac103f10 0x7f1dac1a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:51:37.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db11a3700 1 -- 172.21.15.186:0/4000095564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1dac102370 msgr2=0x7f1dac199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:51:37.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db11a3700 1 --2- 172.21.15.186:0/4000095564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1dac102370 0x7f1dac199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:37.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db11a3700 1 -- 172.21.15.186:0/4000095564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dac103f10 msgr2=0x7f1dac1a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:51:37.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db11a3700 1 --2- 172.21.15.186:0/4000095564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dac103f10 0x7f1dac1a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:37.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db11a3700 1 -- 172.21.15.186:0/4000095564 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dac1a1480 con 0x7f1dac103570 2024-04-26T17:51:37.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1db11a3700 1 --2- 172.21.15.186:0/4000095564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1dac103570 0x7f1dac199d60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1da0006070 tx=0x7f1da000cf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:51:37.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1daaffd700 1 -- 172.21.15.186:0/4000095564 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da0021020 con 0x7f1dac103570 2024-04-26T17:51:37.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.489+0000 7f1daaffd700 1 -- 172.21.15.186:0/4000095564 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1da001b910 con 0x7f1dac103570 2024-04-26T17:51:37.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.490+0000 7f1db3c08700 1 -- 172.21.15.186:0/4000095564 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dac1a1710 con 0x7f1dac103570 2024-04-26T17:51:37.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.490+0000 7f1daaffd700 1 -- 172.21.15.186:0/4000095564 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da001d990 con 0x7f1dac103570 2024-04-26T17:51:37.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.490+0000 7f1db3c08700 1 -- 172.21.15.186:0/4000095564 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dac1a1c40 con 0x7f1dac103570 2024-04-26T17:51:37.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.491+0000 7f1db3c08700 1 -- 172.21.15.186:0/4000095564 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dac06a110 con 0x7f1dac103570 2024-04-26T17:51:37.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.492+0000 7f1daaffd700 1 -- 172.21.15.186:0/4000095564 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1da0024020 con 0x7f1dac103570 2024-04-26T17:51:37.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.492+0000 7f1daaffd700 1 --2- 172.21.15.186:0/4000095564 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f1d940a3c40 0x7f1d940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:51:37.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.492+0000 7f1daaffd700 1 -- 172.21.15.186:0/4000095564 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7f1da000f020 con 0x7f1dac103570 2024-04-26T17:51:37.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.492+0000 7f1db19a4700 1 --2- 172.21.15.186:0/4000095564 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f1d940a3c40 0x7f1d940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:51:37.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.493+0000 7f1db19a4700 1 --2- 172.21.15.186:0/4000095564 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f1d940a3c40 0x7f1d940a6100 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c006fd0 tx=0x7f1d9c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:51:37.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.496+0000 7f1daaffd700 1 -- 172.21.15.186:0/4000095564 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1da002e380 con 0x7f1dac103570 2024-04-26T17:51:37.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:37.700+0000 7f1db3c08700 1 -- 172.21.15.186:0/4000095564 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f1dac1a1f60 con 0x7f1d940a3c40 2024-04-26T17:51:38.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:37 smithi097 ceph-mon[32419]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:38.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:37 smithi097 ceph-mon[32419]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T17:51:38.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:51:38.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:51:38.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:38.210 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:37 smithi031 ceph-mon[29164]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:38.210 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:37 smithi031 ceph-mon[29164]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T17:51:38.210 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:51:38.210 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:51:38.210 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:38.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:37 smithi186 ceph-mon[32797]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:38.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:37 smithi186 ceph-mon[32797]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T17:51:38.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T17:51:38.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T17:51:38.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:39.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:38 smithi097 ceph-mon[32419]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:51:39.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:38 smithi097 ceph-mon[32419]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T17:51:39.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:38 smithi186 ceph-mon[32797]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:51:39.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:38 smithi186 ceph-mon[32797]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T17:51:39.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:38 smithi031 ceph-mon[29164]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:51:39.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:38 smithi031 ceph-mon[29164]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T17:51:39.879 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:39 smithi186 ceph-mon[32797]: pgmap v240: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T17:51:39.879 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:39 smithi186 ceph-mon[32797]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T17:51:40.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:39 smithi097 ceph-mon[32419]: pgmap v240: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T17:51:40.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:39 smithi097 ceph-mon[32419]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T17:51:40.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:39 smithi031 ceph-mon[29164]: pgmap v240: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T17:51:40.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:39 smithi031 ceph-mon[29164]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T17:51:41.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:40 smithi097 ceph-mon[32419]: from='client.? 172.21.15.186:0/1874027370' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "842aa599-d21d-4e4a-b872-e9ac20d8c0ff"}]: dispatch 2024-04-26T17:51:41.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:40 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "842aa599-d21d-4e4a-b872-e9ac20d8c0ff"}]: dispatch 2024-04-26T17:51:41.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:40 smithi097 ceph-mon[32419]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "842aa599-d21d-4e4a-b872-e9ac20d8c0ff"}]': finished 2024-04-26T17:51:41.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:40 smithi097 ceph-mon[32419]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T17:51:41.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:40 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:51:41.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:40 smithi186 ceph-mon[32797]: from='client.? 172.21.15.186:0/1874027370' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "842aa599-d21d-4e4a-b872-e9ac20d8c0ff"}]: dispatch 2024-04-26T17:51:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:40 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "842aa599-d21d-4e4a-b872-e9ac20d8c0ff"}]: dispatch 2024-04-26T17:51:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:40 smithi186 ceph-mon[32797]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "842aa599-d21d-4e4a-b872-e9ac20d8c0ff"}]': finished 2024-04-26T17:51:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:40 smithi186 ceph-mon[32797]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T17:51:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:40 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:51:41.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:40 smithi031 ceph-mon[29164]: from='client.? 172.21.15.186:0/1874027370' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "842aa599-d21d-4e4a-b872-e9ac20d8c0ff"}]: dispatch 2024-04-26T17:51:41.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:40 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "842aa599-d21d-4e4a-b872-e9ac20d8c0ff"}]: dispatch 2024-04-26T17:51:41.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:40 smithi031 ceph-mon[29164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "842aa599-d21d-4e4a-b872-e9ac20d8c0ff"}]': finished 2024-04-26T17:51:41.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:40 smithi031 ceph-mon[29164]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T17:51:41.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:40 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:51:42.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:41 smithi097 ceph-mon[32419]: pgmap v242: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-26T17:51:42.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:41 smithi097 ceph-mon[32419]: from='client.? 172.21.15.186:0/182093162' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:51:42.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:41 smithi186 ceph-mon[32797]: pgmap v242: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-26T17:51:42.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:41 smithi186 ceph-mon[32797]: from='client.? 172.21.15.186:0/182093162' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:51:42.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:41 smithi031 ceph-mon[29164]: pgmap v242: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-26T17:51:42.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:41 smithi031 ceph-mon[29164]: from='client.? 172.21.15.186:0/182093162' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T17:51:44.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:43 smithi097 ceph-mon[32419]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-26T17:51:44.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:43 smithi186 ceph-mon[32797]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-26T17:51:44.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:43 smithi031 ceph-mon[29164]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-26T17:51:46.027 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:45 smithi186 ceph-mon[32797]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:46.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:45 smithi097 ceph-mon[32419]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:46.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:45 smithi031 ceph-mon[29164]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:48.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:47 smithi097 ceph-mon[32419]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:48.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:47 smithi186 ceph-mon[32797]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:48.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:47 smithi031 ceph-mon[29164]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:49.118 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T17:51:49.118 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T17:51:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:49.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T17:51:49.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:50.115 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:49 smithi186 ceph-mon[32797]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:50.115 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:49 smithi186 ceph-mon[32797]: Deploying daemon osd.11 on smithi186 2024-04-26T17:51:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:49 smithi097 ceph-mon[32419]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:49 smithi097 ceph-mon[32419]: Deploying daemon osd.11 on smithi186 2024-04-26T17:51:50.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:49 smithi031 ceph-mon[29164]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:50.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:49 smithi031 ceph-mon[29164]: Deploying daemon osd.11 on smithi186 2024-04-26T17:51:52.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:51 smithi097 ceph-mon[32419]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:52.247 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:51 smithi186 ceph-mon[32797]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:52.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:51 smithi031 ceph-mon[29164]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:54.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:53 smithi097 ceph-mon[32419]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:54.223 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:53 smithi186 ceph-mon[32797]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:54.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:53 smithi031 ceph-mon[29164]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:55.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:55 smithi186 ceph-mon[32797]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:55.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:55 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:55.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:55 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:55.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:55 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:55.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:55 smithi031 ceph-mon[29164]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:55.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:55 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:55.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:55 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:55.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:55 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:55.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:55 smithi097 ceph-mon[32419]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:55.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:55 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:55.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:55 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:55.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:55 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:57.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:57 smithi186 ceph-mon[32797]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:57 smithi031 ceph-mon[29164]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:57.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:57 smithi097 ceph-mon[32419]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:58.721 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:58.721+0000 7f1daaffd700 1 -- 172.21.15.186:0/4000095564 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f1dac1a1f60 con 0x7f1d940a3c40 2024-04-26T17:51:58.721 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 11 on host 'smithi186' 2024-04-26T17:51:58.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:58.723+0000 7f1db3c08700 1 -- 172.21.15.186:0/4000095564 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f1d940a3c40 msgr2=0x7f1d940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:51:58.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:58.723+0000 7f1db3c08700 1 --2- 172.21.15.186:0/4000095564 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f1d940a3c40 0x7f1d940a6100 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c006fd0 tx=0x7f1d9c008040 comp rx=0 tx=0).stop 2024-04-26T17:51:58.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:58.723+0000 7f1db3c08700 1 -- 172.21.15.186:0/4000095564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1dac103570 msgr2=0x7f1dac199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:51:58.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:58.723+0000 7f1db3c08700 1 --2- 172.21.15.186:0/4000095564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1dac103570 0x7f1dac199d60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1da0006070 tx=0x7f1da000cf80 comp rx=0 tx=0).stop 2024-04-26T17:51:58.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:58.723+0000 7f1db3c08700 1 -- 172.21.15.186:0/4000095564 shutdown_connections 2024-04-26T17:51:58.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:58.723+0000 7f1db3c08700 1 --2- 172.21.15.186:0/4000095564 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f1d940a3c40 0x7f1d940a6100 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:58.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:58.723+0000 7f1db3c08700 1 --2- 172.21.15.186:0/4000095564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1dac102370 0x7f1dac199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:58.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:58.723+0000 7f1db3c08700 1 --2- 172.21.15.186:0/4000095564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dac103f10 0x7f1dac1a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:58.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:58.723+0000 7f1db3c08700 1 --2- 172.21.15.186:0/4000095564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1dac103570 0x7f1dac199d60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:51:58.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:58.723+0000 7f1db3c08700 1 -- 172.21.15.186:0/4000095564 >> 172.21.15.186:0/4000095564 conn(0x7f1dac0fd910 msgr2=0x7f1dac06bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:51:58.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:58.723+0000 7f1db3c08700 1 -- 172.21.15.186:0/4000095564 shutdown_connections 2024-04-26T17:51:58.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:51:58.723+0000 7f1db3c08700 1 -- 172.21.15.186:0/4000095564 wait complete. 2024-04-26T17:51:59.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:59.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:59.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:58 smithi186 ceph-mon[32797]: from='osd.11 [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T17:51:59.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:58 smithi186 ceph-mon[32797]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T17:51:59.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:58 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:59.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:59.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:59.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:58 smithi031 ceph-mon[29164]: from='osd.11 [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T17:51:59.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:58 smithi031 ceph-mon[29164]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T17:51:59.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:58 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:59.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:51:59.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:51:59.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:58 smithi097 ceph-mon[32419]: from='osd.11 [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T17:51:59.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:58 smithi097 ceph-mon[32419]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T17:51:59.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:51:59.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:58 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:51:59.360 DEBUG:teuthology.orchestra.run.smithi186:osd.11> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.11.service 2024-04-26T17:51:59.363 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-26T17:51:59.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T17:51:59.622 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:51:59.852 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:59 smithi186 ceph-mon[32797]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:51:59.852 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:59 smithi186 ceph-mon[32797]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T17:51:59.852 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:59 smithi186 ceph-mon[32797]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T17:51:59.852 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:59 smithi186 ceph-mon[32797]: from='osd.11 [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:51:59.852 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:51:59.852 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:51:59 smithi186 ceph-mon[32797]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:51:59.852 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 17:51:59 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T17:51:59.800+0000 7f2e226d9700 -1 osd.11 0 waiting for initial osdmap 2024-04-26T17:51:59.852 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 17:51:59 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T17:51:59.814+0000 7f2e1e4cc700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T17:52:00.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:59 smithi031 ceph-mon[29164]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:52:00.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:59 smithi031 ceph-mon[29164]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T17:52:00.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:59 smithi031 ceph-mon[29164]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T17:52:00.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:59 smithi031 ceph-mon[29164]: from='osd.11 [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:52:00.027 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:52:00.027 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:51:59 smithi031 ceph-mon[29164]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:52:00.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:59 smithi097 ceph-mon[32419]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:52:00.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:59 smithi097 ceph-mon[32419]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T17:52:00.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:59 smithi097 ceph-mon[32419]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T17:52:00.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:59 smithi097 ceph-mon[32419]: from='osd.11 [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:52:00.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:52:00.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:51:59 smithi097 ceph-mon[32419]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T17:52:00.878 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:00 smithi186 ceph-mon[32797]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T17:52:00.879 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:00 smithi186 ceph-mon[32797]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T17:52:00.879 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:00 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:52:00.879 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:00 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:52:01.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:00 smithi031 ceph-mon[29164]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T17:52:01.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:00 smithi031 ceph-mon[29164]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T17:52:01.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:00 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:52:01.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:00 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:52:01.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:00 smithi097 ceph-mon[32419]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T17:52:01.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:00 smithi097 ceph-mon[32419]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T17:52:01.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:00 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:52:01.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:00 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:52:01.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.333+0000 7fb9a6e46700 1 -- 172.21.15.31:0/1245514959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9a01017b0 msgr2=0x7fb9a0101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:01.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.333+0000 7fb9a6e46700 1 --2- 172.21.15.31:0/1245514959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9a01017b0 0x7fb9a0101bd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb9940099c0 tx=0x7fb994009cd0 comp rx=0 tx=0).stop 2024-04-26T17:52:01.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.333+0000 7fb9a6e46700 1 -- 172.21.15.31:0/1245514959 shutdown_connections 2024-04-26T17:52:01.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.333+0000 7fb9a6e46700 1 --2- 172.21.15.31:0/1245514959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9a01051d0 0x7fb9a010ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:01.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.333+0000 7fb9a6e46700 1 --2- 172.21.15.31:0/1245514959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9a01017b0 0x7fb9a0101bd0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:01.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.333+0000 7fb9a6e46700 1 --2- 172.21.15.31:0/1245514959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a0104830 0x7fb9a0104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:01.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.333+0000 7fb9a6e46700 1 -- 172.21.15.31:0/1245514959 >> 172.21.15.31:0/1245514959 conn(0x7fb9a00fd0f0 msgr2=0x7fb9a00ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:01.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.333+0000 7fb9a6e46700 1 -- 172.21.15.31:0/1245514959 shutdown_connections 2024-04-26T17:52:01.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.333+0000 7fb9a6e46700 1 -- 172.21.15.31:0/1245514959 wait complete. 2024-04-26T17:52:01.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a6e46700 1 Processor -- start 2024-04-26T17:52:01.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a6e46700 1 -- start start 2024-04-26T17:52:01.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a6e46700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9a01017b0 0x7fb9a0104190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:01.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a6e46700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a0104830 0x7fb9a01027e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:01.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a6e46700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9a01051d0 0x7fb9a0102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:01.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a6e46700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a010efd0 con 0x7fb9a01017b0 2024-04-26T17:52:01.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a6e46700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a010ee50 con 0x7fb9a01051d0 2024-04-26T17:52:01.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a6e46700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a010f150 con 0x7fb9a0104830 2024-04-26T17:52:01.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a4be2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9a01017b0 0x7fb9a0104190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:01.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a4be2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9a01017b0 0x7fb9a0104190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35664/0 (socket says 172.21.15.31:35664) 2024-04-26T17:52:01.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a4be2700 1 -- 172.21.15.31:0/2396381288 learned_addr learned my addr 172.21.15.31:0/2396381288 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:52:01.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb99ffff700 1 --2- 172.21.15.31:0/2396381288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a0104830 0x7fb9a01027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:01.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a53e3700 1 --2- 172.21.15.31:0/2396381288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9a01051d0 0x7fb9a0102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:01.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a4be2700 1 -- 172.21.15.31:0/2396381288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a0104830 msgr2=0x7fb9a01027e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:01.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a4be2700 1 --2- 172.21.15.31:0/2396381288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a0104830 0x7fb9a01027e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:01.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a4be2700 1 -- 172.21.15.31:0/2396381288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9a01051d0 msgr2=0x7fb9a0102d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:01.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a4be2700 1 --2- 172.21.15.31:0/2396381288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9a01051d0 0x7fb9a0102d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:01.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.334+0000 7fb9a4be2700 1 -- 172.21.15.31:0/2396381288 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a01035d0 con 0x7fb9a01017b0 2024-04-26T17:52:01.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.335+0000 7fb9a4be2700 1 --2- 172.21.15.31:0/2396381288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9a01017b0 0x7fb9a0104190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb994006010 tx=0x7fb9940047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:01.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.335+0000 7fb99dffb700 1 -- 172.21.15.31:0/2396381288 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb994027070 con 0x7fb9a01017b0 2024-04-26T17:52:01.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.335+0000 7fb99dffb700 1 -- 172.21.15.31:0/2396381288 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9940040e0 con 0x7fb9a01017b0 2024-04-26T17:52:01.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.335+0000 7fb9a6e46700 1 -- 172.21.15.31:0/2396381288 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a01a3770 con 0x7fb9a01017b0 2024-04-26T17:52:01.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.335+0000 7fb99dffb700 1 -- 172.21.15.31:0/2396381288 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99400f460 con 0x7fb9a01017b0 2024-04-26T17:52:01.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.335+0000 7fb9a6e46700 1 -- 172.21.15.31:0/2396381288 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a01a3bc0 con 0x7fb9a01017b0 2024-04-26T17:52:01.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.336+0000 7fb9a6e46700 1 -- 172.21.15.31:0/2396381288 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9a006a110 con 0x7fb9a01017b0 2024-04-26T17:52:01.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.337+0000 7fb99dffb700 1 -- 172.21.15.31:0/2396381288 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb99402a070 con 0x7fb9a01017b0 2024-04-26T17:52:01.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.338+0000 7fb99dffb700 1 --2- 172.21.15.31:0/2396381288 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb9900a3c40 0x7fb9900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:01.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.338+0000 7fb99ffff700 1 --2- 172.21.15.31:0/2396381288 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb9900a3c40 0x7fb9900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:01.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.338+0000 7fb99dffb700 1 -- 172.21.15.31:0/2396381288 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7fb9940d0e70 con 0x7fb9a01017b0 2024-04-26T17:52:01.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.338+0000 7fb99ffff700 1 --2- 172.21.15.31:0/2396381288 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb9900a3c40 0x7fb9900a6100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb98c004630 tx=0x7fb98c00b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:01.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.343+0000 7fb99dffb700 1 -- 172.21.15.31:0/2396381288 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb99405eff0 con 0x7fb9a01017b0 2024-04-26T17:52:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.553+0000 7fb9a6e46700 1 -- 172.21.15.31:0/2396381288 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fb9a0064030 con 0x7fb9a01017b0 2024-04-26T17:52:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.554+0000 7fb99dffb700 1 -- 172.21.15.31:0/2396381288 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7fb994013070 con 0x7fb9a01017b0 2024-04-26T17:52:01.554 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:52:01.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.557+0000 7fb9a6e46700 1 -- 172.21.15.31:0/2396381288 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb9900a3c40 msgr2=0x7fb9900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:01.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.557+0000 7fb9a6e46700 1 --2- 172.21.15.31:0/2396381288 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb9900a3c40 0x7fb9900a6100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb98c004630 tx=0x7fb98c00b3c0 comp rx=0 tx=0).stop 2024-04-26T17:52:01.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.557+0000 7fb9a6e46700 1 -- 172.21.15.31:0/2396381288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9a01017b0 msgr2=0x7fb9a0104190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:01.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.557+0000 7fb9a6e46700 1 --2- 172.21.15.31:0/2396381288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9a01017b0 0x7fb9a0104190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb994006010 tx=0x7fb9940047b0 comp rx=0 tx=0).stop 2024-04-26T17:52:01.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.557+0000 7fb9a6e46700 1 -- 172.21.15.31:0/2396381288 shutdown_connections 2024-04-26T17:52:01.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.557+0000 7fb9a6e46700 1 --2- 172.21.15.31:0/2396381288 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb9900a3c40 0x7fb9900a6100 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:01.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.557+0000 7fb9a6e46700 1 --2- 172.21.15.31:0/2396381288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9a01051d0 0x7fb9a0102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:01.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.557+0000 7fb9a6e46700 1 --2- 172.21.15.31:0/2396381288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9a01017b0 0x7fb9a0104190 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:01.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.557+0000 7fb9a6e46700 1 --2- 172.21.15.31:0/2396381288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a0104830 0x7fb9a01027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:01.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.557+0000 7fb9a6e46700 1 -- 172.21.15.31:0/2396381288 >> 172.21.15.31:0/2396381288 conn(0x7fb9a00fd0f0 msgr2=0x7fb9a0107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:01.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.557+0000 7fb9a6e46700 1 -- 172.21.15.31:0/2396381288 shutdown_connections 2024-04-26T17:52:01.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:01.557+0000 7fb9a6e46700 1 -- 172.21.15.31:0/2396381288 wait complete. 2024-04-26T17:52:02.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:01 smithi031 ceph-mon[29164]: purged_snaps scrub starts 2024-04-26T17:52:02.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:01 smithi031 ceph-mon[29164]: purged_snaps scrub ok 2024-04-26T17:52:02.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:01 smithi031 ceph-mon[29164]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:52:02.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:52:02.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:01 smithi031 ceph-mon[29164]: osd.11 [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352] boot 2024-04-26T17:52:02.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:01 smithi031 ceph-mon[29164]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T17:52:02.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:52:02.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:01 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2396381288' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T17:52:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:01 smithi097 ceph-mon[32419]: purged_snaps scrub starts 2024-04-26T17:52:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:01 smithi097 ceph-mon[32419]: purged_snaps scrub ok 2024-04-26T17:52:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:01 smithi097 ceph-mon[32419]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:52:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:52:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:01 smithi097 ceph-mon[32419]: osd.11 [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352] boot 2024-04-26T17:52:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:01 smithi097 ceph-mon[32419]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T17:52:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:52:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:01 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2396381288' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T17:52:02.165 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:01 smithi186 ceph-mon[32797]: purged_snaps scrub starts 2024-04-26T17:52:02.165 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:01 smithi186 ceph-mon[32797]: purged_snaps scrub ok 2024-04-26T17:52:02.165 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:01 smithi186 ceph-mon[32797]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-26T17:52:02.165 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:52:02.165 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:01 smithi186 ceph-mon[32797]: osd.11 [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352] boot 2024-04-26T17:52:02.166 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:01 smithi186 ceph-mon[32797]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T17:52:02.166 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:52:02.166 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:01 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2396381288' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T17:52:02.334 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1714153920,"num_in_osds":12,"osd_in_since":1714153900,"num_remapped_pgs":0} 2024-04-26T17:52:02.335 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T17:52:02.603 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:52:02.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:02 smithi031 ceph-mon[29164]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T17:52:02.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:02.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:02.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:52:02.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:52:02.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:52:02.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:52:02.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:02.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:02.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:52:02.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:02 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:03.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:02 smithi097 ceph-mon[32419]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T17:52:03.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:03.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:03.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:52:03.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:52:03.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:52:03.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:52:03.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:03.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:03.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:52:03.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:02 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:02 smithi186 ceph-mon[32797]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T17:52:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:52:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:52:03.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:52:03.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:52:03.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:03.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:03.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:52:03.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:02 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:04.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:03 smithi097 ceph-mon[32419]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:04.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:03 smithi097 ceph-mon[32419]: Detected new or changed devices on smithi186 2024-04-26T17:52:04.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:03 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi186 to 4272M 2024-04-26T17:52:04.164 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:03 smithi031 ceph-mon[29164]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:04.164 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:03 smithi031 ceph-mon[29164]: Detected new or changed devices on smithi186 2024-04-26T17:52:04.164 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:03 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi186 to 4272M 2024-04-26T17:52:04.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:03 smithi186 ceph-mon[32797]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:04.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:03 smithi186 ceph-mon[32797]: Detected new or changed devices on smithi186 2024-04-26T17:52:04.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:03 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi186 to 4272M 2024-04-26T17:52:04.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.396+0000 7f9883819700 1 -- 172.21.15.31:0/388610494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f987c102320 msgr2=0x7f987c1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:04.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.396+0000 7f9883819700 1 --2- 172.21.15.31:0/388610494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f987c102320 0x7f987c1027a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f986c009990 tx=0x7f986c009ca0 comp rx=0 tx=0).stop 2024-04-26T17:52:04.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.396+0000 7f9883819700 1 -- 172.21.15.31:0/388610494 shutdown_connections 2024-04-26T17:52:04.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.396+0000 7f9883819700 1 --2- 172.21.15.31:0/388610494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f987c102ce0 0x7f987c10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:04.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.396+0000 7f9883819700 1 --2- 172.21.15.31:0/388610494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f987c108390 0x7f987c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:04.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.396+0000 7f9883819700 1 --2- 172.21.15.31:0/388610494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f987c102320 0x7f987c1027a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:04.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.396+0000 7f9883819700 1 -- 172.21.15.31:0/388610494 >> 172.21.15.31:0/388610494 conn(0x7f987c0792d0 msgr2=0x7f987c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:04.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.396+0000 7f9883819700 1 -- 172.21.15.31:0/388610494 shutdown_connections 2024-04-26T17:52:04.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.396+0000 7f9883819700 1 -- 172.21.15.31:0/388610494 wait complete. 2024-04-26T17:52:04.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.397+0000 7f9883819700 1 Processor -- start 2024-04-26T17:52:04.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.397+0000 7f9883819700 1 -- start start 2024-04-26T17:52:04.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.397+0000 7f9883819700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f987c102320 0x7f987c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:04.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.397+0000 7f9883819700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f987c102ce0 0x7f987c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:04.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.397+0000 7f9883819700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f987c108390 0x7f987c1998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:04.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.397+0000 7f9883819700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f987c1124b0 con 0x7f987c102ce0 2024-04-26T17:52:04.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.397+0000 7f9883819700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f987c112330 con 0x7f987c102320 2024-04-26T17:52:04.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.397+0000 7f9883819700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f987c112630 con 0x7f987c108390 2024-04-26T17:52:04.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.397+0000 7f9880db4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f987c102ce0 0x7f987c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:04.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.397+0000 7f9880db4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f987c102ce0 0x7f987c19fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35670/0 (socket says 172.21.15.31:35670) 2024-04-26T17:52:04.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.397+0000 7f9880db4700 1 -- 172.21.15.31:0/3660110659 learned_addr learned my addr 172.21.15.31:0/3660110659 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:52:04.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.397+0000 7f9881db6700 1 --2- 172.21.15.31:0/3660110659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f987c108390 0x7f987c1998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:04.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.397+0000 7f98815b5700 1 --2- 172.21.15.31:0/3660110659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f987c102320 0x7f987c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:04.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.398+0000 7f9880db4700 1 -- 172.21.15.31:0/3660110659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f987c108390 msgr2=0x7f987c1998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:04.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.398+0000 7f9880db4700 1 --2- 172.21.15.31:0/3660110659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f987c108390 0x7f987c1998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:04.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.398+0000 7f9880db4700 1 -- 172.21.15.31:0/3660110659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f987c102320 msgr2=0x7f987c19f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:04.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.398+0000 7f9880db4700 1 --2- 172.21.15.31:0/3660110659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f987c102320 0x7f987c19f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:04.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.398+0000 7f9880db4700 1 -- 172.21.15.31:0/3660110659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987c19a040 con 0x7f987c102ce0 2024-04-26T17:52:04.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.398+0000 7f9880db4700 1 --2- 172.21.15.31:0/3660110659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f987c102ce0 0x7f987c19fc60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f986c000c00 tx=0x7f986c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:04.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.398+0000 7f98727fc700 1 -- 172.21.15.31:0/3660110659 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c027070 con 0x7f987c102ce0 2024-04-26T17:52:04.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.398+0000 7f98727fc700 1 -- 172.21.15.31:0/3660110659 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f986c003f20 con 0x7f987c102ce0 2024-04-26T17:52:04.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.398+0000 7f98727fc700 1 -- 172.21.15.31:0/3660110659 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c022ca0 con 0x7f987c102ce0 2024-04-26T17:52:04.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.398+0000 7f9883819700 1 -- 172.21.15.31:0/3660110659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f987c19a2d0 con 0x7f987c102ce0 2024-04-26T17:52:04.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.398+0000 7f9883819700 1 -- 172.21.15.31:0/3660110659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f987c1a65f0 con 0x7f987c102ce0 2024-04-26T17:52:04.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.399+0000 7f9883819700 1 -- 172.21.15.31:0/3660110659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f987c103a40 con 0x7f987c102ce0 2024-04-26T17:52:04.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.400+0000 7f98727fc700 1 -- 172.21.15.31:0/3660110659 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f986c02a070 con 0x7f987c102ce0 2024-04-26T17:52:04.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.401+0000 7f98727fc700 1 --2- 172.21.15.31:0/3660110659 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f98680a3cc0 0x7f98680a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:04.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.401+0000 7f98815b5700 1 --2- 172.21.15.31:0/3660110659 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f98680a3cc0 0x7f98680a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:04.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.401+0000 7f98727fc700 1 -- 172.21.15.31:0/3660110659 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f986c0d0cd0 con 0x7f987c102ce0 2024-04-26T17:52:04.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.401+0000 7f98815b5700 1 --2- 172.21.15.31:0/3660110659 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f98680a3cc0 0x7f98680a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9878005950 tx=0x7f987800a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:04.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.405+0000 7f98727fc700 1 -- 172.21.15.31:0/3660110659 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f986c05eff0 con 0x7f987c102ce0 2024-04-26T17:52:04.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.621+0000 7f9883819700 1 -- 172.21.15.31:0/3660110659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f987c1a67c0 con 0x7f987c102ce0 2024-04-26T17:52:04.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.622+0000 7f98727fc700 1 -- 172.21.15.31:0/3660110659 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18427 (secure 0 0 0) 0x7f986c013070 con 0x7f987c102ce0 2024-04-26T17:52:04.624 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:52:04.624 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":81,"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","created":"2024-04-26T17:44:48.618250+0000","modified":"2024-04-26T17:52:01.808177+0000","last_up_change":"2024-04-26T17:52:00.798886+0000","last_in_change":"2024-04-26T17:51:40.613593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T17:48:12.307250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f684ee5-eeee-454a-a00b-e5d6d2901572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6803","nonce":1957398328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6805","nonce":1957398328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6809","nonce":1957398328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6807","nonce":1957398328}]},"public_addr":"172.21.15.31:6803/1957398328","cluster_addr":"172.21.15.31:6805/1957398328","heartbeat_back_addr":"172.21.15.31:6809/1957398328","heartbeat_front_addr":"172.21.15.31:6807/1957398328","state":["exists","up"]},{"osd":1,"uuid":"08be6b26-cb60-41f7-be8b-01a0c93ebafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6801","nonce":4209866427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6803","nonce":4209866427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6807","nonce":4209866427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6805","nonce":4209866427}]},"public_addr":"172.21.15.97:6801/4209866427","cluster_addr":"172.21.15.97:6803/4209866427","heartbeat_back_addr":"172.21.15.97:6807/4209866427","heartbeat_front_addr":"172.21.15.97:6805/4209866427","state":["exists","up"]},{"osd":2,"uuid":"498216b1-a0ac-4923-ade8-ce50fb97d472","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6801","nonce":223535147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6803","nonce":223535147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6807","nonce":223535147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6805","nonce":223535147}]},"public_addr":"172.21.15.186:6801/223535147","cluster_addr":"172.21.15.186:6803/223535147","heartbeat_back_addr":"172.21.15.186:6807/223535147","heartbeat_front_addr":"172.21.15.186:6805/223535147","state":["exists","up"]},{"osd":3,"uuid":"568a352d-a430-4f0f-9a49-6f992e0ceae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6811","nonce":2640112341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6813","nonce":2640112341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6817","nonce":2640112341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6815","nonce":2640112341}]},"public_addr":"172.21.15.31:6811/2640112341","cluster_addr":"172.21.15.31:6813/2640112341","heartbeat_back_addr":"172.21.15.31:6817/2640112341","heartbeat_front_addr":"172.21.15.31:6815/2640112341","state":["exists","up"]},{"osd":4,"uuid":"112ccbf7-24e8-473c-9852-2e7167dd3108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6809","nonce":1359263143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6811","nonce":1359263143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6815","nonce":1359263143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6813","nonce":1359263143}]},"public_addr":"172.21.15.97:6809/1359263143","cluster_addr":"172.21.15.97:6811/1359263143","heartbeat_back_addr":"172.21.15.97:6815/1359263143","heartbeat_front_addr":"172.21.15.97:6813/1359263143","state":["exists","up"]},{"osd":5,"uuid":"eb17759e-1ad8-4bda-9f61-54d2a9781535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6809","nonce":2006134626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6811","nonce":2006134626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6815","nonce":2006134626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6813","nonce":2006134626}]},"public_addr":"172.21.15.186:6809/2006134626","cluster_addr":"172.21.15.186:6811/2006134626","heartbeat_back_addr":"172.21.15.186:6815/2006134626","heartbeat_front_addr":"172.21.15.186:6813/2006134626","state":["exists","up"]},{"osd":6,"uuid":"f1d11e38-40e9-4db4-868f-145f1ffc0d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6819","nonce":2955759395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6821","nonce":2955759395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6825","nonce":2955759395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6823","nonce":2955759395}]},"public_addr":"172.21.15.31:6819/2955759395","cluster_addr":"172.21.15.31:6821/2955759395","heartbeat_back_addr":"172.21.15.31:6825/2955759395","heartbeat_front_addr":"172.21.15.31:6823/2955759395","state":["exists","up"]},{"osd":7,"uuid":"4e95f728-23ff-4bca-9b94-afde2662a5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6817","nonce":3940083075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6819","nonce":3940083075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6823","nonce":3940083075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6821","nonce":3940083075}]},"public_addr":"172.21.15.97:6817/3940083075","cluster_addr":"172.21.15.97:6819/3940083075","heartbeat_back_addr":"172.21.15.97:6823/3940083075","heartbeat_front_addr":"172.21.15.97:6821/3940083075","state":["exists","up"]},{"osd":8,"uuid":"5f91c915-2568-4d20-b645-00dad2efe952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6817","nonce":3193558170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6819","nonce":3193558170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6823","nonce":3193558170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6821","nonce":3193558170}]},"public_addr":"172.21.15.186:6817/3193558170","cluster_addr":"172.21.15.186:6819/3193558170","heartbeat_back_addr":"172.21.15.186:6823/3193558170","heartbeat_front_addr":"172.21.15.186:6821/3193558170","state":["exists","up"]},{"osd":9,"uuid":"12fda87d-830a-461f-a6cb-4699c52dc5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6827","nonce":3660801905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6829","nonce":3660801905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6833","nonce":3660801905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6831","nonce":3660801905}]},"public_addr":"172.21.15.31:6827/3660801905","cluster_addr":"172.21.15.31:6829/3660801905","heartbeat_back_addr":"172.21.15.31:6833/3660801905","heartbeat_front_addr":"172.21.15.31:6831/3660801905","state":["exists","up"]},{"osd":10,"uuid":"92a639b3-918b-485c-bd30-258b4224993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6825","nonce":4097344903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6827","nonce":4097344903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6831","nonce":4097344903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6829","nonce":4097344903}]},"public_addr":"172.21.15.97:6825/4097344903","cluster_addr":"172.21.15.97:6827/4097344903","heartbeat_back_addr":"172.21.15.97:6831/4097344903","heartbeat_front_addr":"172.21.15.97:6829/4097344903","state":["exists","up"]},{"osd":11,"uuid":"842aa599-d21d-4e4a-b872-e9ac20d8c0ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6825","nonce":1063973352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6827","nonce":1063973352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6831","nonce":1063973352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6829","nonce":1063973352}]},"public_addr":"172.21.15.186:6825/1063973352","cluster_addr":"172.21.15.186:6827/1063973352","heartbeat_back_addr":"172.21.15.186:6831/1063973352","heartbeat_front_addr":"172.21.15.186:6829/1063973352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:21.305331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:44.877932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:08.784665+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:34.381762+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:58.999358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:23.620237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:49.961719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:15.363702+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:40.599703+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:07.391645+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:32.800851+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:59.384148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/718966776":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/2035551671":"2024-04-27T17:45:48.218816+0000","172.21.15.31:6801/153341213":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/810461143":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/3148082955":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6801/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/288292084":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/3101596726":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/153341213":"2024-04-27T17:45:48.218816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T17:52:04.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.625+0000 7f9883819700 1 -- 172.21.15.31:0/3660110659 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f98680a3cc0 msgr2=0x7f98680a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:04.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.625+0000 7f9883819700 1 --2- 172.21.15.31:0/3660110659 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f98680a3cc0 0x7f98680a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9878005950 tx=0x7f987800a300 comp rx=0 tx=0).stop 2024-04-26T17:52:04.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.625+0000 7f9883819700 1 -- 172.21.15.31:0/3660110659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f987c102ce0 msgr2=0x7f987c19fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:04.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.625+0000 7f9883819700 1 --2- 172.21.15.31:0/3660110659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f987c102ce0 0x7f987c19fc60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f986c000c00 tx=0x7f986c0047b0 comp rx=0 tx=0).stop 2024-04-26T17:52:04.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.625+0000 7f9883819700 1 -- 172.21.15.31:0/3660110659 shutdown_connections 2024-04-26T17:52:04.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.625+0000 7f9883819700 1 --2- 172.21.15.31:0/3660110659 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f98680a3cc0 0x7f98680a6180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:04.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.625+0000 7f9883819700 1 --2- 172.21.15.31:0/3660110659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f987c102320 0x7f987c19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:04.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.625+0000 7f9883819700 1 --2- 172.21.15.31:0/3660110659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f987c108390 0x7f987c1998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:04.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.625+0000 7f9883819700 1 --2- 172.21.15.31:0/3660110659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f987c102ce0 0x7f987c19fc60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:04.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.625+0000 7f9883819700 1 -- 172.21.15.31:0/3660110659 >> 172.21.15.31:0/3660110659 conn(0x7f987c0792d0 msgr2=0x7f987c101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:04.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.625+0000 7f9883819700 1 -- 172.21.15.31:0/3660110659 shutdown_connections 2024-04-26T17:52:04.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:04.625+0000 7f9883819700 1 -- 172.21.15.31:0/3660110659 wait complete. 2024-04-26T17:52:04.997 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:04 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3660110659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:52:05.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:04 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3660110659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:52:05.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:04 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3660110659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:52:05.332 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T17:48:12.307250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T17:52:05.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-26T17:52:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:52:05.958 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:05 smithi031 ceph-mon[29164]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:06.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:05 smithi097 ceph-mon[32419]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:06.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:05 smithi186 ceph-mon[32797]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:07.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.357+0000 7fee51431700 1 -- 172.21.15.31:0/3302108495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee4c06c590 msgr2=0x7fee4c110310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:07.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.357+0000 7fee51431700 1 --2- 172.21.15.31:0/3302108495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee4c06c590 0x7fee4c110310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fee4400b0a0 tx=0x7fee4400b3b0 comp rx=0 tx=0).stop 2024-04-26T17:52:07.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 -- 172.21.15.31:0/3302108495 shutdown_connections 2024-04-26T17:52:07.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 --2- 172.21.15.31:0/3302108495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee4c06c590 0x7fee4c110310 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:07.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 --2- 172.21.15.31:0/3302108495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee4c1094d0 0x7fee4c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:07.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 --2- 172.21.15.31:0/3302108495 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee4c06bbd0 0x7fee4c06c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:07.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 -- 172.21.15.31:0/3302108495 >> 172.21.15.31:0/3302108495 conn(0x7fee4c0793c0 msgr2=0x7fee4c0797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:07.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 -- 172.21.15.31:0/3302108495 shutdown_connections 2024-04-26T17:52:07.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 -- 172.21.15.31:0/3302108495 wait complete. 2024-04-26T17:52:07.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 Processor -- start 2024-04-26T17:52:07.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 -- start start 2024-04-26T17:52:07.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee4c06bbd0 0x7fee4c102390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:07.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee4c06c590 0x7fee4c1028d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:07.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee4c1094d0 0x7fee4c102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:07.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c113600 con 0x7fee4c06bbd0 2024-04-26T17:52:07.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c113480 con 0x7fee4c06c590 2024-04-26T17:52:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.358+0000 7fee51431700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c113780 con 0x7fee4c1094d0 2024-04-26T17:52:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.359+0000 7fee4affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee4c06bbd0 0x7fee4c102390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.359+0000 7fee4affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee4c06bbd0 0x7fee4c102390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59844/0 (socket says 172.21.15.31:59844) 2024-04-26T17:52:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.359+0000 7fee4affd700 1 -- 172.21.15.31:0/2047306619 learned_addr learned my addr 172.21.15.31:0/2047306619 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:52:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.359+0000 7fee43fff700 1 --2- 172.21.15.31:0/2047306619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee4c06c590 0x7fee4c1028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.359+0000 7fee4b7fe700 1 --2- 172.21.15.31:0/2047306619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee4c1094d0 0x7fee4c102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.359+0000 7fee4affd700 1 -- 172.21.15.31:0/2047306619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee4c1094d0 msgr2=0x7fee4c102e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.359+0000 7fee4affd700 1 --2- 172.21.15.31:0/2047306619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee4c1094d0 0x7fee4c102e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.359+0000 7fee4affd700 1 -- 172.21.15.31:0/2047306619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee4c06c590 msgr2=0x7fee4c1028d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.359+0000 7fee4affd700 1 --2- 172.21.15.31:0/2047306619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee4c06c590 0x7fee4c1028d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.359+0000 7fee4affd700 1 -- 172.21.15.31:0/2047306619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4c1a8000 con 0x7fee4c06bbd0 2024-04-26T17:52:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.359+0000 7fee4affd700 1 --2- 172.21.15.31:0/2047306619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee4c06bbd0 0x7fee4c102390 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fee3c00d980 tx=0x7fee3c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.359+0000 7fee48ff9700 1 -- 172.21.15.31:0/2047306619 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c017070 con 0x7fee4c06bbd0 2024-04-26T17:52:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.359+0000 7fee48ff9700 1 -- 172.21.15.31:0/2047306619 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee3c005020 con 0x7fee4c06bbd0 2024-04-26T17:52:07.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.359+0000 7fee51431700 1 -- 172.21.15.31:0/2047306619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee4c1a82f0 con 0x7fee4c06bbd0 2024-04-26T17:52:07.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.360+0000 7fee48ff9700 1 -- 172.21.15.31:0/2047306619 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c009b50 con 0x7fee4c06bbd0 2024-04-26T17:52:07.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.360+0000 7fee51431700 1 -- 172.21.15.31:0/2047306619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee4c1a87e0 con 0x7fee4c06bbd0 2024-04-26T17:52:07.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.361+0000 7fee51431700 1 -- 172.21.15.31:0/2047306619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee4c104b80 con 0x7fee4c06bbd0 2024-04-26T17:52:07.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.362+0000 7fee48ff9700 1 -- 172.21.15.31:0/2047306619 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fee3c0051d0 con 0x7fee4c06bbd0 2024-04-26T17:52:07.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.362+0000 7fee48ff9700 1 --2- 172.21.15.31:0/2047306619 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fee380a3c40 0x7fee380a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:07.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.363+0000 7fee43fff700 1 --2- 172.21.15.31:0/2047306619 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fee380a3c40 0x7fee380a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:07.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.363+0000 7fee48ff9700 1 -- 172.21.15.31:0/2047306619 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fee3c0cfea0 con 0x7fee4c06bbd0 2024-04-26T17:52:07.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.363+0000 7fee43fff700 1 --2- 172.21.15.31:0/2047306619 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fee380a3c40 0x7fee380a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0798c0 tx=0x7fee3400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:07.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.367+0000 7fee48ff9700 1 -- 172.21.15.31:0/2047306619 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee3c020020 con 0x7fee4c06bbd0 2024-04-26T17:52:07.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.570+0000 7fee51431700 1 -- 172.21.15.31:0/2047306619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fee4c02e490 con 0x7fee4c06bbd0 2024-04-26T17:52:07.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.571+0000 7fee48ff9700 1 -- 172.21.15.31:0/2047306619 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fee3c09cc80 con 0x7fee4c06bbd0 2024-04-26T17:52:07.571 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-26T17:52:07.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.573+0000 7fee51431700 1 -- 172.21.15.31:0/2047306619 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fee380a3c40 msgr2=0x7fee380a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:07.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.573+0000 7fee51431700 1 --2- 172.21.15.31:0/2047306619 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fee380a3c40 0x7fee380a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0798c0 tx=0x7fee3400b410 comp rx=0 tx=0).stop 2024-04-26T17:52:07.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.573+0000 7fee51431700 1 -- 172.21.15.31:0/2047306619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee4c06bbd0 msgr2=0x7fee4c102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:07.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.573+0000 7fee51431700 1 --2- 172.21.15.31:0/2047306619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee4c06bbd0 0x7fee4c102390 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fee3c00d980 tx=0x7fee3c00dc90 comp rx=0 tx=0).stop 2024-04-26T17:52:07.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.573+0000 7fee51431700 1 -- 172.21.15.31:0/2047306619 shutdown_connections 2024-04-26T17:52:07.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.573+0000 7fee51431700 1 --2- 172.21.15.31:0/2047306619 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fee380a3c40 0x7fee380a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:07.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.573+0000 7fee51431700 1 --2- 172.21.15.31:0/2047306619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee4c1094d0 0x7fee4c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:07.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.573+0000 7fee51431700 1 --2- 172.21.15.31:0/2047306619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee4c06bbd0 0x7fee4c102390 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:07.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.573+0000 7fee51431700 1 --2- 172.21.15.31:0/2047306619 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee4c06c590 0x7fee4c1028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:07.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.573+0000 7fee51431700 1 -- 172.21.15.31:0/2047306619 >> 172.21.15.31:0/2047306619 conn(0x7fee4c0793c0 msgr2=0x7fee4c1002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:07.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.573+0000 7fee51431700 1 -- 172.21.15.31:0/2047306619 shutdown_connections 2024-04-26T17:52:07.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:07.573+0000 7fee51431700 1 -- 172.21.15.31:0/2047306619 wait complete. 2024-04-26T17:52:08.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:07 smithi097 ceph-mon[32419]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:08.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:07 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2047306619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T17:52:08.194 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:07 smithi031 ceph-mon[29164]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:08.194 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:07 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2047306619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T17:52:08.225 INFO:tasks.cephadm:Adding mds.a on smithi031 2024-04-26T17:52:08.225 INFO:tasks.cephadm:Adding mds.d on smithi031 2024-04-26T17:52:08.225 INFO:tasks.cephadm:Adding mds.g on smithi031 2024-04-26T17:52:08.225 INFO:tasks.cephadm:Adding mds.j on smithi031 2024-04-26T17:52:08.225 INFO:tasks.cephadm:Adding mds.b on smithi097 2024-04-26T17:52:08.226 INFO:tasks.cephadm:Adding mds.e on smithi097 2024-04-26T17:52:08.226 INFO:tasks.cephadm:Adding mds.h on smithi097 2024-04-26T17:52:08.226 INFO:tasks.cephadm:Adding mds.k on smithi097 2024-04-26T17:52:08.226 INFO:tasks.cephadm:Adding mds.c on smithi186 2024-04-26T17:52:08.226 INFO:tasks.cephadm:Adding mds.f on smithi186 2024-04-26T17:52:08.226 INFO:tasks.cephadm:Adding mds.i on smithi186 2024-04-26T17:52:08.226 INFO:tasks.cephadm:Adding mds.l on smithi186 2024-04-26T17:52:08.226 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi097=b;smithi097=e;smithi097=h;smithi097=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l' 2024-04-26T17:52:08.254 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:07 smithi186 ceph-mon[32797]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:08.254 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:07 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2047306619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T17:52:08.481 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T17:52:09.960 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:09 smithi186 ceph-mon[32797]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:10.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:09 smithi097 ceph-mon[32419]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:10.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.189+0000 7f6a18c43700 1 -- 172.21.15.186:0/1773940214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a14102380 msgr2=0x7f6a141027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:10.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.189+0000 7f6a18c43700 1 --2- 172.21.15.186:0/1773940214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a14102380 0x7f6a141027a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6a04009990 tx=0x7f6a04009ca0 comp rx=0 tx=0).stop 2024-04-26T17:52:10.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.189+0000 7f6a18c43700 1 -- 172.21.15.186:0/1773940214 shutdown_connections 2024-04-26T17:52:10.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.189+0000 7f6a18c43700 1 --2- 172.21.15.186:0/1773940214 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a14103f20 0x7f6a1410a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:10.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.189+0000 7f6a18c43700 1 --2- 172.21.15.186:0/1773940214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a14102380 0x7f6a141027a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:10.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.189+0000 7f6a18c43700 1 --2- 172.21.15.186:0/1773940214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a14103580 0x7f6a141039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:10.190 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.189+0000 7f6a18c43700 1 -- 172.21.15.186:0/1773940214 >> 172.21.15.186:0/1773940214 conn(0x7f6a140fd920 msgr2=0x7f6a140ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:10.190 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.189+0000 7f6a18c43700 1 -- 172.21.15.186:0/1773940214 shutdown_connections 2024-04-26T17:52:10.190 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.189+0000 7f6a18c43700 1 -- 172.21.15.186:0/1773940214 wait complete. 2024-04-26T17:52:10.192 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.189+0000 7f6a18c43700 1 Processor -- start 2024-04-26T17:52:10.192 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a18c43700 1 -- start start 2024-04-26T17:52:10.192 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a18c43700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a14102380 0x7f6a1406bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:10.192 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a18c43700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a14103580 0x7f6a1419e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:10.192 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a18c43700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a14103f20 0x7f6a141a09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:10.192 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a18c43700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1410de70 con 0x7f6a14102380 2024-04-26T17:52:10.192 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a18c43700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1410dcf0 con 0x7f6a14103580 2024-04-26T17:52:10.192 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a18c43700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1410dff0 con 0x7f6a14103f20 2024-04-26T17:52:10.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a12d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a14103f20 0x7f6a141a09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:10.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a12d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a14103f20 0x7f6a141a09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:39648/0 (socket says 172.21.15.186:39648) 2024-04-26T17:52:10.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a12d9d700 1 -- 172.21.15.186:0/806856170 learned_addr learned my addr 172.21.15.186:0/806856170 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T17:52:10.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a09d9b700 1 --2- 172.21.15.186:0/806856170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a14103580 0x7f6a1419e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:10.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a1259c700 1 --2- 172.21.15.186:0/806856170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a14102380 0x7f6a1406bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:10.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a12d9d700 1 -- 172.21.15.186:0/806856170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a14103580 msgr2=0x7f6a1419e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:10.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a12d9d700 1 --2- 172.21.15.186:0/806856170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a14103580 0x7f6a1419e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:10.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a12d9d700 1 -- 172.21.15.186:0/806856170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a14102380 msgr2=0x7f6a1406bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:10.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a12d9d700 1 --2- 172.21.15.186:0/806856170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a14102380 0x7f6a1406bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:10.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.190+0000 7f6a12d9d700 1 -- 172.21.15.186:0/806856170 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a141a12c0 con 0x7f6a14103f20 2024-04-26T17:52:10.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.191+0000 7f6a12d9d700 1 --2- 172.21.15.186:0/806856170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a14103f20 0x7f6a141a09f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c00b970 tx=0x7f6a0c00bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:10.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.191+0000 7f6a0bfff700 1 -- 172.21.15.186:0/806856170 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c019070 con 0x7f6a14103f20 2024-04-26T17:52:10.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.191+0000 7f6a0bfff700 1 -- 172.21.15.186:0/806856170 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a0c00cde0 con 0x7f6a14103f20 2024-04-26T17:52:10.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.191+0000 7f6a18c43700 1 -- 172.21.15.186:0/806856170 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a141a15b0 con 0x7f6a14103f20 2024-04-26T17:52:10.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.191+0000 7f6a0bfff700 1 -- 172.21.15.186:0/806856170 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c007370 con 0x7f6a14103f20 2024-04-26T17:52:10.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.191+0000 7f6a18c43700 1 -- 172.21.15.186:0/806856170 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a141a1aa0 con 0x7f6a14103f20 2024-04-26T17:52:10.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.192+0000 7f6a18c43700 1 -- 172.21.15.186:0/806856170 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a1406a110 con 0x7f6a14103f20 2024-04-26T17:52:10.195 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.193+0000 7f6a0bfff700 1 -- 172.21.15.186:0/806856170 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6a0c01c070 con 0x7f6a14103f20 2024-04-26T17:52:10.195 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.193+0000 7f6a0bfff700 1 --2- 172.21.15.186:0/806856170 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6a000a3c40 0x7f6a000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:10.195 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.194+0000 7f6a0bfff700 1 -- 172.21.15.186:0/806856170 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6a0c0cf490 con 0x7f6a14103f20 2024-04-26T17:52:10.195 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.194+0000 7f6a1259c700 1 --2- 172.21.15.186:0/806856170 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6a000a3c40 0x7f6a000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:10.195 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.194+0000 7f6a1259c700 1 --2- 172.21.15.186:0/806856170 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6a000a3c40 0x7f6a000a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6a04006010 tx=0x7f6a04003860 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:10.198 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.198+0000 7f6a0bfff700 1 -- 172.21.15.186:0/806856170 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a0c09c310 con 0x7f6a14103f20 2024-04-26T17:52:10.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:09 smithi031 ceph-mon[29164]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:10.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.408+0000 7f6a18c43700 1 -- 172.21.15.186:0/806856170 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi097=b;smithi097=e;smithi097=h;smithi097=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}) -- 0x7f6a14063a20 con 0x7f6a000a3c40 2024-04-26T17:52:10.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.416+0000 7f6a0bfff700 1 -- 172.21.15.186:0/806856170 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f6a14063a20 con 0x7f6a000a3c40 2024-04-26T17:52:10.416 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mds.all update... 2024-04-26T17:52:10.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.419+0000 7f6a18c43700 1 -- 172.21.15.186:0/806856170 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6a000a3c40 msgr2=0x7f6a000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:10.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.419+0000 7f6a18c43700 1 --2- 172.21.15.186:0/806856170 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6a000a3c40 0x7f6a000a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6a04006010 tx=0x7f6a04003860 comp rx=0 tx=0).stop 2024-04-26T17:52:10.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.419+0000 7f6a18c43700 1 -- 172.21.15.186:0/806856170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a14103f20 msgr2=0x7f6a141a09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:10.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.419+0000 7f6a18c43700 1 --2- 172.21.15.186:0/806856170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a14103f20 0x7f6a141a09f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c00b970 tx=0x7f6a0c00bc80 comp rx=0 tx=0).stop 2024-04-26T17:52:10.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.419+0000 7f6a18c43700 1 -- 172.21.15.186:0/806856170 shutdown_connections 2024-04-26T17:52:10.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.419+0000 7f6a18c43700 1 --2- 172.21.15.186:0/806856170 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6a000a3c40 0x7f6a000a6100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:10.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.419+0000 7f6a18c43700 1 --2- 172.21.15.186:0/806856170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a14103f20 0x7f6a141a09f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:10.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.419+0000 7f6a18c43700 1 --2- 172.21.15.186:0/806856170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a14102380 0x7f6a1406bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:10.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.419+0000 7f6a18c43700 1 --2- 172.21.15.186:0/806856170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a14103580 0x7f6a1419e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:10.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.419+0000 7f6a18c43700 1 -- 172.21.15.186:0/806856170 >> 172.21.15.186:0/806856170 conn(0x7f6a140fd920 msgr2=0x7f6a14106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:10.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.419+0000 7f6a18c43700 1 -- 172.21.15.186:0/806856170 shutdown_connections 2024-04-26T17:52:10.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T17:52:10.419+0000 7f6a18c43700 1 -- 172.21.15.186:0/806856170 wait complete. 2024-04-26T17:52:11.057 DEBUG:teuthology.orchestra.run.smithi031:mds.a> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.a.service 2024-04-26T17:52:11.060 DEBUG:teuthology.orchestra.run.smithi031:mds.d> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.d.service 2024-04-26T17:52:11.104 DEBUG:teuthology.orchestra.run.smithi031:mds.g> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.g.service 2024-04-26T17:52:11.147 DEBUG:teuthology.orchestra.run.smithi031:mds.j> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.j.service 2024-04-26T17:52:11.190 DEBUG:teuthology.orchestra.run.smithi097:mds.b> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.b.service 2024-04-26T17:52:11.192 DEBUG:teuthology.orchestra.run.smithi097:mds.e> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.e.service 2024-04-26T17:52:11.235 DEBUG:teuthology.orchestra.run.smithi097:mds.h> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.h.service 2024-04-26T17:52:11.279 DEBUG:teuthology.orchestra.run.smithi097:mds.k> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.k.service 2024-04-26T17:52:11.322 DEBUG:teuthology.orchestra.run.smithi186:mds.c> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.c.service 2024-04-26T17:52:11.324 DEBUG:teuthology.orchestra.run.smithi186:mds.f> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.f.service 2024-04-26T17:52:11.368 DEBUG:teuthology.orchestra.run.smithi186:mds.i> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.i.service 2024-04-26T17:52:11.411 DEBUG:teuthology.orchestra.run.smithi186:mds.l> sudo journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.l.service 2024-04-26T17:52:11.454 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-26T17:52:11.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:52:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:11 smithi097 ceph-mon[32419]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:11 smithi097 ceph-mon[32419]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi097=b;smithi097=e;smithi097=h;smithi097=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:52:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:11 smithi097 ceph-mon[32419]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi097=b;smithi097=e;smithi097=h;smithi097=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-26T17:52:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:52:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:52:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:11.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:11.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:11 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:11.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:11 smithi097 ceph-mon[32419]: Deploying daemon mds.l on smithi186 2024-04-26T17:52:11.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:11 smithi186 ceph-mon[32797]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:11.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:11 smithi186 ceph-mon[32797]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi097=b;smithi097=e;smithi097=h;smithi097=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:52:11.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:11 smithi186 ceph-mon[32797]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi097=b;smithi097=e;smithi097=h;smithi097=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-26T17:52:11.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:11.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:52:11.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:11.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:52:11.660 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:11.660 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:11.660 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:11.660 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:11 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:11.660 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:11 smithi186 ceph-mon[32797]: Deploying daemon mds.l on smithi186 2024-04-26T17:52:11.702 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:52:11.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:11 smithi031 ceph-mon[29164]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:11.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:11 smithi031 ceph-mon[29164]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi097=b;smithi097=e;smithi097=h;smithi097=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:52:11.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:11 smithi031 ceph-mon[29164]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi097=b;smithi097=e;smithi097=h;smithi097=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-26T17:52:11.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:11.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:52:11.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:11.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:52:11.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:11.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:11.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:11.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:11 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:11.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:11 smithi031 ceph-mon[29164]: Deploying daemon mds.l on smithi186 2024-04-26T17:52:13.267 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 17:52:13 smithi186 systemd[1]: Starting Ceph mds.l for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:52:13.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.532+0000 7f612df16700 1 -- 172.21.15.31:0/2749667082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6128102ce0 msgr2=0x7f612810f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:13.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.532+0000 7f612df16700 1 --2- 172.21.15.31:0/2749667082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6128102ce0 0x7f612810f1d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6120009990 tx=0x7f6120009ca0 comp rx=0 tx=0).stop 2024-04-26T17:52:13.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.532+0000 7f612df16700 1 -- 172.21.15.31:0/2749667082 shutdown_connections 2024-04-26T17:52:13.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.532+0000 7f612df16700 1 --2- 172.21.15.31:0/2749667082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6128102ce0 0x7f612810f1d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:13.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.532+0000 7f612df16700 1 --2- 172.21.15.31:0/2749667082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6128108390 0x7f6128108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:13.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.532+0000 7f612df16700 1 --2- 172.21.15.31:0/2749667082 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6128102320 0x7f61281027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:13.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.532+0000 7f612df16700 1 -- 172.21.15.31:0/2749667082 >> 172.21.15.31:0/2749667082 conn(0x7f61280792d0 msgr2=0x7f61280796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:13.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.532+0000 7f612df16700 1 -- 172.21.15.31:0/2749667082 shutdown_connections 2024-04-26T17:52:13.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.532+0000 7f612df16700 1 -- 172.21.15.31:0/2749667082 wait complete. 2024-04-26T17:52:13.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.533+0000 7f612df16700 1 Processor -- start 2024-04-26T17:52:13.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.533+0000 7f612df16700 1 -- start start 2024-04-26T17:52:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.533+0000 7f612df16700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6128102320 0x7f612819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.533+0000 7f612df16700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6128102ce0 0x7f612819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.533+0000 7f612df16700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6128108390 0x7f61281987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.533+0000 7f612df16700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f61281124b0 con 0x7f6128108390 2024-04-26T17:52:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.533+0000 7f612df16700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6128112330 con 0x7f6128102320 2024-04-26T17:52:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.533+0000 7f612df16700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6128112630 con 0x7f6128102ce0 2024-04-26T17:52:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f61277fe700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6128102320 0x7f612819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f6127fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6128108390 0x7f61281987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f61277fe700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6128102320 0x7f612819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:39096/0 (socket says 172.21.15.31:39096) 2024-04-26T17:52:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f6127fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6128108390 0x7f61281987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59862/0 (socket says 172.21.15.31:59862) 2024-04-26T17:52:13.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f61277fe700 1 -- 172.21.15.31:0/240578966 learned_addr learned my addr 172.21.15.31:0/240578966 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:52:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f6126ffd700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6128102ce0 0x7f612819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f6127fff700 1 -- 172.21.15.31:0/240578966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6128102ce0 msgr2=0x7f612819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f6127fff700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6128102ce0 0x7f612819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f6127fff700 1 -- 172.21.15.31:0/240578966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6128102320 msgr2=0x7f612819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f6127fff700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6128102320 0x7f612819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f6127fff700 1 -- 172.21.15.31:0/240578966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6128198fb0 con 0x7f6128108390 2024-04-26T17:52:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f61277fe700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6128102320 0x7f612819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:52:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f6126ffd700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6128102ce0 0x7f612819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:52:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f6127fff700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6128108390 0x7f61281987d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f61200059d0 tx=0x7f6120009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f6124ff9700 1 -- 172.21.15.31:0/240578966 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120003680 con 0x7f6128108390 2024-04-26T17:52:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f6124ff9700 1 -- 172.21.15.31:0/240578966 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6120004ce0 con 0x7f6128108390 2024-04-26T17:52:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f612df16700 1 -- 172.21.15.31:0/240578966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6128199240 con 0x7f6128108390 2024-04-26T17:52:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.534+0000 7f6124ff9700 1 -- 172.21.15.31:0/240578966 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120003cd0 con 0x7f6128108390 2024-04-26T17:52:13.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.535+0000 7f612df16700 1 -- 172.21.15.31:0/240578966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61281a54b0 con 0x7f6128108390 2024-04-26T17:52:13.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.536+0000 7f612df16700 1 -- 172.21.15.31:0/240578966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6128103a40 con 0x7f6128108390 2024-04-26T17:52:13.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.538+0000 7f6124ff9700 1 -- 172.21.15.31:0/240578966 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6120029060 con 0x7f6128108390 2024-04-26T17:52:13.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.538+0000 7f6124ff9700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f61100a3de0 0x7f61100a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:13.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.538+0000 7f61277fe700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f61100a3de0 0x7f61100a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:13.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.538+0000 7f6124ff9700 1 -- 172.21.15.31:0/240578966 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f61200d0b80 con 0x7f6128108390 2024-04-26T17:52:13.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.539+0000 7f61277fe700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f61100a3de0 0x7f61100a62a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6118009550 tx=0x7f6118009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:13.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.542+0000 7f6124ff9700 1 -- 172.21.15.31:0/240578966 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f612005dff0 con 0x7f6128108390 2024-04-26T17:52:13.637 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:13 smithi186 ceph-mon[32797]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:13.638 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 17:52:13 smithi186 podman[62740]: 2024-04-26 17:52:13.283891713 +0000 UTC m=+0.018116965 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:52:13.638 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 17:52:13 smithi186 podman[62740]: 2024-04-26 17:52:13.394514433 +0000 UTC m=+0.128739731 container create 883a1e6b36df35e25c9e204b54338f4a8a38c06ddfed6640bfb04817c3d96418 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-04-26T17:52:13.638 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 17:52:13 smithi186 podman[62740]: 2024-04-26 17:52:13.638537676 +0000 UTC m=+0.372762913 container init 883a1e6b36df35e25c9e204b54338f4a8a38c06ddfed6640bfb04817c3d96418 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-l, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-26T17:52:13.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.806+0000 7f612df16700 1 -- 172.21.15.31:0/240578966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6128199d00 con 0x7f6128108390 2024-04-26T17:52:13.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:13 smithi031 ceph-mon[29164]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:13.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.807+0000 7f6124ff9700 1 -- 172.21.15.31:0/240578966 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f6120013070 con 0x7f6128108390 2024-04-26T17:52:13.807 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:52:13.807 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T17:52:13.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.809+0000 7f612df16700 1 -- 172.21.15.31:0/240578966 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f61100a3de0 msgr2=0x7f61100a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:13.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.809+0000 7f612df16700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f61100a3de0 0x7f61100a62a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6118009550 tx=0x7f6118009040 comp rx=0 tx=0).stop 2024-04-26T17:52:13.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.809+0000 7f612df16700 1 -- 172.21.15.31:0/240578966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6128108390 msgr2=0x7f61281987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:13.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.809+0000 7f612df16700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6128108390 0x7f61281987d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f61200059d0 tx=0x7f6120009580 comp rx=0 tx=0).stop 2024-04-26T17:52:13.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.809+0000 7f612df16700 1 -- 172.21.15.31:0/240578966 shutdown_connections 2024-04-26T17:52:13.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.809+0000 7f612df16700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f61100a3de0 0x7f61100a62a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:13.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.809+0000 7f612df16700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6128102320 0x7f612819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:13.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.809+0000 7f612df16700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6128108390 0x7f61281987d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:13.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.809+0000 7f612df16700 1 --2- 172.21.15.31:0/240578966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6128102ce0 0x7f612819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:13.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.809+0000 7f612df16700 1 -- 172.21.15.31:0/240578966 >> 172.21.15.31:0/240578966 conn(0x7f61280792d0 msgr2=0x7f6128100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:13.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.810+0000 7f612df16700 1 -- 172.21.15.31:0/240578966 shutdown_connections 2024-04-26T17:52:13.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:13.810+0000 7f612df16700 1 -- 172.21.15.31:0/240578966 wait complete. 2024-04-26T17:52:13.811 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 2 2024-04-26T17:52:13.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:13 smithi097 ceph-mon[32419]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:14.016 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 17:52:13 smithi186 podman[62740]: 2024-04-26 17:52:13.64135033 +0000 UTC m=+0.375575566 container start 883a1e6b36df35e25c9e204b54338f4a8a38c06ddfed6640bfb04817c3d96418 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T17:52:14.017 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 17:52:13 smithi186 ceph-mds[62759]: -- 172.21.15.186:0/943080514 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564232065a40 con 0x564232038800 2024-04-26T17:52:14.017 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 17:52:13 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-l[62755]: starting mds.l at 2024-04-26T17:52:14.017 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 17:52:13 smithi186 bash[62740]: 883a1e6b36df35e25c9e204b54338f4a8a38c06ddfed6640bfb04817c3d96418 2024-04-26T17:52:14.017 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 17:52:13 smithi186 systemd[1]: Started Ceph mds.l for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:52:14.771 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:14 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.186:6832/1597494912,v1:172.21.15.186:6833/1597494912] up:boot 2024-04-26T17:52:14.771 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:14 smithi097 ceph-mon[32419]: fsmap 1 up:standby 2024-04-26T17:52:14.772 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T17:52:14.772 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:14.772 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:14 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/240578966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:14.772 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:14.772 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:14.772 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:14.772 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:14.772 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:14 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:14.772 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:14 smithi097 ceph-mon[32419]: Deploying daemon mds.k on smithi097 2024-04-26T17:52:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:14 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.186:6832/1597494912,v1:172.21.15.186:6833/1597494912] up:boot 2024-04-26T17:52:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:14 smithi186 ceph-mon[32797]: fsmap 1 up:standby 2024-04-26T17:52:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T17:52:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:14 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/240578966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:14 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:15.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:14 smithi186 ceph-mon[32797]: Deploying daemon mds.k on smithi097 2024-04-26T17:52:15.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:14 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.186:6832/1597494912,v1:172.21.15.186:6833/1597494912] up:boot 2024-04-26T17:52:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:14 smithi031 ceph-mon[29164]: fsmap 1 up:standby 2024-04-26T17:52:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T17:52:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:14 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/240578966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:14 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:14 smithi031 ceph-mon[29164]: Deploying daemon mds.k on smithi097 2024-04-26T17:52:16.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:15 smithi097 ceph-mon[32419]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:16.134 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 17:52:16 smithi097 systemd[1]: Starting Ceph mds.k for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:52:16.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:15 smithi186 ceph-mon[32797]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:16.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:15 smithi031 ceph-mon[29164]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:16.512 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 17:52:16 smithi097 podman[62103]: 2024-04-26 17:52:16.146116517 +0000 UTC m=+0.017967608 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:52:16.513 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 17:52:16 smithi097 podman[62103]: 2024-04-26 17:52:16.268275061 +0000 UTC m=+0.140126197 container create 5f9a7c1d519c0a43ed2c1c1fc832f928bb46791ff41f4eeec4fcac62ce10346e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-k, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T17:52:16.513 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 17:52:16 smithi097 podman[62103]: 2024-04-26 17:52:16.487954644 +0000 UTC m=+0.359805734 container init 5f9a7c1d519c0a43ed2c1c1fc832f928bb46791ff41f4eeec4fcac62ce10346e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T17:52:16.513 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 17:52:16 smithi097 podman[62103]: 2024-04-26 17:52:16.490521456 +0000 UTC m=+0.362372555 container start 5f9a7c1d519c0a43ed2c1c1fc832f928bb46791ff41f4eeec4fcac62ce10346e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T17:52:16.513 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 17:52:16 smithi097 ceph-mds[62122]: -- 172.21.15.97:0/831112063 <== mon.2 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558cef131a40 con 0x558cef104800 2024-04-26T17:52:16.612 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:52:16.879 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:52:16.884 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 17:52:16 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-k[62118]: starting mds.k at 2024-04-26T17:52:16.884 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 17:52:16 smithi097 bash[62103]: 5f9a7c1d519c0a43ed2c1c1fc832f928bb46791ff41f4eeec4fcac62ce10346e 2024-04-26T17:52:16.884 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 17:52:16 smithi097 systemd[1]: Started Ceph mds.k for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:52:18.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:17 smithi186 ceph-mon[32797]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:18.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:18.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:18.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:18.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:18.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:18.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:18.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:17 smithi186 ceph-mon[32797]: Deploying daemon mds.j on smithi031 2024-04-26T17:52:18.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:17 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.97:6832/2883739522,v1:172.21.15.97:6833/2883739522] up:boot 2024-04-26T17:52:18.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:17 smithi186 ceph-mon[32797]: fsmap 2 up:standby 2024-04-26T17:52:18.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:17 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T17:52:18.050 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:17 smithi031 ceph-mon[29164]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:18.050 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:18.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:18.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:18.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:18.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:18.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:18.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:17 smithi031 ceph-mon[29164]: Deploying daemon mds.j on smithi031 2024-04-26T17:52:18.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:17 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.97:6832/2883739522,v1:172.21.15.97:6833/2883739522] up:boot 2024-04-26T17:52:18.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:17 smithi031 ceph-mon[29164]: fsmap 2 up:standby 2024-04-26T17:52:18.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:17 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T17:52:18.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:17 smithi097 ceph-mon[32419]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:18.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:18.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:18.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:18.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:18.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:18.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:18.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:17 smithi097 ceph-mon[32419]: Deploying daemon mds.j on smithi031 2024-04-26T17:52:18.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:17 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.97:6832/2883739522,v1:172.21.15.97:6833/2883739522] up:boot 2024-04-26T17:52:18.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:17 smithi097 ceph-mon[32419]: fsmap 2 up:standby 2024-04-26T17:52:18.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:17 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T17:52:18.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.706+0000 7f03fc430700 1 -- 172.21.15.31:0/158765180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03f41094a0 msgr2=0x7f03f4109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.706+0000 7f03fc430700 1 --2- 172.21.15.31:0/158765180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03f41094a0 0x7f03f4109880 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f03f0009990 tx=0x7f03f0009ca0 comp rx=0 tx=0).stop 2024-04-26T17:52:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.706+0000 7f03fc430700 1 -- 172.21.15.31:0/158765180 shutdown_connections 2024-04-26T17:52:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.706+0000 7f03fc430700 1 --2- 172.21.15.31:0/158765180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f406bad0 0x7f03f41102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.706+0000 7f03fc430700 1 --2- 172.21.15.31:0/158765180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03f41094a0 0x7f03f4109880 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.706+0000 7f03fc430700 1 --2- 172.21.15.31:0/158765180 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03f406b130 0x7f03f406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.706+0000 7f03fc430700 1 -- 172.21.15.31:0/158765180 >> 172.21.15.31:0/158765180 conn(0x7f03f40792d0 msgr2=0x7f03f40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.706+0000 7f03fc430700 1 -- 172.21.15.31:0/158765180 shutdown_connections 2024-04-26T17:52:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.706+0000 7f03fc430700 1 -- 172.21.15.31:0/158765180 wait complete. 2024-04-26T17:52:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.706+0000 7f03fc430700 1 Processor -- start 2024-04-26T17:52:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fc430700 1 -- start start 2024-04-26T17:52:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fc430700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03f406b130 0x7f03f419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fc430700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f406bad0 0x7f03f419fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fc430700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03f41094a0 0x7f03f4199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fc430700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4113610 con 0x7f03f41094a0 2024-04-26T17:52:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fc430700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4113490 con 0x7f03f406b130 2024-04-26T17:52:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fc430700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4113790 con 0x7f03f406bad0 2024-04-26T17:52:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fa9cd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03f41094a0 0x7f03f4199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fa1cc700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03f406b130 0x7f03f419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fa9cd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03f41094a0 0x7f03f4199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56524/0 (socket says 172.21.15.31:56524) 2024-04-26T17:52:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fa9cd700 1 -- 172.21.15.31:0/3073476935 learned_addr learned my addr 172.21.15.31:0/3073476935 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:52:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03f99cb700 1 --2- 172.21.15.31:0/3073476935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f406bad0 0x7f03f419fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fa9cd700 1 -- 172.21.15.31:0/3073476935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f406bad0 msgr2=0x7f03f419fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fa9cd700 1 --2- 172.21.15.31:0/3073476935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f406bad0 0x7f03f419fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fa9cd700 1 -- 172.21.15.31:0/3073476935 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03f406b130 msgr2=0x7f03f419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fa9cd700 1 --2- 172.21.15.31:0/3073476935 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03f406b130 0x7f03f419f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.707+0000 7f03fa9cd700 1 -- 172.21.15.31:0/3073476935 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f419a0d0 con 0x7f03f41094a0 2024-04-26T17:52:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.708+0000 7f03fa9cd700 1 --2- 172.21.15.31:0/3073476935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03f41094a0 0x7f03f4199970 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00b970 tx=0x7f03ec00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:18.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.708+0000 7f03eb7fe700 1 -- 172.21.15.31:0/3073476935 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec00d860 con 0x7f03f41094a0 2024-04-26T17:52:18.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.708+0000 7f03eb7fe700 1 -- 172.21.15.31:0/3073476935 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03ec00e430 con 0x7f03f41094a0 2024-04-26T17:52:18.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.708+0000 7f03fc430700 1 -- 172.21.15.31:0/3073476935 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f419a3c0 con 0x7f03f41094a0 2024-04-26T17:52:18.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.708+0000 7f03eb7fe700 1 -- 172.21.15.31:0/3073476935 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec017660 con 0x7f03f41094a0 2024-04-26T17:52:18.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.708+0000 7f03fc430700 1 -- 172.21.15.31:0/3073476935 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f41a6680 con 0x7f03f41094a0 2024-04-26T17:52:18.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.709+0000 7f03fc430700 1 -- 172.21.15.31:0/3073476935 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f406a110 con 0x7f03f41094a0 2024-04-26T17:52:18.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.710+0000 7f03eb7fe700 1 -- 172.21.15.31:0/3073476935 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f03ec00da00 con 0x7f03f41094a0 2024-04-26T17:52:18.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.710+0000 7f03eb7fe700 1 --2- 172.21.15.31:0/3073476935 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f03e00a3cc0 0x7f03e00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:18.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.711+0000 7f03fa1cc700 1 --2- 172.21.15.31:0/3073476935 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f03e00a3cc0 0x7f03e00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:18.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.711+0000 7f03eb7fe700 1 -- 172.21.15.31:0/3073476935 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f03ec0cfbe0 con 0x7f03f41094a0 2024-04-26T17:52:18.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.711+0000 7f03fa1cc700 1 --2- 172.21.15.31:0/3073476935 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f03e00a3cc0 0x7f03e00a6180 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f03f419aab0 tx=0x7f03f0005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:18.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.715+0000 7f03eb7fe700 1 -- 172.21.15.31:0/3073476935 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03ec013030 con 0x7f03f41094a0 2024-04-26T17:52:18.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.980+0000 7f03fc430700 1 -- 172.21.15.31:0/3073476935 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03f402e490 con 0x7f03f41094a0 2024-04-26T17:52:18.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.981+0000 7f03eb7fe700 1 -- 172.21.15.31:0/3073476935 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f03ec09c9c0 con 0x7f03f41094a0 2024-04-26T17:52:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:52:18.981 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T17:52:18.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.983+0000 7f03fc430700 1 -- 172.21.15.31:0/3073476935 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f03e00a3cc0 msgr2=0x7f03e00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:18.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.983+0000 7f03fc430700 1 --2- 172.21.15.31:0/3073476935 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f03e00a3cc0 0x7f03e00a6180 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f03f419aab0 tx=0x7f03f0005960 comp rx=0 tx=0).stop 2024-04-26T17:52:18.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.983+0000 7f03fc430700 1 -- 172.21.15.31:0/3073476935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03f41094a0 msgr2=0x7f03f4199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:18.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.983+0000 7f03fc430700 1 --2- 172.21.15.31:0/3073476935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03f41094a0 0x7f03f4199970 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00b970 tx=0x7f03ec00bc80 comp rx=0 tx=0).stop 2024-04-26T17:52:18.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.983+0000 7f03fc430700 1 -- 172.21.15.31:0/3073476935 shutdown_connections 2024-04-26T17:52:18.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.983+0000 7f03fc430700 1 --2- 172.21.15.31:0/3073476935 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f03e00a3cc0 0x7f03e00a6180 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:18.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.983+0000 7f03fc430700 1 --2- 172.21.15.31:0/3073476935 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03f406b130 0x7f03f419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:18.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.983+0000 7f03fc430700 1 --2- 172.21.15.31:0/3073476935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03f41094a0 0x7f03f4199970 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:18.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.983+0000 7f03fc430700 1 --2- 172.21.15.31:0/3073476935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f406bad0 0x7f03f419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:18.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.983+0000 7f03fc430700 1 -- 172.21.15.31:0/3073476935 >> 172.21.15.31:0/3073476935 conn(0x7f03f40792d0 msgr2=0x7f03f4100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:18.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.983+0000 7f03fc430700 1 -- 172.21.15.31:0/3073476935 shutdown_connections 2024-04-26T17:52:18.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:18.983+0000 7f03fc430700 1 -- 172.21.15.31:0/3073476935 wait complete. 2024-04-26T17:52:18.984 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 3 2024-04-26T17:52:20.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:19 smithi186 ceph-mon[32797]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:19 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3073476935' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:20.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:19 smithi031 ceph-mon[29164]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:20.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:19 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3073476935' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:20.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:19 smithi097 ceph-mon[32419]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:20.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:19 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3073476935' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:21.309 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 17:52:21 smithi031 systemd[1]: Starting Ceph mds.j for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:52:21.694 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:21 smithi031 ceph-mon[29164]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:21.695 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 17:52:21 smithi031 podman[67777]: 2024-04-26 17:52:21.343595733 +0000 UTC m=+0.019019806 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:52:21.695 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 17:52:21 smithi031 podman[67777]: 2024-04-26 17:52:21.468462045 +0000 UTC m=+0.143886131 container create e44088701a82347d470f592e4c1ba965d4c6e194ac131b787a28ac7d3a8cbb1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-j, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T17:52:21.695 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 17:52:21 smithi031 podman[67777]: 2024-04-26 17:52:21.696085332 +0000 UTC m=+0.371509376 container init e44088701a82347d470f592e4c1ba965d4c6e194ac131b787a28ac7d3a8cbb1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-04-26T17:52:22.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:21 smithi186 ceph-mon[32797]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:22.059 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 17:52:21 smithi031 podman[67777]: 2024-04-26 17:52:21.699155967 +0000 UTC m=+0.374580012 container start e44088701a82347d470f592e4c1ba965d4c6e194ac131b787a28ac7d3a8cbb1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T17:52:22.059 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 17:52:21 smithi031 ceph-mds[67796]: -- 172.21.15.31:0/847005514 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562d92de5a40 con 0x562d92db9000 2024-04-26T17:52:22.059 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 17:52:21 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-j[67792]: starting mds.j at 2024-04-26T17:52:22.059 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 17:52:21 smithi031 bash[67777]: e44088701a82347d470f592e4c1ba965d4c6e194ac131b787a28ac7d3a8cbb1d 2024-04-26T17:52:22.060 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 17:52:21 smithi031 systemd[1]: Started Ceph mds.j for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:52:22.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:21 smithi097 ceph-mon[32419]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:22.217 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:52:23.079 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:52:23.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:22 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:23.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:22 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:23.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:22 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:23.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:22 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:23.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:22 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:23.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:22 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:23.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:22 smithi097 ceph-mon[32419]: Deploying daemon mds.d on smithi031 2024-04-26T17:52:23.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:22 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.31:6834/3371658851,v1:172.21.15.31:6835/3371658851] up:boot 2024-04-26T17:52:23.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:22 smithi097 ceph-mon[32419]: fsmap 3 up:standby 2024-04-26T17:52:23.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:22 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T17:52:23.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:22 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:22 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:22 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:22 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:22 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:22 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:22 smithi186 ceph-mon[32797]: Deploying daemon mds.d on smithi031 2024-04-26T17:52:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:22 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.31:6834/3371658851,v1:172.21.15.31:6835/3371658851] up:boot 2024-04-26T17:52:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:22 smithi186 ceph-mon[32797]: fsmap 3 up:standby 2024-04-26T17:52:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:22 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T17:52:23.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:22 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:22 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:22 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:22 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:22 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:22 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:22 smithi031 ceph-mon[29164]: Deploying daemon mds.d on smithi031 2024-04-26T17:52:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:22 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.31:6834/3371658851,v1:172.21.15.31:6835/3371658851] up:boot 2024-04-26T17:52:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:22 smithi031 ceph-mon[29164]: fsmap 3 up:standby 2024-04-26T17:52:23.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:22 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T17:52:24.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:23 smithi097 ceph-mon[32419]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:24.159 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:23 smithi031 ceph-mon[29164]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:24.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:23 smithi186 ceph-mon[32797]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:24.537 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 17:52:24 smithi031 systemd[1]: Starting Ceph mds.d for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:52:25.439 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 17:52:25 smithi031 podman[68197]: 2024-04-26 17:52:25.161307577 +0000 UTC m=+0.250021883 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:52:25.439 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 17:52:25 smithi031 podman[68197]: 2024-04-26 17:52:25.326050106 +0000 UTC m=+0.414764397 container create d63a664ed7d0df63aa0905e4999a2cf33362ba55e5df3ae79c87d75b08918792 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-26T17:52:25.704 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 17:52:25 smithi031 podman[68197]: 2024-04-26 17:52:25.621482147 +0000 UTC m=+0.710196447 container init d63a664ed7d0df63aa0905e4999a2cf33362ba55e5df3ae79c87d75b08918792 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T17:52:25.704 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 17:52:25 smithi031 podman[68197]: 2024-04-26 17:52:25.624570934 +0000 UTC m=+0.713285233 container start d63a664ed7d0df63aa0905e4999a2cf33362ba55e5df3ae79c87d75b08918792 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T17:52:25.704 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 17:52:25 smithi031 ceph-mds[68243]: -- 172.21.15.31:0/1121094561 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559b3150ba40 con 0x559b314dec00 2024-04-26T17:52:25.704 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 17:52:25 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-d[68217]: starting mds.d at 2024-04-26T17:52:25.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.616+0000 7f36d3757700 1 -- 172.21.15.31:0/1494661607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36cc103e30 msgr2=0x7f36cc110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:25.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.616+0000 7f36d3757700 1 --2- 172.21.15.31:0/1494661607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36cc103e30 0x7f36cc110290 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f36c400b0a0 tx=0x7f36c400b3b0 comp rx=0 tx=0).stop 2024-04-26T17:52:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.616+0000 7f36d3757700 1 -- 172.21.15.31:0/1494661607 shutdown_connections 2024-04-26T17:52:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.616+0000 7f36d3757700 1 --2- 172.21.15.31:0/1494661607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36cc1094c0 0x7f36cc1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.616+0000 7f36d3757700 1 --2- 172.21.15.31:0/1494661607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36cc103e30 0x7f36cc110290 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.616+0000 7f36d3757700 1 --2- 172.21.15.31:0/1494661607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc103470 0x7f36cc1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.616+0000 7f36d3757700 1 -- 172.21.15.31:0/1494661607 >> 172.21.15.31:0/1494661607 conn(0x7f36cc07a580 msgr2=0x7f36cc07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.616+0000 7f36d3757700 1 -- 172.21.15.31:0/1494661607 shutdown_connections 2024-04-26T17:52:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.616+0000 7f36d3757700 1 -- 172.21.15.31:0/1494661607 wait complete. 2024-04-26T17:52:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.616+0000 7f36d3757700 1 Processor -- start 2024-04-26T17:52:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.616+0000 7f36d3757700 1 -- start start 2024-04-26T17:52:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d3757700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc103470 0x7f36cc19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d3757700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36cc103e30 0x7f36cc19fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d3757700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36cc1094c0 0x7f36cc199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d3757700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f36cc113690 con 0x7f36cc103e30 2024-04-26T17:52:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d3757700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f36cc113510 con 0x7f36cc1094c0 2024-04-26T17:52:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d3757700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f36cc113810 con 0x7f36cc103470 2024-04-26T17:52:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d0cf2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36cc103e30 0x7f36cc19fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d0cf2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36cc103e30 0x7f36cc19fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42656/0 (socket says 172.21.15.31:42656) 2024-04-26T17:52:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d0cf2700 1 -- 172.21.15.31:0/3904388531 learned_addr learned my addr 172.21.15.31:0/3904388531 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:52:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d1cf4700 1 --2- 172.21.15.31:0/3904388531 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36cc1094c0 0x7f36cc199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d14f3700 1 --2- 172.21.15.31:0/3904388531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc103470 0x7f36cc19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d0cf2700 1 -- 172.21.15.31:0/3904388531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc103470 msgr2=0x7f36cc19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d0cf2700 1 --2- 172.21.15.31:0/3904388531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc103470 0x7f36cc19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d0cf2700 1 -- 172.21.15.31:0/3904388531 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36cc1094c0 msgr2=0x7f36cc199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d0cf2700 1 --2- 172.21.15.31:0/3904388531 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36cc1094c0 0x7f36cc199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.617+0000 7f36d0cf2700 1 -- 172.21.15.31:0/3904388531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36cc19a0d0 con 0x7f36cc103e30 2024-04-26T17:52:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.618+0000 7f36d0cf2700 1 --2- 172.21.15.31:0/3904388531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36cc103e30 0x7f36cc19fcf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f36bc00b530 tx=0x7f36bc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.618+0000 7f36c27fc700 1 -- 172.21.15.31:0/3904388531 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc015020 con 0x7f36cc103e30 2024-04-26T17:52:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.618+0000 7f36c27fc700 1 -- 172.21.15.31:0/3904388531 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36bc011ce0 con 0x7f36cc103e30 2024-04-26T17:52:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.618+0000 7f36d3757700 1 -- 172.21.15.31:0/3904388531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36cc19a3c0 con 0x7f36cc103e30 2024-04-26T17:52:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.618+0000 7f36c27fc700 1 -- 172.21.15.31:0/3904388531 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc010dd0 con 0x7f36cc103e30 2024-04-26T17:52:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.618+0000 7f36d3757700 1 -- 172.21.15.31:0/3904388531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36cc1a6680 con 0x7f36cc103e30 2024-04-26T17:52:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.619+0000 7f36d3757700 1 -- 172.21.15.31:0/3904388531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36cc06a110 con 0x7f36cc103e30 2024-04-26T17:52:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.620+0000 7f36c27fc700 1 -- 172.21.15.31:0/3904388531 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f36bc018020 con 0x7f36cc103e30 2024-04-26T17:52:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.621+0000 7f36c27fc700 1 --2- 172.21.15.31:0/3904388531 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f36b80a3c40 0x7f36b80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.621+0000 7f36d14f3700 1 --2- 172.21.15.31:0/3904388531 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f36b80a3c40 0x7f36b80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.621+0000 7f36c27fc700 1 -- 172.21.15.31:0/3904388531 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f36bc0cfc70 con 0x7f36cc103e30 2024-04-26T17:52:25.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.621+0000 7f36d14f3700 1 --2- 172.21.15.31:0/3904388531 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f36b80a3c40 0x7f36b80a6100 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f36c8005950 tx=0x7f36c800a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:25.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.626+0000 7f36c27fc700 1 -- 172.21.15.31:0/3904388531 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36bc01f020 con 0x7f36cc103e30 2024-04-26T17:52:25.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.907+0000 7f36d3757700 1 -- 172.21.15.31:0/3904388531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36cc19aea0 con 0x7f36cc103e30 2024-04-26T17:52:25.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.907+0000 7f36c27fc700 1 -- 172.21.15.31:0/3904388531 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f36bc09ca50 con 0x7f36cc103e30 2024-04-26T17:52:25.907 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:52:25.907 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T17:52:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.909+0000 7f36d3757700 1 -- 172.21.15.31:0/3904388531 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f36b80a3c40 msgr2=0x7f36b80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.909+0000 7f36d3757700 1 --2- 172.21.15.31:0/3904388531 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f36b80a3c40 0x7f36b80a6100 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f36c8005950 tx=0x7f36c800a300 comp rx=0 tx=0).stop 2024-04-26T17:52:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.909+0000 7f36d3757700 1 -- 172.21.15.31:0/3904388531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36cc103e30 msgr2=0x7f36cc19fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.909+0000 7f36d3757700 1 --2- 172.21.15.31:0/3904388531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36cc103e30 0x7f36cc19fcf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f36bc00b530 tx=0x7f36bc00b840 comp rx=0 tx=0).stop 2024-04-26T17:52:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.909+0000 7f36d3757700 1 -- 172.21.15.31:0/3904388531 shutdown_connections 2024-04-26T17:52:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.910+0000 7f36d3757700 1 --2- 172.21.15.31:0/3904388531 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f36b80a3c40 0x7f36b80a6100 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.910+0000 7f36d3757700 1 --2- 172.21.15.31:0/3904388531 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36cc1094c0 0x7f36cc199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.910+0000 7f36d3757700 1 --2- 172.21.15.31:0/3904388531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36cc103470 0x7f36cc19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.910+0000 7f36d3757700 1 --2- 172.21.15.31:0/3904388531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36cc103e30 0x7f36cc19fcf0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.910+0000 7f36d3757700 1 -- 172.21.15.31:0/3904388531 >> 172.21.15.31:0/3904388531 conn(0x7f36cc07a580 msgr2=0x7f36cc102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.910+0000 7f36d3757700 1 -- 172.21.15.31:0/3904388531 shutdown_connections 2024-04-26T17:52:25.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:25.910+0000 7f36d3757700 1 -- 172.21.15.31:0/3904388531 wait complete. 2024-04-26T17:52:25.911 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 5 2024-04-26T17:52:26.060 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 17:52:25 smithi031 bash[68197]: d63a664ed7d0df63aa0905e4999a2cf33362ba55e5df3ae79c87d75b08918792 2024-04-26T17:52:26.060 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 17:52:25 smithi031 systemd[1]: Started Ceph mds.d for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:52:26.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:25 smithi031 ceph-mon[29164]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:26.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:25 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:26.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:25 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:26.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:25 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:26.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:25 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:26.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:25 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:26.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:25 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:26.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:25 smithi097 ceph-mon[32419]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:26.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:25 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:26.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:25 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:26.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:25 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:26.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:25 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:26.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:25 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:26.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:25 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:26.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:25 smithi186 ceph-mon[32797]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:26.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:25 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:26.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:25 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:26.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:25 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:26.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:25 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:26.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:25 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:26.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:25 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:27.110 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:26 smithi097 ceph-mon[32419]: Deploying daemon mds.e on smithi097 2024-04-26T17:52:27.110 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:26 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.31:6836/1425597980,v1:172.21.15.31:6837/1425597980] up:boot 2024-04-26T17:52:27.110 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:26 smithi097 ceph-mon[32419]: fsmap 4 up:standby 2024-04-26T17:52:27.110 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:26 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T17:52:27.111 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:26 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3904388531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:27.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:26 smithi186 ceph-mon[32797]: Deploying daemon mds.e on smithi097 2024-04-26T17:52:27.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:26 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.31:6836/1425597980,v1:172.21.15.31:6837/1425597980] up:boot 2024-04-26T17:52:27.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:26 smithi186 ceph-mon[32797]: fsmap 4 up:standby 2024-04-26T17:52:27.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:26 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T17:52:27.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:26 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3904388531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:27.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:26 smithi031 ceph-mon[29164]: Deploying daemon mds.e on smithi097 2024-04-26T17:52:27.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:26 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.31:6836/1425597980,v1:172.21.15.31:6837/1425597980] up:boot 2024-04-26T17:52:27.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:26 smithi031 ceph-mon[29164]: fsmap 4 up:standby 2024-04-26T17:52:27.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:26 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T17:52:27.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:26 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3904388531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:28.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:27 smithi097 ceph-mon[32419]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:28.134 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 17:52:27 smithi097 systemd[1]: Starting Ceph mds.e for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:52:28.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:27 smithi186 ceph-mon[32797]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:28.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:27 smithi031 ceph-mon[29164]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:28.417 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 17:52:28 smithi097 podman[62422]: 2024-04-26 17:52:28.055642922 +0000 UTC m=+0.018114664 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:52:28.417 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 17:52:28 smithi097 podman[62422]: 2024-04-26 17:52:28.173772967 +0000 UTC m=+0.136244702 container create c07ebece9e02cbfa6b7e5299f34c35da50d46260e95ebd9201dcb99d28309f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-26T17:52:28.417 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 17:52:28 smithi097 podman[62422]: 2024-04-26 17:52:28.392972492 +0000 UTC m=+0.355444225 container init c07ebece9e02cbfa6b7e5299f34c35da50d46260e95ebd9201dcb99d28309f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T17:52:28.417 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 17:52:28 smithi097 podman[62422]: 2024-04-26 17:52:28.395630226 +0000 UTC m=+0.358101962 container start c07ebece9e02cbfa6b7e5299f34c35da50d46260e95ebd9201dcb99d28309f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-26T17:52:28.418 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 17:52:28 smithi097 ceph-mds[62442]: -- 172.21.15.97:0/2545473632 <== mon.2 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555fd4313a40 con 0x555fd42e6800 2024-04-26T17:52:28.571 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:52:28.831 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:52:28.884 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 17:52:28 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-e[62438]: starting mds.e at 2024-04-26T17:52:28.884 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 17:52:28 smithi097 bash[62422]: c07ebece9e02cbfa6b7e5299f34c35da50d46260e95ebd9201dcb99d28309f1a 2024-04-26T17:52:28.884 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 17:52:28 smithi097 systemd[1]: Started Ceph mds.e for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:52:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:29 smithi097 ceph-mon[32419]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:29 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:29 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:29 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:29 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:29 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:29 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:29 smithi097 ceph-mon[32419]: Deploying daemon mds.f on smithi186 2024-04-26T17:52:29.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:29 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.97:6834/1721162755,v1:172.21.15.97:6835/1721162755] up:boot 2024-04-26T17:52:29.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:29 smithi097 ceph-mon[32419]: fsmap 5 up:standby 2024-04-26T17:52:29.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:29 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T17:52:29.741 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:29 smithi186 ceph-mon[32797]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:29.742 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:29 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:29.742 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:29 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:29.742 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:29 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:29.742 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:29 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:29.742 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:29 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:29.742 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:29 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:29.742 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:29 smithi186 ceph-mon[32797]: Deploying daemon mds.f on smithi186 2024-04-26T17:52:29.742 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:29 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.97:6834/1721162755,v1:172.21.15.97:6835/1721162755] up:boot 2024-04-26T17:52:29.742 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:29 smithi186 ceph-mon[32797]: fsmap 5 up:standby 2024-04-26T17:52:29.742 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:29 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T17:52:29.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:29 smithi031 ceph-mon[29164]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:29 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:29 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:29 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:29 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:29 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:29 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:29 smithi031 ceph-mon[29164]: Deploying daemon mds.f on smithi186 2024-04-26T17:52:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:29 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.97:6834/1721162755,v1:172.21.15.97:6835/1721162755] up:boot 2024-04-26T17:52:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:29 smithi031 ceph-mon[29164]: fsmap 5 up:standby 2024-04-26T17:52:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:29 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T17:52:30.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.516+0000 7f8d2f119700 1 -- 172.21.15.31:0/2278564211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d281034a0 msgr2=0x7f8d28103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:30.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.516+0000 7f8d2f119700 1 --2- 172.21.15.31:0/2278564211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d281034a0 0x7f8d28103920 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009990 tx=0x7f8d14009ca0 comp rx=0 tx=0).stop 2024-04-26T17:52:30.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.516+0000 7f8d2f119700 1 -- 172.21.15.31:0/2278564211 shutdown_connections 2024-04-26T17:52:30.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.516+0000 7f8d2f119700 1 --2- 172.21.15.31:0/2278564211 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d28103e60 0x7f8d281102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.516+0000 7f8d2f119700 1 --2- 172.21.15.31:0/2278564211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d281094e0 0x7f8d281098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.516+0000 7f8d2f119700 1 --2- 172.21.15.31:0/2278564211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d281034a0 0x7f8d28103920 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.516+0000 7f8d2f119700 1 -- 172.21.15.31:0/2278564211 >> 172.21.15.31:0/2278564211 conn(0x7f8d2807a590 msgr2=0x7f8d2807a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.516+0000 7f8d2f119700 1 -- 172.21.15.31:0/2278564211 shutdown_connections 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.516+0000 7f8d2f119700 1 -- 172.21.15.31:0/2278564211 wait complete. 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2f119700 1 Processor -- start 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2f119700 1 -- start start 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2f119700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d281034a0 0x7f8d2819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2f119700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d28103e60 0x7f8d2819fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2f119700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d281094e0 0x7f8d28199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2f119700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8d28113660 con 0x7f8d281094e0 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2f119700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f8d281134e0 con 0x7f8d281034a0 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2f119700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8d281137e0 con 0x7f8d28103e60 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2d6b6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d281094e0 0x7f8d28199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2d6b6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d281094e0 0x7f8d28199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42700/0 (socket says 172.21.15.31:42700) 2024-04-26T17:52:30.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2d6b6700 1 -- 172.21.15.31:0/3281163272 learned_addr learned my addr 172.21.15.31:0/3281163272 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:52:30.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d27fff700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d28103e60 0x7f8d2819fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:30.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2ceb5700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d281034a0 0x7f8d2819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:30.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2d6b6700 1 -- 172.21.15.31:0/3281163272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d28103e60 msgr2=0x7f8d2819fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:30.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2d6b6700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d28103e60 0x7f8d2819fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:30.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2d6b6700 1 -- 172.21.15.31:0/3281163272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d281034a0 msgr2=0x7f8d2819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:30.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.517+0000 7f8d2d6b6700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d281034a0 0x7f8d2819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:30.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.518+0000 7f8d2d6b6700 1 -- 172.21.15.31:0/3281163272 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2819a080 con 0x7f8d281094e0 2024-04-26T17:52:30.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.518+0000 7f8d27fff700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d28103e60 0x7f8d2819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:52:30.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.518+0000 7f8d2ceb5700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d281034a0 0x7f8d2819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:52:30.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.518+0000 7f8d2d6b6700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d281094e0 0x7f8d28199920 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8d20007d70 tx=0x7f8d2000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:30.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.518+0000 7f8d25ffb700 1 -- 172.21.15.31:0/3281163272 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20016020 con 0x7f8d281094e0 2024-04-26T17:52:30.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.518+0000 7f8d25ffb700 1 -- 172.21.15.31:0/3281163272 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d2000ad40 con 0x7f8d281094e0 2024-04-26T17:52:30.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.518+0000 7f8d2f119700 1 -- 172.21.15.31:0/3281163272 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d2819a370 con 0x7f8d281094e0 2024-04-26T17:52:30.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.518+0000 7f8d25ffb700 1 -- 172.21.15.31:0/3281163272 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d200115e0 con 0x7f8d281094e0 2024-04-26T17:52:30.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.518+0000 7f8d2f119700 1 -- 172.21.15.31:0/3281163272 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d281a6630 con 0x7f8d281094e0 2024-04-26T17:52:30.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.519+0000 7f8d2f119700 1 -- 172.21.15.31:0/3281163272 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d2806a110 con 0x7f8d281094e0 2024-04-26T17:52:30.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.520+0000 7f8d25ffb700 1 -- 172.21.15.31:0/3281163272 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8d20019020 con 0x7f8d281094e0 2024-04-26T17:52:30.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.521+0000 7f8d25ffb700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8d180a3c40 0x7f8d180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:30.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.521+0000 7f8d2ceb5700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8d180a3c40 0x7f8d180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:30.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.521+0000 7f8d25ffb700 1 -- 172.21.15.31:0/3281163272 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8d200cffb0 con 0x7f8d281094e0 2024-04-26T17:52:30.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.521+0000 7f8d2ceb5700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8d180a3c40 0x7f8d180a6100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8d2819aa60 tx=0x7f8d1c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:30.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.526+0000 7f8d25ffb700 1 -- 172.21.15.31:0/3281163272 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d20020020 con 0x7f8d281094e0 2024-04-26T17:52:30.715 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 17:52:30 smithi186 systemd[1]: Starting Ceph mds.f for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:52:30.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.781+0000 7f8d2f119700 1 -- 172.21.15.31:0/3281163272 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d2802e490 con 0x7f8d281094e0 2024-04-26T17:52:30.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.782+0000 7f8d25ffb700 1 -- 172.21.15.31:0/3281163272 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7f8d2009ce30 con 0x7f8d281094e0 2024-04-26T17:52:30.785 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:52:30.785 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-26T17:52:30.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.784+0000 7f8d2f119700 1 -- 172.21.15.31:0/3281163272 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8d180a3c40 msgr2=0x7f8d180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.784+0000 7f8d2f119700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8d180a3c40 0x7f8d180a6100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8d2819aa60 tx=0x7f8d1c008040 comp rx=0 tx=0).stop 2024-04-26T17:52:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.784+0000 7f8d2f119700 1 -- 172.21.15.31:0/3281163272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d281094e0 msgr2=0x7f8d28199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.784+0000 7f8d2f119700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d281094e0 0x7f8d28199920 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8d20007d70 tx=0x7f8d2000a340 comp rx=0 tx=0).stop 2024-04-26T17:52:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.784+0000 7f8d2f119700 1 -- 172.21.15.31:0/3281163272 shutdown_connections 2024-04-26T17:52:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.784+0000 7f8d2f119700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8d180a3c40 0x7f8d180a6100 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.784+0000 7f8d2f119700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d281034a0 0x7f8d2819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.784+0000 7f8d2f119700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d281094e0 0x7f8d28199920 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.784+0000 7f8d2f119700 1 --2- 172.21.15.31:0/3281163272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d28103e60 0x7f8d2819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.784+0000 7f8d2f119700 1 -- 172.21.15.31:0/3281163272 >> 172.21.15.31:0/3281163272 conn(0x7f8d2807a590 msgr2=0x7f8d28102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.784+0000 7f8d2f119700 1 -- 172.21.15.31:0/3281163272 shutdown_connections 2024-04-26T17:52:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:30.784+0000 7f8d2f119700 1 -- 172.21.15.31:0/3281163272 wait complete. 2024-04-26T17:52:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 6 2024-04-26T17:52:31.017 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 17:52:30 smithi186 podman[63052]: 2024-04-26 17:52:30.772548232 +0000 UTC m=+0.018182234 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:52:31.017 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 17:52:30 smithi186 podman[63052]: 2024-04-26 17:52:30.889596835 +0000 UTC m=+0.135230827 container create ad71e0cc5520dda922246035a6cfc17ef7e46020e7ad26856615543de5a3ee38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-26T17:52:31.517 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 17:52:31 smithi186 podman[63052]: 2024-04-26 17:52:31.142483416 +0000 UTC m=+0.388117410 container init ad71e0cc5520dda922246035a6cfc17ef7e46020e7ad26856615543de5a3ee38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-26T17:52:31.517 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 17:52:31 smithi186 podman[63052]: 2024-04-26 17:52:31.144968852 +0000 UTC m=+0.390602845 container start ad71e0cc5520dda922246035a6cfc17ef7e46020e7ad26856615543de5a3ee38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T17:52:31.517 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 17:52:31 smithi186 ceph-mds[63071]: -- 172.21.15.186:0/4199550938 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559aca735a40 con 0x559aca709000 2024-04-26T17:52:31.517 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 17:52:31 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-f[63067]: starting mds.f at 2024-04-26T17:52:31.517 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 17:52:31 smithi186 bash[63052]: ad71e0cc5520dda922246035a6cfc17ef7e46020e7ad26856615543de5a3ee38 2024-04-26T17:52:31.517 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 17:52:31 smithi186 systemd[1]: Started Ceph mds.f for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:52:31.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:31 smithi031 ceph-mon[29164]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:31.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:31 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3281163272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:31.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:31 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:31.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:31 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:31.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:31 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:31.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:31 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:31.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:31 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:31.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:31 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:31 smithi097 ceph-mon[32419]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:31 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3281163272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:31 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:31 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:31 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:31 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:31 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:31 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:31.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:31 smithi186 ceph-mon[32797]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:31.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:31 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3281163272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:31.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:31 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:31.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:31 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:31.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:31 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:31.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:31 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:31.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:31 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:31.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:31 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:32.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:32 smithi186 ceph-mon[32797]: Deploying daemon mds.c on smithi186 2024-04-26T17:52:32.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:32 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.186:6834/580433615,v1:172.21.15.186:6835/580433615] up:boot 2024-04-26T17:52:32.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:32 smithi186 ceph-mon[32797]: fsmap 6 up:standby 2024-04-26T17:52:32.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:32 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T17:52:32.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:32 smithi031 ceph-mon[29164]: Deploying daemon mds.c on smithi186 2024-04-26T17:52:32.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:32 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.186:6834/580433615,v1:172.21.15.186:6835/580433615] up:boot 2024-04-26T17:52:32.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:32 smithi031 ceph-mon[29164]: fsmap 6 up:standby 2024-04-26T17:52:32.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:32 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T17:52:32.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:32 smithi097 ceph-mon[32419]: Deploying daemon mds.c on smithi186 2024-04-26T17:52:32.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:32 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.186:6834/580433615,v1:172.21.15.186:6835/580433615] up:boot 2024-04-26T17:52:32.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:32 smithi097 ceph-mon[32419]: fsmap 6 up:standby 2024-04-26T17:52:32.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:32 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T17:52:33.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:52:33.517 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 17:52:33 smithi186 systemd[1]: Starting Ceph mds.c for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:52:33.750 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:52:33.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:33 smithi031 ceph-mon[29164]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:33.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:33 smithi097 ceph-mon[32419]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:33.905 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:33 smithi186 ceph-mon[32797]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:33.905 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 17:52:33 smithi186 podman[63362]: 2024-04-26 17:52:33.636540763 +0000 UTC m=+0.110659914 container create 84410eca4b7856845d58a4b2cfd7e948420e9eb959b2db9630a3d46d35068066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-26T17:52:33.905 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 17:52:33 smithi186 podman[63362]: 2024-04-26 17:52:33.543967249 +0000 UTC m=+0.018086428 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:52:33.905 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 17:52:33 smithi186 podman[63362]: 2024-04-26 17:52:33.906448516 +0000 UTC m=+0.380567685 container init 84410eca4b7856845d58a4b2cfd7e948420e9eb959b2db9630a3d46d35068066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-26T17:52:34.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 17:52:33 smithi186 podman[63362]: 2024-04-26 17:52:33.909173269 +0000 UTC m=+0.383292420 container start 84410eca4b7856845d58a4b2cfd7e948420e9eb959b2db9630a3d46d35068066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T17:52:34.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 17:52:33 smithi186 ceph-mds[63382]: -- 172.21.15.186:0/2156696281 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557e6fc4fa40 con 0x557e6fc23000 2024-04-26T17:52:34.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 17:52:33 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: starting mds.c at 2024-04-26T17:52:34.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 17:52:34 smithi186 bash[63362]: 84410eca4b7856845d58a4b2cfd7e948420e9eb959b2db9630a3d46d35068066 2024-04-26T17:52:34.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 17:52:34 smithi186 systemd[1]: Started Ceph mds.c for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:52:35.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:35.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:35.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:35.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:35.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:35.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:35.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:35 smithi031 ceph-mon[29164]: Deploying daemon mds.h on smithi097 2024-04-26T17:52:35.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:35 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:boot 2024-04-26T17:52:35.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:35 smithi031 ceph-mon[29164]: fsmap 7 up:standby 2024-04-26T17:52:35.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:35 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T17:52:35.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:35.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:35.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:35.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:35 smithi097 ceph-mon[32419]: Deploying daemon mds.h on smithi097 2024-04-26T17:52:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:35 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:boot 2024-04-26T17:52:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:35 smithi097 ceph-mon[32419]: fsmap 7 up:standby 2024-04-26T17:52:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:35 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T17:52:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:35 smithi186 ceph-mon[32797]: Deploying daemon mds.h on smithi097 2024-04-26T17:52:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:35 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:boot 2024-04-26T17:52:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:35 smithi186 ceph-mon[32797]: fsmap 7 up:standby 2024-04-26T17:52:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:35 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T17:52:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.567+0000 7fac63106700 1 -- 172.21.15.31:0/3246497239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c102320 msgr2=0x7fac5c1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.567+0000 7fac63106700 1 --2- 172.21.15.31:0/3246497239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c102320 0x7fac5c1027a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fac44009990 tx=0x7fac44009ca0 comp rx=0 tx=0).stop 2024-04-26T17:52:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.567+0000 7fac63106700 1 -- 172.21.15.31:0/3246497239 shutdown_connections 2024-04-26T17:52:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.567+0000 7fac63106700 1 --2- 172.21.15.31:0/3246497239 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c108390 0x7fac5c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.567+0000 7fac63106700 1 --2- 172.21.15.31:0/3246497239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5c102ce0 0x7fac5c10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.567+0000 7fac63106700 1 --2- 172.21.15.31:0/3246497239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c102320 0x7fac5c1027a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.567+0000 7fac63106700 1 -- 172.21.15.31:0/3246497239 >> 172.21.15.31:0/3246497239 conn(0x7fac5c0792d0 msgr2=0x7fac5c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.567+0000 7fac63106700 1 -- 172.21.15.31:0/3246497239 shutdown_connections 2024-04-26T17:52:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.567+0000 7fac63106700 1 -- 172.21.15.31:0/3246497239 wait complete. 2024-04-26T17:52:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac63106700 1 Processor -- start 2024-04-26T17:52:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac63106700 1 -- start start 2024-04-26T17:52:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac63106700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5c102320 0x7fac5c19a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac63106700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c102ce0 0x7fac5c19aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac63106700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c108390 0x7fac5c1a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac63106700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1124e0 con 0x7fac5c108390 2024-04-26T17:52:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac63106700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c112360 con 0x7fac5c102ce0 2024-04-26T17:52:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac63106700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c112660 con 0x7fac5c102320 2024-04-26T17:52:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac616a3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c108390 0x7fac5c1a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac5bfff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c102ce0 0x7fac5c19aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac5bfff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c102ce0 0x7fac5c19aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:47372/0 (socket says 172.21.15.31:47372) 2024-04-26T17:52:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac616a3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c108390 0x7fac5c1a5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52056/0 (socket says 172.21.15.31:52056) 2024-04-26T17:52:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac5bfff700 1 -- 172.21.15.31:0/504740734 learned_addr learned my addr 172.21.15.31:0/504740734 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:52:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac60ea2700 1 --2- 172.21.15.31:0/504740734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5c102320 0x7fac5c19a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac5bfff700 1 -- 172.21.15.31:0/504740734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5c102320 msgr2=0x7fac5c19a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac5bfff700 1 --2- 172.21.15.31:0/504740734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5c102320 0x7fac5c19a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac5bfff700 1 -- 172.21.15.31:0/504740734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c108390 msgr2=0x7fac5c1a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac5bfff700 1 --2- 172.21.15.31:0/504740734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c108390 0x7fac5c1a5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac5bfff700 1 -- 172.21.15.31:0/504740734 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5c1a5f60 con 0x7fac5c102ce0 2024-04-26T17:52:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.568+0000 7fac616a3700 1 --2- 172.21.15.31:0/504740734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c108390 0x7fac5c1a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:52:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.569+0000 7fac5bfff700 1 --2- 172.21.15.31:0/504740734 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c102ce0 0x7fac5c19aee0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fac44000c00 tx=0x7fac44009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.569+0000 7fac59ffb700 1 -- 172.21.15.31:0/504740734 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44027070 con 0x7fac5c102ce0 2024-04-26T17:52:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.569+0000 7fac59ffb700 1 -- 172.21.15.31:0/504740734 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac44022470 con 0x7fac5c102ce0 2024-04-26T17:52:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.569+0000 7fac63106700 1 -- 172.21.15.31:0/504740734 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5c1a6130 con 0x7fac5c102ce0 2024-04-26T17:52:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.569+0000 7fac59ffb700 1 -- 172.21.15.31:0/504740734 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44017650 con 0x7fac5c102ce0 2024-04-26T17:52:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.569+0000 7fac63106700 1 -- 172.21.15.31:0/504740734 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5c1a6520 con 0x7fac5c102ce0 2024-04-26T17:52:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.570+0000 7fac63106700 1 -- 172.21.15.31:0/504740734 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac5c06a110 con 0x7fac5c102ce0 2024-04-26T17:52:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.571+0000 7fac59ffb700 1 -- 172.21.15.31:0/504740734 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fac4402a070 con 0x7fac5c102ce0 2024-04-26T17:52:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.572+0000 7fac59ffb700 1 --2- 172.21.15.31:0/504740734 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac480a3c40 0x7fac480a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.572+0000 7fac60ea2700 1 --2- 172.21.15.31:0/504740734 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac480a3c40 0x7fac480a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.572+0000 7fac59ffb700 1 -- 172.21.15.31:0/504740734 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fac440d0f30 con 0x7fac5c102ce0 2024-04-26T17:52:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.572+0000 7fac60ea2700 1 --2- 172.21.15.31:0/504740734 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac480a3c40 0x7fac480a6100 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00f540 tx=0x7fac4c00b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.575+0000 7fac59ffb700 1 -- 172.21.15.31:0/504740734 <== mon.2 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac44013070 con 0x7fac5c102ce0 2024-04-26T17:52:35.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.832+0000 7fac63106700 1 -- 172.21.15.31:0/504740734 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac5c02e650 con 0x7fac5c102ce0 2024-04-26T17:52:35.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.833+0000 7fac59ffb700 1 -- 172.21.15.31:0/504740734 <== mon.2 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7fac4409dda0 con 0x7fac5c102ce0 2024-04-26T17:52:35.833 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:52:35.833 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24439,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-26T17:52:35.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.835+0000 7fac63106700 1 -- 172.21.15.31:0/504740734 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac480a3c40 msgr2=0x7fac480a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:35.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.835+0000 7fac63106700 1 --2- 172.21.15.31:0/504740734 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac480a3c40 0x7fac480a6100 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00f540 tx=0x7fac4c00b040 comp rx=0 tx=0).stop 2024-04-26T17:52:35.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.835+0000 7fac63106700 1 -- 172.21.15.31:0/504740734 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c102ce0 msgr2=0x7fac5c19aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:35.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.835+0000 7fac63106700 1 --2- 172.21.15.31:0/504740734 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c102ce0 0x7fac5c19aee0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fac44000c00 tx=0x7fac44009580 comp rx=0 tx=0).stop 2024-04-26T17:52:35.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.835+0000 7fac63106700 1 -- 172.21.15.31:0/504740734 shutdown_connections 2024-04-26T17:52:35.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.835+0000 7fac63106700 1 --2- 172.21.15.31:0/504740734 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac480a3c40 0x7fac480a6100 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:35.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.835+0000 7fac63106700 1 --2- 172.21.15.31:0/504740734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c108390 0x7fac5c1a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:35.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.835+0000 7fac63106700 1 --2- 172.21.15.31:0/504740734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5c102320 0x7fac5c19a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:35.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.835+0000 7fac63106700 1 --2- 172.21.15.31:0/504740734 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c102ce0 0x7fac5c19aee0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:35.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.835+0000 7fac63106700 1 -- 172.21.15.31:0/504740734 >> 172.21.15.31:0/504740734 conn(0x7fac5c0792d0 msgr2=0x7fac5c100ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:35.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.835+0000 7fac63106700 1 -- 172.21.15.31:0/504740734 shutdown_connections 2024-04-26T17:52:35.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:35.835+0000 7fac63106700 1 -- 172.21.15.31:0/504740734 wait complete. 2024-04-26T17:52:35.836 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2024-04-26T17:52:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:36 smithi097 ceph-mon[32419]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:36 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/504740734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:36.384 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 17:52:36 smithi097 systemd[1]: Starting Ceph mds.h for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:52:36.500 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:36 smithi031 ceph-mon[29164]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:36.500 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:36 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/504740734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:36.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:36 smithi186 ceph-mon[32797]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:36.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:36 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/504740734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:36.806 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 17:52:36 smithi097 podman[62741]: 2024-04-26 17:52:36.425086408 +0000 UTC m=+0.017768877 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:52:36.806 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 17:52:36 smithi097 podman[62741]: 2024-04-26 17:52:36.554713494 +0000 UTC m=+0.147395949 container create 46498fc53496f589a6479c9e238aa338020e3a3b495c53607979146193715971 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-26T17:52:36.806 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 17:52:36 smithi097 podman[62741]: 2024-04-26 17:52:36.807636083 +0000 UTC m=+0.400318541 container init 46498fc53496f589a6479c9e238aa338020e3a3b495c53607979146193715971 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-26T17:52:37.134 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 17:52:36 smithi097 podman[62741]: 2024-04-26 17:52:36.810209551 +0000 UTC m=+0.402892013 container start 46498fc53496f589a6479c9e238aa338020e3a3b495c53607979146193715971 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-26T17:52:37.134 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 17:52:36 smithi097 ceph-mds[62760]: -- 172.21.15.97:0/1337797508 <== mon.2 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556a0d86da40 con 0x556a0d840c00 2024-04-26T17:52:37.134 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 17:52:36 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-h[62756]: starting mds.h at 2024-04-26T17:52:37.134 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 17:52:36 smithi097 bash[62741]: 46498fc53496f589a6479c9e238aa338020e3a3b495c53607979146193715971 2024-04-26T17:52:37.134 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 17:52:36 smithi097 systemd[1]: Started Ceph mds.h for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:52:38.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:37 smithi031 ceph-mon[29164]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:38.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:37 smithi031 ceph-mon[29164]: Deploying daemon mds.i on smithi186 2024-04-26T17:52:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:37 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] up:boot 2024-04-26T17:52:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:37 smithi031 ceph-mon[29164]: fsmap 8 up:standby 2024-04-26T17:52:38.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:37 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T17:52:38.117 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:37 smithi186 ceph-mon[32797]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:38.117 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:38.117 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:38.117 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:38.117 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:38.117 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:38.118 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:38.118 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:37 smithi186 ceph-mon[32797]: Deploying daemon mds.i on smithi186 2024-04-26T17:52:38.118 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:37 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] up:boot 2024-04-26T17:52:38.118 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:37 smithi186 ceph-mon[32797]: fsmap 8 up:standby 2024-04-26T17:52:38.118 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:37 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T17:52:38.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:37 smithi097 ceph-mon[32419]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:38.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:38.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:38.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:38.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:38.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:38.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:38.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:37 smithi097 ceph-mon[32419]: Deploying daemon mds.i on smithi186 2024-04-26T17:52:38.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:37 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] up:boot 2024-04-26T17:52:38.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:37 smithi097 ceph-mon[32419]: fsmap 8 up:standby 2024-04-26T17:52:38.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:37 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T17:52:38.536 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:52:38.801 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:52:39.517 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 17:52:39 smithi186 systemd[1]: Starting Ceph mds.i for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:52:39.877 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 17:52:39 smithi186 podman[63671]: 2024-04-26 17:52:39.51577311 +0000 UTC m=+0.017924515 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:52:39.877 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 17:52:39 smithi186 podman[63671]: 2024-04-26 17:52:39.625238167 +0000 UTC m=+0.127389572 container create 44a6244684343aaf6ee6b1bfd5adafe29b7d40cbea2e642d07da70f94d39d7ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-26T17:52:39.877 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 17:52:39 smithi186 podman[63671]: 2024-04-26 17:52:39.877845994 +0000 UTC m=+0.379997398 container init 44a6244684343aaf6ee6b1bfd5adafe29b7d40cbea2e642d07da70f94d39d7ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-i, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-26T17:52:40.174 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:39 smithi186 ceph-mon[32797]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:40.174 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 17:52:39 smithi186 podman[63671]: 2024-04-26 17:52:39.880682794 +0000 UTC m=+0.382834202 container start 44a6244684343aaf6ee6b1bfd5adafe29b7d40cbea2e642d07da70f94d39d7ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-i, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-26T17:52:40.174 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 17:52:39 smithi186 ceph-mds[63690]: -- 172.21.15.186:0/3159704686 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5558fe3cda40 con 0x5558fe3a0c00 2024-04-26T17:52:40.174 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 17:52:39 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-i[63686]: starting mds.i at 2024-04-26T17:52:40.174 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 17:52:39 smithi186 bash[63671]: 44a6244684343aaf6ee6b1bfd5adafe29b7d40cbea2e642d07da70f94d39d7ec 2024-04-26T17:52:40.175 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 17:52:39 smithi186 systemd[1]: Started Ceph mds.i for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:52:40.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:39 smithi031 ceph-mon[29164]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:40.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:39 smithi097 ceph-mon[32419]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:40.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.534+0000 7f08168e3700 1 -- 172.21.15.31:0/357438190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f081006b130 msgr2=0x7f081006b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:40.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.534+0000 7f08168e3700 1 --2- 172.21.15.31:0/357438190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f081006b130 0x7f081006b590 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0804009990 tx=0x7f0804009ca0 comp rx=0 tx=0).stop 2024-04-26T17:52:40.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.534+0000 7f08168e3700 1 -- 172.21.15.31:0/357438190 shutdown_connections 2024-04-26T17:52:40.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.534+0000 7f08168e3700 1 --2- 172.21.15.31:0/357438190 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f081006bad0 0x7f08101102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:40.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.534+0000 7f08168e3700 1 --2- 172.21.15.31:0/357438190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08101094a0 0x7f0810109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:40.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.534+0000 7f08168e3700 1 --2- 172.21.15.31:0/357438190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f081006b130 0x7f081006b590 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:40.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.534+0000 7f08168e3700 1 -- 172.21.15.31:0/357438190 >> 172.21.15.31:0/357438190 conn(0x7f08100792d0 msgr2=0x7f08100796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:40.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.534+0000 7f08168e3700 1 -- 172.21.15.31:0/357438190 shutdown_connections 2024-04-26T17:52:40.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.534+0000 7f08168e3700 1 -- 172.21.15.31:0/357438190 wait complete. 2024-04-26T17:52:40.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.535+0000 7f08168e3700 1 Processor -- start 2024-04-26T17:52:40.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.535+0000 7f08168e3700 1 -- start start 2024-04-26T17:52:40.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.535+0000 7f08168e3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f081006b130 0x7f081019f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:40.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.535+0000 7f08168e3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f081006bad0 0x7f081019fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:40.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.535+0000 7f08168e3700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08101094a0 0x7f08101999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:40.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f08168e3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f08101135b0 con 0x7f081006bad0 2024-04-26T17:52:40.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f08168e3700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0810113430 con 0x7f08101094a0 2024-04-26T17:52:40.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f08168e3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0810113730 con 0x7f081006b130 2024-04-26T17:52:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f080ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f081006b130 0x7f081019f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f080f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f081006bad0 0x7f081019fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f080ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f081006b130 0x7f081019f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45152/0 (socket says 172.21.15.31:45152) 2024-04-26T17:52:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f080ffff700 1 -- 172.21.15.31:0/1926057287 learned_addr learned my addr 172.21.15.31:0/1926057287 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:52:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f080f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f081006bad0 0x7f081019fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52074/0 (socket says 172.21.15.31:52074) 2024-04-26T17:52:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f0814e80700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08101094a0 0x7f08101999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f080f7fe700 1 -- 172.21.15.31:0/1926057287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f081006b130 msgr2=0x7f081019f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f080f7fe700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f081006b130 0x7f081019f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f080f7fe700 1 -- 172.21.15.31:0/1926057287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08101094a0 msgr2=0x7f08101999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f080f7fe700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08101094a0 0x7f08101999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f080f7fe700 1 -- 172.21.15.31:0/1926057287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f081019a110 con 0x7f081006bad0 2024-04-26T17:52:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f0814e80700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08101094a0 0x7f08101999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:52:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f080ffff700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f081006b130 0x7f081019f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:52:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f080f7fe700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f081006bad0 0x7f081019fce0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f08040059d0 tx=0x7f08040047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.536+0000 7f080d7fa700 1 -- 172.21.15.31:0/1926057287 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0804027070 con 0x7f081006bad0 2024-04-26T17:52:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.537+0000 7f08168e3700 1 -- 172.21.15.31:0/1926057287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f081019a3a0 con 0x7f081006bad0 2024-04-26T17:52:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.537+0000 7f080d7fa700 1 -- 172.21.15.31:0/1926057287 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0804003fe0 con 0x7f081006bad0 2024-04-26T17:52:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.537+0000 7f080d7fa700 1 -- 172.21.15.31:0/1926057287 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0804022b20 con 0x7f081006bad0 2024-04-26T17:52:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.537+0000 7f08168e3700 1 -- 172.21.15.31:0/1926057287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08101a6670 con 0x7f081006bad0 2024-04-26T17:52:40.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.538+0000 7f08168e3700 1 -- 172.21.15.31:0/1926057287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f081006a110 con 0x7f081006bad0 2024-04-26T17:52:40.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.539+0000 7f080d7fa700 1 -- 172.21.15.31:0/1926057287 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f080402a070 con 0x7f081006bad0 2024-04-26T17:52:40.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.540+0000 7f080d7fa700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f07f80a3c40 0x7f07f80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:40.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.540+0000 7f080ffff700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f07f80a3c40 0x7f07f80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:40.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.540+0000 7f080d7fa700 1 -- 172.21.15.31:0/1926057287 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f08040d1490 con 0x7f081006bad0 2024-04-26T17:52:40.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.541+0000 7f080ffff700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f07f80a3c40 0x7f07f80a6100 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f08101087d0 tx=0x7f080000b480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:40.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.544+0000 7f080d7fa700 1 -- 172.21.15.31:0/1926057287 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0804031070 con 0x7f081006bad0 2024-04-26T17:52:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.809+0000 7f08168e3700 1 -- 172.21.15.31:0/1926057287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f081002e490 con 0x7f081006bad0 2024-04-26T17:52:40.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.810+0000 7f080d7fa700 1 -- 172.21.15.31:0/1926057287 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7f0804013070 con 0x7f081006bad0 2024-04-26T17:52:40.810 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:52:40.810 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24439,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T17:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.812+0000 7f08168e3700 1 -- 172.21.15.31:0/1926057287 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f07f80a3c40 msgr2=0x7f07f80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.812+0000 7f08168e3700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f07f80a3c40 0x7f07f80a6100 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f08101087d0 tx=0x7f080000b480 comp rx=0 tx=0).stop 2024-04-26T17:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.812+0000 7f08168e3700 1 -- 172.21.15.31:0/1926057287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f081006bad0 msgr2=0x7f081019fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.812+0000 7f08168e3700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f081006bad0 0x7f081019fce0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f08040059d0 tx=0x7f08040047b0 comp rx=0 tx=0).stop 2024-04-26T17:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.812+0000 7f08168e3700 1 -- 172.21.15.31:0/1926057287 shutdown_connections 2024-04-26T17:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.812+0000 7f08168e3700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f07f80a3c40 0x7f07f80a6100 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.812+0000 7f08168e3700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f08101094a0 0x7f08101999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.812+0000 7f08168e3700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f081006b130 0x7f081019f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.812+0000 7f08168e3700 1 --2- 172.21.15.31:0/1926057287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f081006bad0 0x7f081019fce0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.812+0000 7f08168e3700 1 -- 172.21.15.31:0/1926057287 >> 172.21.15.31:0/1926057287 conn(0x7f08100792d0 msgr2=0x7f0810100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.812+0000 7f08168e3700 1 -- 172.21.15.31:0/1926057287 shutdown_connections 2024-04-26T17:52:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:40.812+0000 7f08168e3700 1 -- 172.21.15.31:0/1926057287 wait complete. 2024-04-26T17:52:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 10 2024-04-26T17:52:41.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:40 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] up:boot 2024-04-26T17:52:41.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:40 smithi031 ceph-mon[29164]: fsmap 9 up:standby 2024-04-26T17:52:41.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:40 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T17:52:41.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:40 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:41.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:40 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:41.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:40 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:41.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:40 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:41.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:40 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:41.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:40 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:41.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:40 smithi031 ceph-mon[29164]: Deploying daemon mds.g on smithi031 2024-04-26T17:52:41.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:40 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1926057287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:40 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] up:boot 2024-04-26T17:52:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:40 smithi186 ceph-mon[32797]: fsmap 9 up:standby 2024-04-26T17:52:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:40 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T17:52:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:40 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:40 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:40 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:40 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:40 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:40 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:40 smithi186 ceph-mon[32797]: Deploying daemon mds.g on smithi031 2024-04-26T17:52:41.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:40 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1926057287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:40 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] up:boot 2024-04-26T17:52:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:40 smithi097 ceph-mon[32419]: fsmap 9 up:standby 2024-04-26T17:52:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:40 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T17:52:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:40 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:40 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:40 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:40 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:40 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:40 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:40 smithi097 ceph-mon[32419]: Deploying daemon mds.g on smithi031 2024-04-26T17:52:41.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:40 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1926057287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:42.357 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:42 smithi031 ceph-mon[29164]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:42.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:42 smithi097 ceph-mon[32419]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:42.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:42 smithi186 ceph-mon[32797]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:43.059 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 17:52:42 smithi031 systemd[1]: Starting Ceph mds.g for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:52:43.316 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 17:52:43 smithi031 podman[69029]: 2024-04-26 17:52:42.96867577 +0000 UTC m=+0.018330720 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:52:43.316 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 17:52:43 smithi031 podman[69029]: 2024-04-26 17:52:43.069014996 +0000 UTC m=+0.118669938 container create c4605ee34e42730939ce4424e2e6d8b1bbea89b1d5b25bd746129b315ba334ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T17:52:43.317 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 17:52:43 smithi031 podman[69029]: 2024-04-26 17:52:43.314073666 +0000 UTC m=+0.363728613 container init c4605ee34e42730939ce4424e2e6d8b1bbea89b1d5b25bd746129b315ba334ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, ceph=True) 2024-04-26T17:52:43.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:52:43.746 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:52:43.809 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 17:52:43 smithi031 podman[69029]: 2024-04-26 17:52:43.31693895 +0000 UTC m=+0.366593890 container start c4605ee34e42730939ce4424e2e6d8b1bbea89b1d5b25bd746129b315ba334ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-26T17:52:43.809 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 17:52:43 smithi031 ceph-mds[69048]: -- 172.21.15.31:0/2240708685 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5556c3c0fa40 con 0x5556c3be2c00 2024-04-26T17:52:43.810 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 17:52:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-g[69043]: starting mds.g at 2024-04-26T17:52:43.810 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 17:52:43 smithi031 bash[69029]: c4605ee34e42730939ce4424e2e6d8b1bbea89b1d5b25bd746129b315ba334ae 2024-04-26T17:52:43.810 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 17:52:43 smithi031 systemd[1]: Started Ceph mds.g for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:52:44.125 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:44 smithi031 ceph-mon[29164]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:44.125 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:44.125 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:44.125 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:44.125 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:44.126 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:44.126 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:44 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:44.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:44 smithi097 ceph-mon[32419]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:44.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:44.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:44.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:44.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:44.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:44.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:44 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:44.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:44 smithi186 ceph-mon[32797]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:44 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:45.300 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:45 smithi031 ceph-mon[29164]: Deploying daemon mds.a on smithi031 2024-04-26T17:52:45.300 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:45 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.31:6838/2783481787,v1:172.21.15.31:6839/2783481787] up:boot 2024-04-26T17:52:45.300 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:45 smithi031 ceph-mon[29164]: fsmap 10 up:standby 2024-04-26T17:52:45.300 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:45 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T17:52:45.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:45 smithi097 ceph-mon[32419]: Deploying daemon mds.a on smithi031 2024-04-26T17:52:45.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:45 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.31:6838/2783481787,v1:172.21.15.31:6839/2783481787] up:boot 2024-04-26T17:52:45.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:45 smithi097 ceph-mon[32419]: fsmap 10 up:standby 2024-04-26T17:52:45.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:45 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T17:52:45.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:45 smithi186 ceph-mon[32797]: Deploying daemon mds.a on smithi031 2024-04-26T17:52:45.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:45 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.31:6838/2783481787,v1:172.21.15.31:6839/2783481787] up:boot 2024-04-26T17:52:45.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:45 smithi186 ceph-mon[32797]: fsmap 10 up:standby 2024-04-26T17:52:45.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:45 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T17:52:46.124 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:46 smithi031 ceph-mon[29164]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:46.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:46 smithi097 ceph-mon[32419]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:46.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:46 smithi186 ceph-mon[32797]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:46.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.234+0000 7f56fae35700 1 -- 172.21.15.31:0/1246575651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56f4102ce0 msgr2=0x7f56f410f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:46.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.234+0000 7f56fae35700 1 --2- 172.21.15.31:0/1246575651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56f4102ce0 0x7f56f410f1d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00b0a0 tx=0x7f56ec00b3b0 comp rx=0 tx=0).stop 2024-04-26T17:52:46.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.234+0000 7f56fae35700 1 -- 172.21.15.31:0/1246575651 shutdown_connections 2024-04-26T17:52:46.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.234+0000 7f56fae35700 1 --2- 172.21.15.31:0/1246575651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56f4102ce0 0x7f56f410f1d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:46.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.234+0000 7f56fae35700 1 --2- 172.21.15.31:0/1246575651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56f4108390 0x7f56f4108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:46.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.234+0000 7f56fae35700 1 --2- 172.21.15.31:0/1246575651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56f4102320 0x7f56f41027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:46.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.234+0000 7f56fae35700 1 -- 172.21.15.31:0/1246575651 >> 172.21.15.31:0/1246575651 conn(0x7f56f40792d0 msgr2=0x7f56f40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:46.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.234+0000 7f56fae35700 1 -- 172.21.15.31:0/1246575651 shutdown_connections 2024-04-26T17:52:46.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.234+0000 7f56fae35700 1 -- 172.21.15.31:0/1246575651 wait complete. 2024-04-26T17:52:46.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56fae35700 1 Processor -- start 2024-04-26T17:52:46.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56fae35700 1 -- start start 2024-04-26T17:52:46.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56fae35700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56f4102320 0x7f56f419a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:46.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56fae35700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56f4102ce0 0x7f56f419aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:46.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56fae35700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56f4108390 0x7f56f41a5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:46.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56fae35700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f56f4112520 con 0x7f56f4108390 2024-04-26T17:52:46.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56fae35700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f56f41123a0 con 0x7f56f4102320 2024-04-26T17:52:46.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56fae35700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f56f41126a0 con 0x7f56f4102ce0 2024-04-26T17:52:46.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56f93d2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56f4108390 0x7f56f41a5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:46.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56f93d2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56f4108390 0x7f56f41a5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50312/0 (socket says 172.21.15.31:50312) 2024-04-26T17:52:46.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56f93d2700 1 -- 172.21.15.31:0/2367986746 learned_addr learned my addr 172.21.15.31:0/2367986746 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:52:46.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56f8bd1700 1 --2- 172.21.15.31:0/2367986746 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56f4102320 0x7f56f419a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:46.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56f3fff700 1 --2- 172.21.15.31:0/2367986746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56f4102ce0 0x7f56f419aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:46.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56f93d2700 1 -- 172.21.15.31:0/2367986746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56f4102ce0 msgr2=0x7f56f419aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:46.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56f93d2700 1 --2- 172.21.15.31:0/2367986746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56f4102ce0 0x7f56f419aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:46.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56f93d2700 1 -- 172.21.15.31:0/2367986746 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56f4102320 msgr2=0x7f56f419a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:46.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56f93d2700 1 --2- 172.21.15.31:0/2367986746 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56f4102320 0x7f56f419a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:46.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.235+0000 7f56f93d2700 1 -- 172.21.15.31:0/2367986746 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f41a5f40 con 0x7f56f4108390 2024-04-26T17:52:46.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.236+0000 7f56f93d2700 1 --2- 172.21.15.31:0/2367986746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56f4108390 0x7f56f41a5a00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009fd0 tx=0x7f56ec004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:46.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.236+0000 7f56f1ffb700 1 -- 172.21.15.31:0/2367986746 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec021070 con 0x7f56f4108390 2024-04-26T17:52:46.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.236+0000 7f56f1ffb700 1 -- 172.21.15.31:0/2367986746 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56ec007c00 con 0x7f56f4108390 2024-04-26T17:52:46.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.236+0000 7f56f1ffb700 1 -- 172.21.15.31:0/2367986746 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec0052a0 con 0x7f56f4108390 2024-04-26T17:52:46.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.236+0000 7f56fae35700 1 -- 172.21.15.31:0/2367986746 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f41a6110 con 0x7f56f4108390 2024-04-26T17:52:46.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.236+0000 7f56fae35700 1 -- 172.21.15.31:0/2367986746 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f41a6610 con 0x7f56f4108390 2024-04-26T17:52:46.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.237+0000 7f56fae35700 1 -- 172.21.15.31:0/2367986746 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f406a110 con 0x7f56f4108390 2024-04-26T17:52:46.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.239+0000 7f56f1ffb700 1 -- 172.21.15.31:0/2367986746 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f56ec024070 con 0x7f56f4108390 2024-04-26T17:52:46.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.239+0000 7f56f1ffb700 1 --2- 172.21.15.31:0/2367986746 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f56dc0a3cc0 0x7f56dc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:46.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.240+0000 7f56f8bd1700 1 --2- 172.21.15.31:0/2367986746 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f56dc0a3cc0 0x7f56dc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:46.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.240+0000 7f56f1ffb700 1 -- 172.21.15.31:0/2367986746 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f56ec00f020 con 0x7f56f4108390 2024-04-26T17:52:46.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.240+0000 7f56f8bd1700 1 --2- 172.21.15.31:0/2367986746 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f56dc0a3cc0 0x7f56dc0a6180 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f56e40045b0 tx=0x7f56e400b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:46.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.243+0000 7f56f1ffb700 1 -- 172.21.15.31:0/2367986746 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56ec02b030 con 0x7f56f4108390 2024-04-26T17:52:46.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.507+0000 7f56fae35700 1 -- 172.21.15.31:0/2367986746 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56f402e490 con 0x7f56f4108390 2024-04-26T17:52:46.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.507+0000 7f56f1ffb700 1 -- 172.21.15.31:0/2367986746 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7f56ec09e470 con 0x7f56f4108390 2024-04-26T17:52:46.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.510+0000 7f56fae35700 1 -- 172.21.15.31:0/2367986746 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f56dc0a3cc0 msgr2=0x7f56dc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:46.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.510+0000 7f56fae35700 1 --2- 172.21.15.31:0/2367986746 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f56dc0a3cc0 0x7f56dc0a6180 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f56e40045b0 tx=0x7f56e400b3c0 comp rx=0 tx=0).stop 2024-04-26T17:52:46.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.510+0000 7f56fae35700 1 -- 172.21.15.31:0/2367986746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56f4108390 msgr2=0x7f56f41a5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:46.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.510+0000 7f56fae35700 1 --2- 172.21.15.31:0/2367986746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56f4108390 0x7f56f41a5a00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009fd0 tx=0x7f56ec004dc0 comp rx=0 tx=0).stop 2024-04-26T17:52:46.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.510+0000 7f56fae35700 1 -- 172.21.15.31:0/2367986746 shutdown_connections 2024-04-26T17:52:46.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.510+0000 7f56fae35700 1 --2- 172.21.15.31:0/2367986746 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f56dc0a3cc0 0x7f56dc0a6180 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:46.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.510+0000 7f56fae35700 1 --2- 172.21.15.31:0/2367986746 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56f4102320 0x7f56f419a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:46.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.510+0000 7f56fae35700 1 --2- 172.21.15.31:0/2367986746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56f4108390 0x7f56f41a5a00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:46.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.510+0000 7f56fae35700 1 --2- 172.21.15.31:0/2367986746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56f4102ce0 0x7f56f419aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:46.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.510+0000 7f56fae35700 1 -- 172.21.15.31:0/2367986746 >> 172.21.15.31:0/2367986746 conn(0x7f56f40792d0 msgr2=0x7f56f4100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:46.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.510+0000 7f56fae35700 1 -- 172.21.15.31:0/2367986746 shutdown_connections 2024-04-26T17:52:46.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:46.510+0000 7f56fae35700 1 -- 172.21.15.31:0/2367986746 wait complete. 2024-04-26T17:52:46.665 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 11 2024-04-26T17:52:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:52:46.666 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24439,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T17:52:47.091 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 17:52:47 smithi031 systemd[1]: Starting Ceph mds.a for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:52:47.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:47 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2367986746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:47.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:47 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2367986746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:47.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:47 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2367986746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:47.727 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 17:52:47 smithi031 podman[69496]: 2024-04-26 17:52:47.518111023 +0000 UTC m=+0.019106671 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:52:47.727 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 17:52:47 smithi031 podman[69496]: 2024-04-26 17:52:47.618175516 +0000 UTC m=+0.119171181 container create 1647cb60c9742be5339601fff6fcb2fbcdebca9a053d6a23b0351b8f6902dfc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-26T17:52:48.105 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 17:52:47 smithi031 podman[69496]: 2024-04-26 17:52:47.845868211 +0000 UTC m=+0.346863834 container init 1647cb60c9742be5339601fff6fcb2fbcdebca9a053d6a23b0351b8f6902dfc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T17:52:48.105 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 17:52:47 smithi031 podman[69496]: 2024-04-26 17:52:47.848683694 +0000 UTC m=+0.349679316 container start 1647cb60c9742be5339601fff6fcb2fbcdebca9a053d6a23b0351b8f6902dfc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5) 2024-04-26T17:52:48.106 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 17:52:47 smithi031 ceph-mds[69515]: -- 172.21.15.31:0/3397514507 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fd1f251a40 con 0x55fd1f224c00 2024-04-26T17:52:48.106 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 17:52:47 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-a[69511]: starting mds.a at 2024-04-26T17:52:48.106 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 17:52:47 smithi031 bash[69496]: 1647cb60c9742be5339601fff6fcb2fbcdebca9a053d6a23b0351b8f6902dfc9 2024-04-26T17:52:48.106 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 17:52:47 smithi031 systemd[1]: Started Ceph mds.a for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:52:48.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:48 smithi097 ceph-mon[32419]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:48.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:48.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:48.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:48.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:48.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:48.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:48.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:48 smithi186 ceph-mon[32797]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:48.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:48.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:48.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:48.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:48.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:48.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:48.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:48 smithi031 ceph-mon[29164]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:48.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:48.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:48.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:48.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T17:52:48.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T17:52:48.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:49.338 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:49 smithi097 ceph-mon[32419]: Deploying daemon mds.b on smithi097 2024-04-26T17:52:49.339 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:49 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.31:6840/2984904862,v1:172.21.15.31:6841/2984904862] up:boot 2024-04-26T17:52:49.339 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:49 smithi097 ceph-mon[32419]: fsmap 11 up:standby 2024-04-26T17:52:49.339 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T17:52:49.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:52:49.468 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:49 smithi031 ceph-mon[29164]: Deploying daemon mds.b on smithi097 2024-04-26T17:52:49.468 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:49 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.31:6840/2984904862,v1:172.21.15.31:6841/2984904862] up:boot 2024-04-26T17:52:49.468 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:49 smithi031 ceph-mon[29164]: fsmap 11 up:standby 2024-04-26T17:52:49.468 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T17:52:49.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:49 smithi186 ceph-mon[32797]: Deploying daemon mds.b on smithi097 2024-04-26T17:52:49.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:49 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.31:6840/2984904862,v1:172.21.15.31:6841/2984904862] up:boot 2024-04-26T17:52:49.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:49 smithi186 ceph-mon[32797]: fsmap 11 up:standby 2024-04-26T17:52:49.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T17:52:49.705 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:52:50.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:50 smithi097 ceph-mon[32419]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:50.384 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 17:52:50 smithi097 systemd[1]: Starting Ceph mds.b for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T17:52:50.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:50 smithi186 ceph-mon[32797]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:50.544 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:50 smithi031 ceph-mon[29164]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:50.720 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 17:52:50 smithi097 podman[63054]: 2024-04-26 17:52:50.35043303 +0000 UTC m=+0.017629633 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T17:52:50.720 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 17:52:50 smithi097 podman[63054]: 2024-04-26 17:52:50.468960443 +0000 UTC m=+0.136157075 container create 6ab9bc8bb9b0b41aea137e49f67971e38325d4627e2ac6b38d3b0fe138fff597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T17:52:50.720 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 17:52:50 smithi097 podman[63054]: 2024-04-26 17:52:50.721591779 +0000 UTC m=+0.388788368 container init 6ab9bc8bb9b0b41aea137e49f67971e38325d4627e2ac6b38d3b0fe138fff597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-26T17:52:51.134 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 17:52:50 smithi097 podman[63054]: 2024-04-26 17:52:50.724389209 +0000 UTC m=+0.391585808 container start 6ab9bc8bb9b0b41aea137e49f67971e38325d4627e2ac6b38d3b0fe138fff597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T17:52:51.134 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 17:52:50 smithi097 ceph-mds[63073]: -- 172.21.15.97:0/53632109 <== mon.2 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5587529a9a40 con 0x55875297c800 2024-04-26T17:52:51.134 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 17:52:50 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: starting mds.b at 2024-04-26T17:52:51.134 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 17:52:50 smithi097 bash[63054]: 6ab9bc8bb9b0b41aea137e49f67971e38325d4627e2ac6b38d3b0fe138fff597 2024-04-26T17:52:51.134 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 17:52:50 smithi097 systemd[1]: Started Ceph mds.b for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T17:52:51.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.396+0000 7f5d75120700 1 -- 172.21.15.31:0/1512149970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d70103e40 msgr2=0x7f5d70110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:51.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.396+0000 7f5d75120700 1 --2- 172.21.15.31:0/1512149970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d70103e40 0x7f5d70110320 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800b0a0 tx=0x7f5d6800b3b0 comp rx=0 tx=0).stop 2024-04-26T17:52:51.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.396+0000 7f5d75120700 1 -- 172.21.15.31:0/1512149970 shutdown_connections 2024-04-26T17:52:51.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.396+0000 7f5d75120700 1 --2- 172.21.15.31:0/1512149970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d70103e40 0x7f5d70110320 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:51.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.396+0000 7f5d75120700 1 --2- 172.21.15.31:0/1512149970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d701094e0 0x7f5d701098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:51.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.396+0000 7f5d75120700 1 --2- 172.21.15.31:0/1512149970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d701034a0 0x7f5d70103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:51.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.396+0000 7f5d75120700 1 -- 172.21.15.31:0/1512149970 >> 172.21.15.31:0/1512149970 conn(0x7f5d7007a5d0 msgr2=0x7f5d7007a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:51.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.397+0000 7f5d75120700 1 -- 172.21.15.31:0/1512149970 shutdown_connections 2024-04-26T17:52:51.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.397+0000 7f5d75120700 1 -- 172.21.15.31:0/1512149970 wait complete. 2024-04-26T17:52:51.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.397+0000 7f5d75120700 1 Processor -- start 2024-04-26T17:52:51.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d75120700 1 -- start start 2024-04-26T17:52:51.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d75120700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d701034a0 0x7f5d70075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:51.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d75120700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d70103e40 0x7f5d70075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:51.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d75120700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d701094e0 0x7f5d700761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:51.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d75120700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5d70113600 con 0x7f5d70103e40 2024-04-26T17:52:51.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d75120700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f5d70113480 con 0x7f5d701094e0 2024-04-26T17:52:51.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d75120700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5d70113780 con 0x7f5d701034a0 2024-04-26T17:52:51.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d6e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d70103e40 0x7f5d70075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:51.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d6ed9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d701034a0 0x7f5d70075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:51.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d6e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d70103e40 0x7f5d70075c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50354/0 (socket says 172.21.15.31:50354) 2024-04-26T17:52:51.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d6e59c700 1 -- 172.21.15.31:0/2098953281 learned_addr learned my addr 172.21.15.31:0/2098953281 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:52:51.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d6f59e700 1 --2- 172.21.15.31:0/2098953281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d701094e0 0x7f5d700761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:51.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d6e59c700 1 -- 172.21.15.31:0/2098953281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d701034a0 msgr2=0x7f5d70075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:51.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d6e59c700 1 --2- 172.21.15.31:0/2098953281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d701034a0 0x7f5d70075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:51.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d6e59c700 1 -- 172.21.15.31:0/2098953281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d701094e0 msgr2=0x7f5d700761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:51.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d6e59c700 1 --2- 172.21.15.31:0/2098953281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d701094e0 0x7f5d700761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:51.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.398+0000 7f5d6e59c700 1 -- 172.21.15.31:0/2098953281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d701a8060 con 0x7f5d70103e40 2024-04-26T17:52:51.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.399+0000 7f5d6e59c700 1 --2- 172.21.15.31:0/2098953281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d70103e40 0x7f5d70075c60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000d950 tx=0x7f5d6000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:51.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.399+0000 7f5d67fff700 1 -- 172.21.15.31:0/2098953281 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d60016070 con 0x7f5d70103e40 2024-04-26T17:52:51.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.399+0000 7f5d75120700 1 -- 172.21.15.31:0/2098953281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d701a8350 con 0x7f5d70103e40 2024-04-26T17:52:51.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.399+0000 7f5d67fff700 1 -- 172.21.15.31:0/2098953281 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d6000fd40 con 0x7f5d70103e40 2024-04-26T17:52:51.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.399+0000 7f5d67fff700 1 -- 172.21.15.31:0/2098953281 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6000e3b0 con 0x7f5d70103e40 2024-04-26T17:52:51.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.399+0000 7f5d75120700 1 -- 172.21.15.31:0/2098953281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d701a88e0 con 0x7f5d70103e40 2024-04-26T17:52:51.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.400+0000 7f5d75120700 1 -- 172.21.15.31:0/2098953281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d7006a110 con 0x7f5d70103e40 2024-04-26T17:52:51.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.401+0000 7f5d67fff700 1 -- 172.21.15.31:0/2098953281 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5d60019070 con 0x7f5d70103e40 2024-04-26T17:52:51.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.402+0000 7f5d67fff700 1 --2- 172.21.15.31:0/2098953281 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f5d5c0a3cc0 0x7f5d5c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:51.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.402+0000 7f5d6ed9d700 1 --2- 172.21.15.31:0/2098953281 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f5d5c0a3cc0 0x7f5d5c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:51.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.402+0000 7f5d67fff700 1 -- 172.21.15.31:0/2098953281 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5d600cfd70 con 0x7f5d70103e40 2024-04-26T17:52:51.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.402+0000 7f5d6ed9d700 1 --2- 172.21.15.31:0/2098953281 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f5d5c0a3cc0 0x7f5d5c0a6180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5d580045b0 tx=0x7f5d5800b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:51.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.405+0000 7f5d67fff700 1 -- 172.21.15.31:0/2098953281 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d6009cb50 con 0x7f5d70103e40 2024-04-26T17:52:51.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.681+0000 7f5d75120700 1 -- 172.21.15.31:0/2098953281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d70076c50 con 0x7f5d70103e40 2024-04-26T17:52:51.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.682+0000 7f5d67fff700 1 -- 172.21.15.31:0/2098953281 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10482 (secure 0 0 0) 0x7f5d6009c450 con 0x7f5d70103e40 2024-04-26T17:52:51.683 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:52:51.684 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24428,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24439,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24473,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-26T17:52:51.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.684+0000 7f5d75120700 1 -- 172.21.15.31:0/2098953281 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f5d5c0a3cc0 msgr2=0x7f5d5c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.684+0000 7f5d75120700 1 --2- 172.21.15.31:0/2098953281 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f5d5c0a3cc0 0x7f5d5c0a6180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5d580045b0 tx=0x7f5d5800b3c0 comp rx=0 tx=0).stop 2024-04-26T17:52:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.684+0000 7f5d75120700 1 -- 172.21.15.31:0/2098953281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d70103e40 msgr2=0x7f5d70075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.684+0000 7f5d75120700 1 --2- 172.21.15.31:0/2098953281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d70103e40 0x7f5d70075c60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000d950 tx=0x7f5d6000dc60 comp rx=0 tx=0).stop 2024-04-26T17:52:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.684+0000 7f5d75120700 1 -- 172.21.15.31:0/2098953281 shutdown_connections 2024-04-26T17:52:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.684+0000 7f5d75120700 1 --2- 172.21.15.31:0/2098953281 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f5d5c0a3cc0 0x7f5d5c0a6180 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.684+0000 7f5d75120700 1 --2- 172.21.15.31:0/2098953281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d701094e0 0x7f5d700761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.684+0000 7f5d75120700 1 --2- 172.21.15.31:0/2098953281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d701034a0 0x7f5d70075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.684+0000 7f5d75120700 1 --2- 172.21.15.31:0/2098953281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d70103e40 0x7f5d70075c60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.684+0000 7f5d75120700 1 -- 172.21.15.31:0/2098953281 >> 172.21.15.31:0/2098953281 conn(0x7f5d7007a5d0 msgr2=0x7f5d70101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.685+0000 7f5d75120700 1 -- 172.21.15.31:0/2098953281 shutdown_connections 2024-04-26T17:52:51.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:51.685+0000 7f5d75120700 1 -- 172.21.15.31:0/2098953281 wait complete. 2024-04-26T17:52:51.686 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2024-04-26T17:52:52.049 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:51 smithi031 ceph-mon[29164]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:52.049 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:51 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.049 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:51 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.049 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:51 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.049 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:51 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.049 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:51 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.050 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:51 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:52:52.050 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:51 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] up:boot 2024-04-26T17:52:52.050 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:51 smithi031 ceph-mon[29164]: fsmap 12 up:standby 2024-04-26T17:52:52.050 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:51 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T17:52:52.050 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:51 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2098953281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:51 smithi097 ceph-mon[32419]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:51 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:51 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:51 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:51 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:51 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:51 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:52:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:51 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] up:boot 2024-04-26T17:52:52.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:51 smithi097 ceph-mon[32419]: fsmap 12 up:standby 2024-04-26T17:52:52.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:51 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T17:52:52.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:51 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2098953281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:52.178 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:51 smithi186 ceph-mon[32797]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:52.178 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:51 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.178 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:51 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:51 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:51 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:51 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:52.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:51 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:52:52.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:51 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] up:boot 2024-04-26T17:52:52.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:51 smithi186 ceph-mon[32797]: fsmap 12 up:standby 2024-04-26T17:52:52.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:51 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T17:52:52.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:51 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2098953281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:52:52.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-04-26T17:52:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:52:54.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:53 smithi097 ceph-mon[32419]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:54.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:53 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:53 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:53 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:53 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:53 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.164 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:53 smithi031 ceph-mon[29164]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:54.164 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:53 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.164 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:53 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.164 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:53 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:53 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:53 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:53 smithi186 ceph-mon[32797]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:53 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:53 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:53 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:53 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:53 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:54.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.796+0000 7fb7c3d31700 1 -- 172.21.15.31:0/147027428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7bc103470 msgr2=0x7fb7bc1038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:54.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.796+0000 7fb7c3d31700 1 --2- 172.21.15.31:0/147027428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7bc103470 0x7fb7bc1038f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009990 tx=0x7fb7ac009ca0 comp rx=0 tx=0).stop 2024-04-26T17:52:54.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.796+0000 7fb7c3d31700 1 -- 172.21.15.31:0/147027428 shutdown_connections 2024-04-26T17:52:54.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.796+0000 7fb7c3d31700 1 --2- 172.21.15.31:0/147027428 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7bc103e30 0x7fb7bc110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:54.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.796+0000 7fb7c3d31700 1 --2- 172.21.15.31:0/147027428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7bc1094c0 0x7fb7bc1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:54.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.796+0000 7fb7c3d31700 1 --2- 172.21.15.31:0/147027428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7bc103470 0x7fb7bc1038f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:54.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.796+0000 7fb7c3d31700 1 -- 172.21.15.31:0/147027428 >> 172.21.15.31:0/147027428 conn(0x7fb7bc07a580 msgr2=0x7fb7bc07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:54.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.797+0000 7fb7c3d31700 1 -- 172.21.15.31:0/147027428 shutdown_connections 2024-04-26T17:52:54.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.797+0000 7fb7c3d31700 1 -- 172.21.15.31:0/147027428 wait complete. 2024-04-26T17:52:54.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.797+0000 7fb7c3d31700 1 Processor -- start 2024-04-26T17:52:54.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.797+0000 7fb7c3d31700 1 -- start start 2024-04-26T17:52:54.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.797+0000 7fb7c3d31700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7bc103470 0x7fb7bc19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:54.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.797+0000 7fb7c3d31700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7bc103e30 0x7fb7bc19ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:54.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.797+0000 7fb7c3d31700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7bc1094c0 0x7fb7bc199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:54.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.797+0000 7fb7c3d31700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1136c0 con 0x7fb7bc103470 2024-04-26T17:52:54.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.797+0000 7fb7c3d31700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc113540 con 0x7fb7bc103e30 2024-04-26T17:52:54.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.797+0000 7fb7c3d31700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc113840 con 0x7fb7bc1094c0 2024-04-26T17:52:54.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.797+0000 7fb7c1acd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7bc103470 0x7fb7bc19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:54.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7c1acd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7bc103470 0x7fb7bc19fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56546/0 (socket says 172.21.15.31:56546) 2024-04-26T17:52:54.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7c1acd700 1 -- 172.21.15.31:0/2850613692 learned_addr learned my addr 172.21.15.31:0/2850613692 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:52:54.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7c22ce700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7bc1094c0 0x7fb7bc199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:54.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7c12cc700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7bc103e30 0x7fb7bc19ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:54.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7c1acd700 1 -- 172.21.15.31:0/2850613692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7bc1094c0 msgr2=0x7fb7bc199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:54.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7c1acd700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7bc1094c0 0x7fb7bc199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7c1acd700 1 -- 172.21.15.31:0/2850613692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7bc103e30 msgr2=0x7fb7bc19ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7c1acd700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7bc103e30 0x7fb7bc19ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7c1acd700 1 -- 172.21.15.31:0/2850613692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7bc19a410 con 0x7fb7bc103470 2024-04-26T17:52:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7c12cc700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7bc103e30 0x7fb7bc19ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:52:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7c1acd700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7bc103470 0x7fb7bc19fa60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800b530 tx=0x7fb7b800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7b2ffd700 1 -- 172.21.15.31:0/2850613692 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8015020 con 0x7fb7bc103470 2024-04-26T17:52:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7b2ffd700 1 -- 172.21.15.31:0/2850613692 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7b8011ce0 con 0x7fb7bc103470 2024-04-26T17:52:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7b2ffd700 1 -- 172.21.15.31:0/2850613692 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8010e40 con 0x7fb7bc103470 2024-04-26T17:52:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7c3d31700 1 -- 172.21.15.31:0/2850613692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7bc19a700 con 0x7fb7bc103470 2024-04-26T17:52:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.798+0000 7fb7c3d31700 1 -- 172.21.15.31:0/2850613692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7bc1a69e0 con 0x7fb7bc103470 2024-04-26T17:52:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.799+0000 7fb7c3d31700 1 -- 172.21.15.31:0/2850613692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7bc104b60 con 0x7fb7bc103470 2024-04-26T17:52:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.801+0000 7fb7b2ffd700 1 -- 172.21.15.31:0/2850613692 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb7b8018020 con 0x7fb7bc103470 2024-04-26T17:52:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.801+0000 7fb7b2ffd700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb7a80a3cc0 0x7fb7a80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.801+0000 7fb7c12cc700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb7a80a3cc0 0x7fb7a80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.801+0000 7fb7b2ffd700 1 -- 172.21.15.31:0/2850613692 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb7b80cfc90 con 0x7fb7bc103470 2024-04-26T17:52:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.802+0000 7fb7c12cc700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb7a80a3cc0 0x7fb7a80a6180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac006010 tx=0x7fb7ac01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:54.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:54.805+0000 7fb7b2ffd700 1 -- 172.21.15.31:0/2850613692 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb7b801f020 con 0x7fb7bc103470 2024-04-26T17:52:55.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:55.046+0000 7fb7c3d31700 1 -- 172.21.15.31:0/2850613692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fb7bc02e490 con 0x7fb7bc103470 2024-04-26T17:52:55.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:55.047+0000 7fb7b2ffd700 1 -- 172.21.15.31:0/2850613692 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fb7b809cb10 con 0x7fb7bc103470 2024-04-26T17:52:55.047 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:52:55.047 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-04-26T17:52:55.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T17:52:55.047 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-04-26T17:52:55.047 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-04-26T17:52:55.047 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:52:55.047 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-04-26T17:52:55.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:55.049+0000 7fb7c3d31700 1 -- 172.21.15.31:0/2850613692 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb7a80a3cc0 msgr2=0x7fb7a80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:55.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:55.049+0000 7fb7c3d31700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb7a80a3cc0 0x7fb7a80a6180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac006010 tx=0x7fb7ac01a040 comp rx=0 tx=0).stop 2024-04-26T17:52:55.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:55.049+0000 7fb7c3d31700 1 -- 172.21.15.31:0/2850613692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7bc103470 msgr2=0x7fb7bc19fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:55.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:55.049+0000 7fb7c3d31700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7bc103470 0x7fb7bc19fa60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800b530 tx=0x7fb7b800b840 comp rx=0 tx=0).stop 2024-04-26T17:52:55.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:55.049+0000 7fb7c3d31700 1 -- 172.21.15.31:0/2850613692 shutdown_connections 2024-04-26T17:52:55.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:55.049+0000 7fb7c3d31700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb7a80a3cc0 0x7fb7a80a6180 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:55.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:55.049+0000 7fb7c3d31700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7bc1094c0 0x7fb7bc199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:55.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:55.049+0000 7fb7c3d31700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7bc103470 0x7fb7bc19fa60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:55.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:55.049+0000 7fb7c3d31700 1 --2- 172.21.15.31:0/2850613692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb7bc103e30 0x7fb7bc19ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:55.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:55.049+0000 7fb7c3d31700 1 -- 172.21.15.31:0/2850613692 >> 172.21.15.31:0/2850613692 conn(0x7fb7bc07a580 msgr2=0x7fb7bc1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:55.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:55.049+0000 7fb7c3d31700 1 -- 172.21.15.31:0/2850613692 shutdown_connections 2024-04-26T17:52:55.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:55.050+0000 7fb7c3d31700 1 -- 172.21.15.31:0/2850613692 wait complete. 2024-04-26T17:52:55.739 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T17:52:55.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T17:52:56.289 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:56 smithi031 ceph-mon[29164]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:56.289 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:56.289 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:56.289 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:56.289 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:52:56.289 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:56 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2850613692' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T17:52:56.290 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:56.290 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:56 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:52:56.312 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:56 smithi097 ceph-mon[32419]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:56.312 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:56.312 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:56.312 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:56.312 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:52:56.312 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:56 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2850613692' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T17:52:56.312 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:56.312 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:56 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:52:56.315 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:56 smithi186 ceph-mon[32797]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:56.315 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:56.315 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:56.315 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:56.315 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:52:56.315 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:56 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2850613692' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T17:52:56.315 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:56.315 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:56 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:52:56.632 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:52:58.036 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:57 smithi031 ceph-mon[29164]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:58.037 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:57 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:58.037 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:57 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:58.037 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:57 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:58.037 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:57 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:58.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:57 smithi186 ceph-mon[32797]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:58.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:57 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:58.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:57 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:58.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:57 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:58.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:57 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:58.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:57 smithi097 ceph-mon[32419]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:58.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:57 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:58.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:57 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:58.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:57 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:58.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:57 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:58.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.396+0000 7fa07b133700 1 -- 172.21.15.31:0/3824262661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa074103310 msgr2=0x7fa074106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:58.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.396+0000 7fa07b133700 1 --2- 172.21.15.31:0/3824262661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa074103310 0x7fa074106f80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa0680099c0 tx=0x7fa068009cd0 comp rx=0 tx=0).stop 2024-04-26T17:52:58.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.396+0000 7fa07b133700 1 -- 172.21.15.31:0/3824262661 shutdown_connections 2024-04-26T17:52:58.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.396+0000 7fa07b133700 1 --2- 172.21.15.31:0/3824262661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa074102960 0x7fa074102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:58.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.396+0000 7fa07b133700 1 --2- 172.21.15.31:0/3824262661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0741076b0 0x7fa07410f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:58.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.396+0000 7fa07b133700 1 --2- 172.21.15.31:0/3824262661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa074103310 0x7fa074106f80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:58.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.396+0000 7fa07b133700 1 -- 172.21.15.31:0/3824262661 >> 172.21.15.31:0/3824262661 conn(0x7fa0740792d0 msgr2=0x7fa0740796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:58.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.396+0000 7fa07b133700 1 -- 172.21.15.31:0/3824262661 shutdown_connections 2024-04-26T17:52:58.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.396+0000 7fa07b133700 1 -- 172.21.15.31:0/3824262661 wait complete. 2024-04-26T17:52:58.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.397+0000 7fa07b133700 1 Processor -- start 2024-04-26T17:52:58.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.397+0000 7fa07b133700 1 -- start start 2024-04-26T17:52:58.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.397+0000 7fa07b133700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa074102960 0x7fa07419bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:58.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.397+0000 7fa07b133700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa074103310 0x7fa07419c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:58.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.397+0000 7fa07b133700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0741076b0 0x7fa0741a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:58.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.397+0000 7fa07b133700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa074112550 con 0x7fa074102960 2024-04-26T17:52:58.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.397+0000 7fa07b133700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa0741123d0 con 0x7fa0741076b0 2024-04-26T17:52:58.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.397+0000 7fa07b133700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa0741126d0 con 0x7fa074103310 2024-04-26T17:52:58.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.397+0000 7fa078ecf700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa074102960 0x7fa07419bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:58.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.397+0000 7fa078ecf700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa074102960 0x7fa07419bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56576/0 (socket says 172.21.15.31:56576) 2024-04-26T17:52:58.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.397+0000 7fa078ecf700 1 -- 172.21.15.31:0/1839371638 learned_addr learned my addr 172.21.15.31:0/1839371638 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:52:58.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.398+0000 7fa0796d0700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0741076b0 0x7fa0741a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:58.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.398+0000 7fa073fff700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa074103310 0x7fa07419c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:58.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.398+0000 7fa078ecf700 1 -- 172.21.15.31:0/1839371638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa074103310 msgr2=0x7fa07419c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:58.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.398+0000 7fa078ecf700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa074103310 0x7fa07419c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:58.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.398+0000 7fa078ecf700 1 -- 172.21.15.31:0/1839371638 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0741076b0 msgr2=0x7fa0741a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:58.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.398+0000 7fa078ecf700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0741076b0 0x7fa0741a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:58.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.398+0000 7fa078ecf700 1 -- 172.21.15.31:0/1839371638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0741a0d80 con 0x7fa074102960 2024-04-26T17:52:58.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.398+0000 7fa073fff700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa074103310 0x7fa07419c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:52:58.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.398+0000 7fa078ecf700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa074102960 0x7fa07419bd70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa06400e9a0 tx=0x7fa06400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:58.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.398+0000 7fa071ffb700 1 -- 172.21.15.31:0/1839371638 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa06400cbd0 con 0x7fa074102960 2024-04-26T17:52:58.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.398+0000 7fa07b133700 1 -- 172.21.15.31:0/1839371638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0741a1070 con 0x7fa074102960 2024-04-26T17:52:58.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.398+0000 7fa071ffb700 1 -- 172.21.15.31:0/1839371638 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa06400cd70 con 0x7fa074102960 2024-04-26T17:52:58.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.398+0000 7fa071ffb700 1 -- 172.21.15.31:0/1839371638 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa064018680 con 0x7fa074102960 2024-04-26T17:52:58.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.398+0000 7fa07b133700 1 -- 172.21.15.31:0/1839371638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0741a88f0 con 0x7fa074102960 2024-04-26T17:52:58.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.399+0000 7fa07b133700 1 -- 172.21.15.31:0/1839371638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa074103a60 con 0x7fa074102960 2024-04-26T17:52:58.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.401+0000 7fa071ffb700 1 -- 172.21.15.31:0/1839371638 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa064010970 con 0x7fa074102960 2024-04-26T17:52:58.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.401+0000 7fa071ffb700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa05c0a3c40 0x7fa05c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:52:58.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.402+0000 7fa073fff700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa05c0a3c40 0x7fa05c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:52:58.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.402+0000 7fa071ffb700 1 -- 172.21.15.31:0/1839371638 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa0640d0260 con 0x7fa074102960 2024-04-26T17:52:58.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.402+0000 7fa073fff700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa05c0a3c40 0x7fa05c0a6100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa068006010 tx=0x7fa068000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:52:58.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.406+0000 7fa071ffb700 1 -- 172.21.15.31:0/1839371638 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa06409d0e0 con 0x7fa074102960 2024-04-26T17:52:58.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:58.644+0000 7fa07b133700 1 -- 172.21.15.31:0/1839371638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa0741a8b20 con 0x7fa074102960 2024-04-26T17:52:59.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:59.568+0000 7fa071ffb700 1 -- 172.21.15.31:0/1839371638 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fa06409c9e0 con 0x7fa074102960 2024-04-26T17:52:59.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:59.570+0000 7fa07b133700 1 -- 172.21.15.31:0/1839371638 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa05c0a3c40 msgr2=0x7fa05c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:59.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:59.570+0000 7fa07b133700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa05c0a3c40 0x7fa05c0a6100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa068006010 tx=0x7fa068000bc0 comp rx=0 tx=0).stop 2024-04-26T17:52:59.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:59.571+0000 7fa07b133700 1 -- 172.21.15.31:0/1839371638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa074102960 msgr2=0x7fa07419bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:52:59.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:59.571+0000 7fa07b133700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa074102960 0x7fa07419bd70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa06400e9a0 tx=0x7fa06400ed60 comp rx=0 tx=0).stop 2024-04-26T17:52:59.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:59.571+0000 7fa07b133700 1 -- 172.21.15.31:0/1839371638 shutdown_connections 2024-04-26T17:52:59.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:59.571+0000 7fa07b133700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa05c0a3c40 0x7fa05c0a6100 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:59.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:59.571+0000 7fa07b133700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0741076b0 0x7fa0741a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:59.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:59.571+0000 7fa07b133700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa074102960 0x7fa07419bd70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:59.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:59.571+0000 7fa07b133700 1 --2- 172.21.15.31:0/1839371638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa074103310 0x7fa07419c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:52:59.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:59.571+0000 7fa07b133700 1 -- 172.21.15.31:0/1839371638 >> 172.21.15.31:0/1839371638 conn(0x7fa0740792d0 msgr2=0x7fa07406bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:52:59.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:59.571+0000 7fa07b133700 1 -- 172.21.15.31:0/1839371638 shutdown_connections 2024-04-26T17:52:59.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:52:59.572+0000 7fa07b133700 1 -- 172.21.15.31:0/1839371638 wait complete. 2024-04-26T17:52:59.574 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-04-26T17:52:59.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:59 smithi097 ceph-mon[32419]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:59.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:59 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1839371638' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T17:52:59.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:59.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:59.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:59.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:52:59.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:52:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:59.901 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:59 smithi031 ceph-mon[29164]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:52:59.901 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:59 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1839371638' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T17:52:59.901 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:59.901 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:52:59.902 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:52:59.902 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:52:59.902 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:52:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:53:00.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:59 smithi186 ceph-mon[32797]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:53:00.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:59 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1839371638' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T17:53:00.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:53:00.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:53:00.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:53:00.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:53:00.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:52:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:53:00.313 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T17:53:00.572 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:00.843 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:00 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1839371638' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T17:53:00.843 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:00 smithi031 ceph-mon[29164]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T17:53:00.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:00 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1839371638' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T17:53:00.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:00 smithi097 ceph-mon[32419]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T17:53:01.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:00 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1839371638' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T17:53:01.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:00 smithi186 ceph-mon[32797]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T17:53:01.712 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:01 smithi031 ceph-mon[29164]: pgmap v288: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:53:01.712 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:01 smithi031 ceph-mon[29164]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T17:53:01.712 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:01 smithi031 ceph-mon[29164]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T17:53:01.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:01 smithi097 ceph-mon[32419]: pgmap v288: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:53:01.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:01 smithi097 ceph-mon[32419]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T17:53:01.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:01 smithi097 ceph-mon[32419]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T17:53:02.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:01 smithi186 ceph-mon[32797]: pgmap v288: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:53:02.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:01 smithi186 ceph-mon[32797]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T17:53:02.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:01 smithi186 ceph-mon[32797]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T17:53:02.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.265+0000 7f3cde9c6700 1 -- 172.21.15.31:0/875622363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cd8103030 msgr2=0x7f3cd810f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:02.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.265+0000 7f3cde9c6700 1 --2- 172.21.15.31:0/875622363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cd8103030 0x7f3cd810f1b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3cd000b0d0 tx=0x7f3cd000b3e0 comp rx=0 tx=0).stop 2024-04-26T17:53:02.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.265+0000 7f3cde9c6700 1 -- 172.21.15.31:0/875622363 shutdown_connections 2024-04-26T17:53:02.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.265+0000 7f3cde9c6700 1 --2- 172.21.15.31:0/875622363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cd8103030 0x7f3cd810f1b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:02.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.265+0000 7f3cde9c6700 1 --2- 172.21.15.31:0/875622363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cd81073a0 0x7f3cd8107780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:02.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.265+0000 7f3cde9c6700 1 --2- 172.21.15.31:0/875622363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd8102670 0x7f3cd8102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:02.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.265+0000 7f3cde9c6700 1 -- 172.21.15.31:0/875622363 >> 172.21.15.31:0/875622363 conn(0x7f3cd8078190 msgr2=0x7f3cd80785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:02.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.265+0000 7f3cde9c6700 1 -- 172.21.15.31:0/875622363 shutdown_connections 2024-04-26T17:53:02.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.265+0000 7f3cde9c6700 1 -- 172.21.15.31:0/875622363 wait complete. 2024-04-26T17:53:02.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.266+0000 7f3cde9c6700 1 Processor -- start 2024-04-26T17:53:02.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.266+0000 7f3cde9c6700 1 -- start start 2024-04-26T17:53:02.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.266+0000 7f3cde9c6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cd8102670 0x7f3cd819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:02.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.266+0000 7f3cde9c6700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd8103030 0x7f3cd819fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:02.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.266+0000 7f3cde9c6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cd81073a0 0x7f3cd8199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:02.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.266+0000 7f3cde9c6700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd81125d0 con 0x7f3cd8102670 2024-04-26T17:53:02.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.266+0000 7f3cde9c6700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd8112450 con 0x7f3cd8103030 2024-04-26T17:53:02.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.266+0000 7f3cde9c6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd8112750 con 0x7f3cd81073a0 2024-04-26T17:53:02.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cd7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cd8102670 0x7f3cd819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:02.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cd7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cd8102670 0x7f3cd819f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56582/0 (socket says 172.21.15.31:56582) 2024-04-26T17:53:02.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cd7fff700 1 -- 172.21.15.31:0/2442369297 learned_addr learned my addr 172.21.15.31:0/2442369297 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:02.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cd77fe700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd8103030 0x7f3cd819fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:02.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cdcf63700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cd81073a0 0x7f3cd8199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:02.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cd7fff700 1 -- 172.21.15.31:0/2442369297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cd81073a0 msgr2=0x7f3cd8199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cd7fff700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cd81073a0 0x7f3cd8199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cd7fff700 1 -- 172.21.15.31:0/2442369297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd8103030 msgr2=0x7f3cd819fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cd7fff700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd8103030 0x7f3cd819fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cd7fff700 1 -- 172.21.15.31:0/2442369297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd819a0d0 con 0x7f3cd8102670 2024-04-26T17:53:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cdcf63700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cd81073a0 0x7f3cd8199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:53:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cd7fff700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cd8102670 0x7f3cd819f7b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3cc800b530 tx=0x7f3cc800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cd57fa700 1 -- 172.21.15.31:0/2442369297 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8015020 con 0x7f3cd8102670 2024-04-26T17:53:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cd57fa700 1 -- 172.21.15.31:0/2442369297 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cc8011ce0 con 0x7f3cd8102670 2024-04-26T17:53:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cde9c6700 1 -- 172.21.15.31:0/2442369297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd819a3c0 con 0x7f3cd8102670 2024-04-26T17:53:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cd57fa700 1 -- 172.21.15.31:0/2442369297 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8010d20 con 0x7f3cd8102670 2024-04-26T17:53:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.267+0000 7f3cde9c6700 1 -- 172.21.15.31:0/2442369297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cd81a6680 con 0x7f3cd8102670 2024-04-26T17:53:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.268+0000 7f3cde9c6700 1 -- 172.21.15.31:0/2442369297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cd8103a90 con 0x7f3cd8102670 2024-04-26T17:53:02.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.270+0000 7f3cd57fa700 1 -- 172.21.15.31:0/2442369297 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3cc8018020 con 0x7f3cd8102670 2024-04-26T17:53:02.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.270+0000 7f3cd57fa700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3cc00a3c40 0x7f3cc00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:02.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.270+0000 7f3cd57fa700 1 -- 172.21.15.31:0/2442369297 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7497+0+0 (secure 0 0 0) 0x7f3cc80cfe50 con 0x7f3cd8102670 2024-04-26T17:53:02.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.271+0000 7f3cd77fe700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3cc00a3c40 0x7f3cc00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:02.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.271+0000 7f3cd77fe700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3cc00a3c40 0x7f3cc00a6100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc005950 tx=0x7f3ccc0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:02.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.274+0000 7f3cd57fa700 1 -- 172.21.15.31:0/2442369297 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3cc801f020 con 0x7f3cd8102670 2024-04-26T17:53:02.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.483+0000 7f3cde9c6700 1 -- 172.21.15.31:0/2442369297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f3cd819aea0 con 0x7f3cd8102670 2024-04-26T17:53:02.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.590+0000 7f3cd57fa700 1 -- 172.21.15.31:0/2442369297 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v85) ==== 170+0+0 (secure 0 0 0) 0x7f3cc809cb60 con 0x7f3cd8102670 2024-04-26T17:53:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.593+0000 7f3cde9c6700 1 -- 172.21.15.31:0/2442369297 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3cc00a3c40 msgr2=0x7f3cc00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.593+0000 7f3cde9c6700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3cc00a3c40 0x7f3cc00a6100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc005950 tx=0x7f3ccc0058e0 comp rx=0 tx=0).stop 2024-04-26T17:53:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.593+0000 7f3cde9c6700 1 -- 172.21.15.31:0/2442369297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cd8102670 msgr2=0x7f3cd819f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.593+0000 7f3cde9c6700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cd8102670 0x7f3cd819f7b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3cc800b530 tx=0x7f3cc800b8f0 comp rx=0 tx=0).stop 2024-04-26T17:53:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.593+0000 7f3cde9c6700 1 -- 172.21.15.31:0/2442369297 shutdown_connections 2024-04-26T17:53:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.593+0000 7f3cde9c6700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3cc00a3c40 0x7f3cc00a6100 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.593+0000 7f3cde9c6700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cd81073a0 0x7f3cd8199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.593+0000 7f3cde9c6700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cd8102670 0x7f3cd819f7b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:02.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.593+0000 7f3cde9c6700 1 --2- 172.21.15.31:0/2442369297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3cd8103030 0x7f3cd819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:02.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.593+0000 7f3cde9c6700 1 -- 172.21.15.31:0/2442369297 >> 172.21.15.31:0/2442369297 conn(0x7f3cd8078190 msgr2=0x7f3cd806be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:02.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.593+0000 7f3cde9c6700 1 -- 172.21.15.31:0/2442369297 shutdown_connections 2024-04-26T17:53:02.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:02.593+0000 7f3cde9c6700 1 -- 172.21.15.31:0/2442369297 wait complete. 2024-04-26T17:53:02.594 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-04-26T17:53:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:02 smithi097 ceph-mon[32419]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T17:53:02.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:02 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2442369297' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T17:53:02.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:02 smithi031 ceph-mon[29164]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T17:53:02.908 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:02 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2442369297' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T17:53:03.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:02 smithi186 ceph-mon[32797]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T17:53:03.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:02 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2442369297' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T17:53:03.250 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T17:53:03.503 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:03.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:03 smithi097 ceph-mon[32419]: pgmap v291: 33 pgs: 1 creating+activating, 8 creating+peering, 20 active+clean, 4 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:53:03.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:03 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2442369297' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T17:53:03.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:03 smithi097 ceph-mon[32419]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T17:53:03.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:03 smithi031 ceph-mon[29164]: pgmap v291: 33 pgs: 1 creating+activating, 8 creating+peering, 20 active+clean, 4 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:53:03.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:03 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2442369297' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T17:53:03.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:03 smithi031 ceph-mon[29164]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T17:53:04.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:03 smithi186 ceph-mon[32797]: pgmap v291: 33 pgs: 1 creating+activating, 8 creating+peering, 20 active+clean, 4 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:53:04.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:03 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2442369297' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T17:53:04.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:03 smithi186 ceph-mon[32797]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T17:53:04.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:04 smithi097 ceph-mon[32419]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T17:53:04.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:04 smithi097 ceph-mon[32419]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T17:53:04.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:04 smithi031 ceph-mon[29164]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T17:53:04.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:04 smithi031 ceph-mon[29164]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T17:53:05.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:04 smithi186 ceph-mon[32797]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T17:53:05.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:04 smithi186 ceph-mon[32797]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T17:53:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.157+0000 7f9f64a5a700 1 -- 172.21.15.31:0/3938644013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f60102320 msgr2=0x7f9f601027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.157+0000 7f9f64a5a700 1 --2- 172.21.15.31:0/3938644013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f60102320 0x7f9f601027a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009990 tx=0x7f9f54009ca0 comp rx=0 tx=0).stop 2024-04-26T17:53:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.157+0000 7f9f64a5a700 1 -- 172.21.15.31:0/3938644013 shutdown_connections 2024-04-26T17:53:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.157+0000 7f9f64a5a700 1 --2- 172.21.15.31:0/3938644013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f60108390 0x7f9f60108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.157+0000 7f9f64a5a700 1 --2- 172.21.15.31:0/3938644013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f60102ce0 0x7f9f6010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.157+0000 7f9f64a5a700 1 --2- 172.21.15.31:0/3938644013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f60102320 0x7f9f601027a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.157+0000 7f9f64a5a700 1 -- 172.21.15.31:0/3938644013 >> 172.21.15.31:0/3938644013 conn(0x7f9f600792d0 msgr2=0x7f9f600796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.157+0000 7f9f64a5a700 1 -- 172.21.15.31:0/3938644013 shutdown_connections 2024-04-26T17:53:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.157+0000 7f9f64a5a700 1 -- 172.21.15.31:0/3938644013 wait complete. 2024-04-26T17:53:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.158+0000 7f9f64a5a700 1 Processor -- start 2024-04-26T17:53:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.158+0000 7f9f64a5a700 1 -- start start 2024-04-26T17:53:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.158+0000 7f9f64a5a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f60102320 0x7f9f6019bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.158+0000 7f9f64a5a700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f60102ce0 0x7f9f6019c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.158+0000 7f9f64a5a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f60108390 0x7f9f601a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.158+0000 7f9f64a5a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9f601124e0 con 0x7f9f60108390 2024-04-26T17:53:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.158+0000 7f9f64a5a700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f9f60112360 con 0x7f9f60102ce0 2024-04-26T17:53:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.158+0000 7f9f64a5a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9f60112660 con 0x7f9f60102320 2024-04-26T17:53:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.158+0000 7f9f5ed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f60108390 0x7f9f601a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.158+0000 7f9f5ed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f60108390 0x7f9f601a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41500/0 (socket says 172.21.15.31:41500) 2024-04-26T17:53:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.158+0000 7f9f5ed9d700 1 -- 172.21.15.31:0/2352676369 learned_addr learned my addr 172.21.15.31:0/2352676369 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.159+0000 7f9f5e59c700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f60102320 0x7f9f6019bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.159+0000 7f9f5dd9b700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f60102ce0 0x7f9f6019c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.159+0000 7f9f5ed9d700 1 -- 172.21.15.31:0/2352676369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f60102320 msgr2=0x7f9f6019bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.159+0000 7f9f5ed9d700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f60102320 0x7f9f6019bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.159+0000 7f9f5ed9d700 1 -- 172.21.15.31:0/2352676369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f60102ce0 msgr2=0x7f9f6019c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.159+0000 7f9f5ed9d700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f60102ce0 0x7f9f6019c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.159+0000 7f9f5ed9d700 1 -- 172.21.15.31:0/2352676369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f601a0d80 con 0x7f9f60108390 2024-04-26T17:53:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.159+0000 7f9f5dd9b700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f60102ce0 0x7f9f6019c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:53:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.159+0000 7f9f5ed9d700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f60108390 0x7f9f601a0640 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9f58007ee0 tx=0x7f9f5800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.159+0000 7f9f4f7fe700 1 -- 172.21.15.31:0/2352676369 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5800dbd0 con 0x7f9f60108390 2024-04-26T17:53:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.159+0000 7f9f4f7fe700 1 -- 172.21.15.31:0/2352676369 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f5800dd70 con 0x7f9f60108390 2024-04-26T17:53:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.159+0000 7f9f64a5a700 1 -- 172.21.15.31:0/2352676369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f601a1070 con 0x7f9f60108390 2024-04-26T17:53:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.159+0000 7f9f4f7fe700 1 -- 172.21.15.31:0/2352676369 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58005990 con 0x7f9f60108390 2024-04-26T17:53:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.159+0000 7f9f64a5a700 1 -- 172.21.15.31:0/2352676369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f601a88f0 con 0x7f9f60108390 2024-04-26T17:53:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.160+0000 7f9f64a5a700 1 -- 172.21.15.31:0/2352676369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f6006a110 con 0x7f9f60108390 2024-04-26T17:53:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.161+0000 7f9f4f7fe700 1 -- 172.21.15.31:0/2352676369 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9f5801a020 con 0x7f9f60108390 2024-04-26T17:53:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.162+0000 7f9f4f7fe700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9f480a3c40 0x7f9f480a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.162+0000 7f9f4f7fe700 1 -- 172.21.15.31:0/2352676369 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7879+0+0 (secure 0 0 0) 0x7f9f580d0090 con 0x7f9f60108390 2024-04-26T17:53:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.162+0000 7f9f5e59c700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9f480a3c40 0x7f9f480a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.162+0000 7f9f5e59c700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9f480a3c40 0x7f9f480a6100 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9f50009bd0 tx=0x7f9f50009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.166+0000 7f9f4f7fe700 1 -- 172.21.15.31:0/2352676369 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9f58010070 con 0x7f9f60108390 2024-04-26T17:53:05.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.424+0000 7f9f64a5a700 1 -- 172.21.15.31:0/2352676369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f9f601a8b30 con 0x7f9f60108390 2024-04-26T17:53:05.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.610+0000 7f9f4f7fe700 1 -- 172.21.15.31:0/2352676369 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f9f5809cc20 con 0x7f9f60108390 2024-04-26T17:53:05.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.613+0000 7f9f64a5a700 1 -- 172.21.15.31:0/2352676369 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9f480a3c40 msgr2=0x7f9f480a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:05.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.613+0000 7f9f64a5a700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9f480a3c40 0x7f9f480a6100 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9f50009bd0 tx=0x7f9f50009260 comp rx=0 tx=0).stop 2024-04-26T17:53:05.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.613+0000 7f9f64a5a700 1 -- 172.21.15.31:0/2352676369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f60108390 msgr2=0x7f9f601a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:05.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.613+0000 7f9f64a5a700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f60108390 0x7f9f601a0640 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9f58007ee0 tx=0x7f9f5800d3b0 comp rx=0 tx=0).stop 2024-04-26T17:53:05.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.613+0000 7f9f64a5a700 1 -- 172.21.15.31:0/2352676369 shutdown_connections 2024-04-26T17:53:05.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.613+0000 7f9f64a5a700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9f480a3c40 0x7f9f480a6100 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:05.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.613+0000 7f9f64a5a700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f60108390 0x7f9f601a0640 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:05.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.613+0000 7f9f64a5a700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f60102320 0x7f9f6019bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:05.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.613+0000 7f9f64a5a700 1 --2- 172.21.15.31:0/2352676369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f60102ce0 0x7f9f6019c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:05.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.613+0000 7f9f64a5a700 1 -- 172.21.15.31:0/2352676369 >> 172.21.15.31:0/2352676369 conn(0x7f9f600792d0 msgr2=0x7f9f6006b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:05.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.613+0000 7f9f64a5a700 1 -- 172.21.15.31:0/2352676369 shutdown_connections 2024-04-26T17:53:05.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:05.613+0000 7f9f64a5a700 1 -- 172.21.15.31:0/2352676369 wait complete. 2024-04-26T17:53:05.614 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 2 and data pool 3 2024-04-26T17:53:05.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:05 smithi031 ceph-mon[29164]: pgmap v294: 97 pgs: 1 creating+activating, 20 creating+peering, 32 active+clean, 44 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:53:05.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:05 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2352676369' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T17:53:05.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:05 smithi031 ceph-mon[29164]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T17:53:05.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:05 smithi097 ceph-mon[32419]: pgmap v294: 97 pgs: 1 creating+activating, 20 creating+peering, 32 active+clean, 44 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:53:05.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:05 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2352676369' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T17:53:05.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:05 smithi097 ceph-mon[32419]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T17:53:06.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:05 smithi186 ceph-mon[32797]: pgmap v294: 97 pgs: 1 creating+activating, 20 creating+peering, 32 active+clean, 44 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:53:06.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:05 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2352676369' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T17:53:06.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:05 smithi186 ceph-mon[32797]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T17:53:06.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T17:53:06.606 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:06.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:06 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2352676369' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T17:53:06.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:06 smithi031 ceph-mon[29164]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T17:53:06.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:06 smithi031 ceph-mon[29164]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T17:53:06.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:06 smithi031 ceph-mon[29164]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T17:53:06.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:06 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2352676369' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T17:53:06.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:06 smithi097 ceph-mon[32419]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T17:53:06.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:06 smithi097 ceph-mon[32419]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T17:53:06.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:06 smithi097 ceph-mon[32419]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T17:53:07.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:06 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2352676369' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T17:53:07.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:06 smithi186 ceph-mon[32797]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T17:53:07.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:06 smithi186 ceph-mon[32797]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T17:53:07.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:06 smithi186 ceph-mon[32797]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T17:53:07.751 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:07 smithi031 ceph-mon[29164]: pgmap v297: 97 pgs: 12 creating+peering, 60 active+clean, 25 unknown; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-04-26T17:53:07.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:07 smithi031 ceph-mon[29164]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T17:53:07.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:07 smithi031 ceph-mon[29164]: Cluster is now healthy 2024-04-26T17:53:07.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:07 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] up:active 2024-04-26T17:53:07.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:07 smithi031 ceph-mon[29164]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T17:53:07.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:07 smithi031 ceph-mon[29164]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T17:53:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:07 smithi097 ceph-mon[32419]: pgmap v297: 97 pgs: 12 creating+peering, 60 active+clean, 25 unknown; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-04-26T17:53:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:07 smithi097 ceph-mon[32419]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T17:53:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:07 smithi097 ceph-mon[32419]: Cluster is now healthy 2024-04-26T17:53:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:07 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] up:active 2024-04-26T17:53:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:07 smithi097 ceph-mon[32419]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T17:53:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:07 smithi097 ceph-mon[32419]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T17:53:08.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:07 smithi186 ceph-mon[32797]: pgmap v297: 97 pgs: 12 creating+peering, 60 active+clean, 25 unknown; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-04-26T17:53:08.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:07 smithi186 ceph-mon[32797]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T17:53:08.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:07 smithi186 ceph-mon[32797]: Cluster is now healthy 2024-04-26T17:53:08.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:07 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] up:active 2024-04-26T17:53:08.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:07 smithi186 ceph-mon[32797]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T17:53:08.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:07 smithi186 ceph-mon[32797]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T17:53:08.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.316+0000 7fe98b025700 1 -- 172.21.15.31:0/3537904495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe984102320 msgr2=0x7fe9841027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:08.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.316+0000 7fe98b025700 1 --2- 172.21.15.31:0/3537904495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe984102320 0x7fe9841027a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00b0a0 tx=0x7fe97c00b3b0 comp rx=0 tx=0).stop 2024-04-26T17:53:08.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.316+0000 7fe98b025700 1 -- 172.21.15.31:0/3537904495 shutdown_connections 2024-04-26T17:53:08.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.316+0000 7fe98b025700 1 --2- 172.21.15.31:0/3537904495 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe984102ce0 0x7fe98410f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:08.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.316+0000 7fe98b025700 1 --2- 172.21.15.31:0/3537904495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe984108390 0x7fe984108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:08.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.316+0000 7fe98b025700 1 --2- 172.21.15.31:0/3537904495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe984102320 0x7fe9841027a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:08.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.316+0000 7fe98b025700 1 -- 172.21.15.31:0/3537904495 >> 172.21.15.31:0/3537904495 conn(0x7fe9840792d0 msgr2=0x7fe9840796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:08.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.316+0000 7fe98b025700 1 -- 172.21.15.31:0/3537904495 shutdown_connections 2024-04-26T17:53:08.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.316+0000 7fe98b025700 1 -- 172.21.15.31:0/3537904495 wait complete. 2024-04-26T17:53:08.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe98b025700 1 Processor -- start 2024-04-26T17:53:08.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe98b025700 1 -- start start 2024-04-26T17:53:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe98b025700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe984102320 0x7fe984104000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe98b025700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe984102ce0 0x7fe984104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe98b025700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe984108390 0x7fe984104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe98b025700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe9841125b0 con 0x7fe984108390 2024-04-26T17:53:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe98b025700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe984112430 con 0x7fe984102ce0 2024-04-26T17:53:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe98b025700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe984112730 con 0x7fe984102320 2024-04-26T17:53:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe9895c2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe984108390 0x7fe984104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe9895c2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe984108390 0x7fe984104a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41532/0 (socket says 172.21.15.31:41532) 2024-04-26T17:53:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe9895c2700 1 -- 172.21.15.31:0/1928819793 learned_addr learned my addr 172.21.15.31:0/1928819793 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe988dc1700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe984102320 0x7fe984104000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe983fff700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe984102ce0 0x7fe984104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe9895c2700 1 -- 172.21.15.31:0/1928819793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe984102320 msgr2=0x7fe984104000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe9895c2700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe984102320 0x7fe984104000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:08.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe9895c2700 1 -- 172.21.15.31:0/1928819793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe984102ce0 msgr2=0x7fe984104540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:08.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe9895c2700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe984102ce0 0x7fe984104540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:08.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe9895c2700 1 -- 172.21.15.31:0/1928819793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9841a81a0 con 0x7fe984108390 2024-04-26T17:53:08.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.317+0000 7fe988dc1700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe984102320 0x7fe984104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:53:08.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.318+0000 7fe9895c2700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe984108390 0x7fe984104a80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe97400b550 tx=0x7fe97400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:08.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.318+0000 7fe981ffb700 1 -- 172.21.15.31:0/1928819793 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe974015020 con 0x7fe984108390 2024-04-26T17:53:08.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.318+0000 7fe981ffb700 1 -- 172.21.15.31:0/1928819793 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe974011ce0 con 0x7fe984108390 2024-04-26T17:53:08.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.318+0000 7fe98b025700 1 -- 172.21.15.31:0/1928819793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9841a8490 con 0x7fe984108390 2024-04-26T17:53:08.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.318+0000 7fe981ffb700 1 -- 172.21.15.31:0/1928819793 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe974010dd0 con 0x7fe984108390 2024-04-26T17:53:08.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.318+0000 7fe98b025700 1 -- 172.21.15.31:0/1928819793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9841a89d0 con 0x7fe984108390 2024-04-26T17:53:08.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.319+0000 7fe98b025700 1 -- 172.21.15.31:0/1928819793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe98406a110 con 0x7fe984108390 2024-04-26T17:53:08.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.320+0000 7fe981ffb700 1 -- 172.21.15.31:0/1928819793 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe974018020 con 0x7fe984108390 2024-04-26T17:53:08.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.321+0000 7fe981ffb700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe9700a3cc0 0x7fe9700a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:08.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.321+0000 7fe988dc1700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe9700a3cc0 0x7fe9700a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:08.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.321+0000 7fe981ffb700 1 -- 172.21.15.31:0/1928819793 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fe9740cfa20 con 0x7fe984108390 2024-04-26T17:53:08.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.321+0000 7fe988dc1700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe9700a3cc0 0x7fe9700a6180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe96c0099d0 tx=0x7fe96c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:08.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.325+0000 7fe981ffb700 1 -- 172.21.15.31:0/1928819793 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe97401f020 con 0x7fe984108390 2024-04-26T17:53:08.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.538+0000 7fe98b025700 1 -- 172.21.15.31:0/1928819793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe9841a8c90 con 0x7fe984108390 2024-04-26T17:53:08.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.540+0000 7fe981ffb700 1 -- 172.21.15.31:0/1928819793 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22190 (secure 0 0 0) 0x7fe97409c4b0 con 0x7fe984108390 2024-04-26T17:53:08.540 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:53:08.540 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":88,"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","created":"2024-04-26T17:44:48.618250+0000","modified":"2024-04-26T17:53:05.425953+0000","last_up_change":"2024-04-26T17:52:00.798886+0000","last_in_change":"2024-04-26T17:51:40.613593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T17:48:12.307250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T17:52:58.650338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T17:53:02.488144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f684ee5-eeee-454a-a00b-e5d6d2901572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6803","nonce":1957398328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6805","nonce":1957398328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6809","nonce":1957398328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6807","nonce":1957398328}]},"public_addr":"172.21.15.31:6803/1957398328","cluster_addr":"172.21.15.31:6805/1957398328","heartbeat_back_addr":"172.21.15.31:6809/1957398328","heartbeat_front_addr":"172.21.15.31:6807/1957398328","state":["exists","up"]},{"osd":1,"uuid":"08be6b26-cb60-41f7-be8b-01a0c93ebafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6801","nonce":4209866427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6803","nonce":4209866427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6807","nonce":4209866427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6805","nonce":4209866427}]},"public_addr":"172.21.15.97:6801/4209866427","cluster_addr":"172.21.15.97:6803/4209866427","heartbeat_back_addr":"172.21.15.97:6807/4209866427","heartbeat_front_addr":"172.21.15.97:6805/4209866427","state":["exists","up"]},{"osd":2,"uuid":"498216b1-a0ac-4923-ade8-ce50fb97d472","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6801","nonce":223535147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6803","nonce":223535147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6807","nonce":223535147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6805","nonce":223535147}]},"public_addr":"172.21.15.186:6801/223535147","cluster_addr":"172.21.15.186:6803/223535147","heartbeat_back_addr":"172.21.15.186:6807/223535147","heartbeat_front_addr":"172.21.15.186:6805/223535147","state":["exists","up"]},{"osd":3,"uuid":"568a352d-a430-4f0f-9a49-6f992e0ceae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6811","nonce":2640112341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6813","nonce":2640112341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6817","nonce":2640112341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6815","nonce":2640112341}]},"public_addr":"172.21.15.31:6811/2640112341","cluster_addr":"172.21.15.31:6813/2640112341","heartbeat_back_addr":"172.21.15.31:6817/2640112341","heartbeat_front_addr":"172.21.15.31:6815/2640112341","state":["exists","up"]},{"osd":4,"uuid":"112ccbf7-24e8-473c-9852-2e7167dd3108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6809","nonce":1359263143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6811","nonce":1359263143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6815","nonce":1359263143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6813","nonce":1359263143}]},"public_addr":"172.21.15.97:6809/1359263143","cluster_addr":"172.21.15.97:6811/1359263143","heartbeat_back_addr":"172.21.15.97:6815/1359263143","heartbeat_front_addr":"172.21.15.97:6813/1359263143","state":["exists","up"]},{"osd":5,"uuid":"eb17759e-1ad8-4bda-9f61-54d2a9781535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6809","nonce":2006134626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6811","nonce":2006134626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6815","nonce":2006134626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6813","nonce":2006134626}]},"public_addr":"172.21.15.186:6809/2006134626","cluster_addr":"172.21.15.186:6811/2006134626","heartbeat_back_addr":"172.21.15.186:6815/2006134626","heartbeat_front_addr":"172.21.15.186:6813/2006134626","state":["exists","up"]},{"osd":6,"uuid":"f1d11e38-40e9-4db4-868f-145f1ffc0d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6819","nonce":2955759395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6821","nonce":2955759395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6825","nonce":2955759395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6823","nonce":2955759395}]},"public_addr":"172.21.15.31:6819/2955759395","cluster_addr":"172.21.15.31:6821/2955759395","heartbeat_back_addr":"172.21.15.31:6825/2955759395","heartbeat_front_addr":"172.21.15.31:6823/2955759395","state":["exists","up"]},{"osd":7,"uuid":"4e95f728-23ff-4bca-9b94-afde2662a5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6817","nonce":3940083075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6819","nonce":3940083075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6823","nonce":3940083075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6821","nonce":3940083075}]},"public_addr":"172.21.15.97:6817/3940083075","cluster_addr":"172.21.15.97:6819/3940083075","heartbeat_back_addr":"172.21.15.97:6823/3940083075","heartbeat_front_addr":"172.21.15.97:6821/3940083075","state":["exists","up"]},{"osd":8,"uuid":"5f91c915-2568-4d20-b645-00dad2efe952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6817","nonce":3193558170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6819","nonce":3193558170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6823","nonce":3193558170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6821","nonce":3193558170}]},"public_addr":"172.21.15.186:6817/3193558170","cluster_addr":"172.21.15.186:6819/3193558170","heartbeat_back_addr":"172.21.15.186:6823/3193558170","heartbeat_front_addr":"172.21.15.186:6821/3193558170","state":["exists","up"]},{"osd":9,"uuid":"12fda87d-830a-461f-a6cb-4699c52dc5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6827","nonce":3660801905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6829","nonce":3660801905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6833","nonce":3660801905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6831","nonce":3660801905}]},"public_addr":"172.21.15.31:6827/3660801905","cluster_addr":"172.21.15.31:6829/3660801905","heartbeat_back_addr":"172.21.15.31:6833/3660801905","heartbeat_front_addr":"172.21.15.31:6831/3660801905","state":["exists","up"]},{"osd":10,"uuid":"92a639b3-918b-485c-bd30-258b4224993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6825","nonce":4097344903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6827","nonce":4097344903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6831","nonce":4097344903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6829","nonce":4097344903}]},"public_addr":"172.21.15.97:6825/4097344903","cluster_addr":"172.21.15.97:6827/4097344903","heartbeat_back_addr":"172.21.15.97:6831/4097344903","heartbeat_front_addr":"172.21.15.97:6829/4097344903","state":["exists","up"]},{"osd":11,"uuid":"842aa599-d21d-4e4a-b872-e9ac20d8c0ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6825","nonce":1063973352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6827","nonce":1063973352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6831","nonce":1063973352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6829","nonce":1063973352}]},"public_addr":"172.21.15.186:6825/1063973352","cluster_addr":"172.21.15.186:6827/1063973352","heartbeat_back_addr":"172.21.15.186:6831/1063973352","heartbeat_front_addr":"172.21.15.186:6829/1063973352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:21.305331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:44.877932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:08.784665+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:34.381762+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:58.999358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:23.620237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:49.961719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:15.363702+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:40.599703+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:07.391645+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:32.800851+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:59.384148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/718966776":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/2035551671":"2024-04-27T17:45:48.218816+0000","172.21.15.31:6801/153341213":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/810461143":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/3148082955":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6801/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/288292084":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/3101596726":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/153341213":"2024-04-27T17:45:48.218816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T17:53:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.542+0000 7fe98b025700 1 -- 172.21.15.31:0/1928819793 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe9700a3cc0 msgr2=0x7fe9700a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.542+0000 7fe98b025700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe9700a3cc0 0x7fe9700a6180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe96c0099d0 tx=0x7fe96c008040 comp rx=0 tx=0).stop 2024-04-26T17:53:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.542+0000 7fe98b025700 1 -- 172.21.15.31:0/1928819793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe984108390 msgr2=0x7fe984104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.542+0000 7fe98b025700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe984108390 0x7fe984104a80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe97400b550 tx=0x7fe97400b860 comp rx=0 tx=0).stop 2024-04-26T17:53:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.542+0000 7fe98b025700 1 -- 172.21.15.31:0/1928819793 shutdown_connections 2024-04-26T17:53:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.542+0000 7fe98b025700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe9700a3cc0 0x7fe9700a6180 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.542+0000 7fe98b025700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe984108390 0x7fe984104a80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.542+0000 7fe98b025700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe984102320 0x7fe984104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.542+0000 7fe98b025700 1 --2- 172.21.15.31:0/1928819793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe984102ce0 0x7fe984104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.542+0000 7fe98b025700 1 -- 172.21.15.31:0/1928819793 >> 172.21.15.31:0/1928819793 conn(0x7fe9840792d0 msgr2=0x7fe984101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.542+0000 7fe98b025700 1 -- 172.21.15.31:0/1928819793 shutdown_connections 2024-04-26T17:53:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:08.542+0000 7fe98b025700 1 -- 172.21.15.31:0/1928819793 wait complete. 2024-04-26T17:53:08.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:08 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1928819793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:53:08.910 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:08 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1928819793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:53:09.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:08 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1928819793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:53:09.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T17:53:09.620 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:09.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:09 smithi097 ceph-mon[32419]: pgmap v298: 97 pgs: 97 active+clean; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-04-26T17:53:09.972 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:09 smithi031 ceph-mon[29164]: pgmap v298: 97 pgs: 97 active+clean; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-04-26T17:53:10.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:09 smithi186 ceph-mon[32797]: pgmap v298: 97 pgs: 97 active+clean; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-04-26T17:53:11.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.567+0000 7f2145bef700 1 -- 172.21.15.31:0/2794098999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2140108390 msgr2=0x7f2140108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:11.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.567+0000 7f2145bef700 1 --2- 172.21.15.31:0/2794098999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2140108390 0x7f2140108770 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2130009990 tx=0x7f2130009ca0 comp rx=0 tx=0).stop 2024-04-26T17:53:11.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.567+0000 7f2145bef700 1 -- 172.21.15.31:0/2794098999 shutdown_connections 2024-04-26T17:53:11.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.567+0000 7f2145bef700 1 --2- 172.21.15.31:0/2794098999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2140102ce0 0x7f214010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:11.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.567+0000 7f2145bef700 1 --2- 172.21.15.31:0/2794098999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2140108390 0x7f2140108770 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:11.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.567+0000 7f2145bef700 1 --2- 172.21.15.31:0/2794098999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2140102320 0x7f21401027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:11.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.567+0000 7f2145bef700 1 -- 172.21.15.31:0/2794098999 >> 172.21.15.31:0/2794098999 conn(0x7f21400792d0 msgr2=0x7f21400796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:11.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.567+0000 7f2145bef700 1 -- 172.21.15.31:0/2794098999 shutdown_connections 2024-04-26T17:53:11.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.567+0000 7f2145bef700 1 -- 172.21.15.31:0/2794098999 wait complete. 2024-04-26T17:53:11.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.567+0000 7f2145bef700 1 Processor -- start 2024-04-26T17:53:11.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f2145bef700 1 -- start start 2024-04-26T17:53:11.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f2145bef700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2140102320 0x7f214019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:11.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f2145bef700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2140102ce0 0x7f214019ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:11.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f2145bef700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2140108390 0x7f21401a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:11.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f2145bef700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2140112550 con 0x7f2140102320 2024-04-26T17:53:11.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f2145bef700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f21401123d0 con 0x7f2140102ce0 2024-04-26T17:53:11.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f2145bef700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f21401126d0 con 0x7f2140108390 2024-04-26T17:53:11.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f213f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2140102320 0x7f214019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:11.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f213f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2140102320 0x7f214019a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41538/0 (socket says 172.21.15.31:41538) 2024-04-26T17:53:11.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f213f7fe700 1 -- 172.21.15.31:0/2303297083 learned_addr learned my addr 172.21.15.31:0/2303297083 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:11.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f213effd700 1 --2- 172.21.15.31:0/2303297083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2140102ce0 0x7f214019ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:11.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f213f7fe700 1 -- 172.21.15.31:0/2303297083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2140108390 msgr2=0x7f21401a59d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T17:53:11.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f213f7fe700 1 --2- 172.21.15.31:0/2303297083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2140108390 0x7f21401a59d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:11.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f213f7fe700 1 -- 172.21.15.31:0/2303297083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2140102ce0 msgr2=0x7f214019ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:11.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f213f7fe700 1 --2- 172.21.15.31:0/2303297083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2140102ce0 0x7f214019ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:11.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f213f7fe700 1 -- 172.21.15.31:0/2303297083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21401a5f10 con 0x7f2140102320 2024-04-26T17:53:11.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.568+0000 7f213f7fe700 1 --2- 172.21.15.31:0/2303297083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2140102320 0x7f214019a950 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2130009f70 tx=0x7f2130005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:11.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.569+0000 7f213cff9700 1 -- 172.21.15.31:0/2303297083 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2130027070 con 0x7f2140102320 2024-04-26T17:53:11.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.569+0000 7f213cff9700 1 -- 172.21.15.31:0/2303297083 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21300040b0 con 0x7f2140102320 2024-04-26T17:53:11.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.569+0000 7f2145bef700 1 -- 172.21.15.31:0/2303297083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21401a60e0 con 0x7f2140102320 2024-04-26T17:53:11.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.569+0000 7f213cff9700 1 -- 172.21.15.31:0/2303297083 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2130022e40 con 0x7f2140102320 2024-04-26T17:53:11.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.569+0000 7f2145bef700 1 -- 172.21.15.31:0/2303297083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21401a6530 con 0x7f2140102320 2024-04-26T17:53:11.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.570+0000 7f2145bef700 1 -- 172.21.15.31:0/2303297083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f214006a110 con 0x7f2140102320 2024-04-26T17:53:11.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.571+0000 7f213cff9700 1 -- 172.21.15.31:0/2303297083 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f213002a070 con 0x7f2140102320 2024-04-26T17:53:11.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.572+0000 7f213cff9700 1 --2- 172.21.15.31:0/2303297083 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f21280a3c40 0x7f21280a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:11.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.572+0000 7f213effd700 1 --2- 172.21.15.31:0/2303297083 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f21280a3c40 0x7f21280a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:11.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.572+0000 7f213cff9700 1 -- 172.21.15.31:0/2303297083 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f21300d1540 con 0x7f2140102320 2024-04-26T17:53:11.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.572+0000 7f213effd700 1 --2- 172.21.15.31:0/2303297083 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f21280a3c40 0x7f21280a6100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2134005ea0 tx=0x7f2134005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:11.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.576+0000 7f213cff9700 1 -- 172.21.15.31:0/2303297083 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f213005eff0 con 0x7f2140102320 2024-04-26T17:53:11.634 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 17:53:11 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T17:53:11.138+0000 7f83a0ee8700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T17:53:11.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.790+0000 7f2145bef700 1 -- 172.21.15.31:0/2303297083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f214002e650 con 0x7f2140102320 2024-04-26T17:53:11.790 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:11 smithi031 ceph-mon[29164]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-26T17:53:11.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.791+0000 7f213cff9700 1 -- 172.21.15.31:0/2303297083 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22190 (secure 0 0 0) 0x7f2130013070 con 0x7f2140102320 2024-04-26T17:53:11.792 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:53:11.792 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":88,"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","created":"2024-04-26T17:44:48.618250+0000","modified":"2024-04-26T17:53:05.425953+0000","last_up_change":"2024-04-26T17:52:00.798886+0000","last_in_change":"2024-04-26T17:51:40.613593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T17:48:12.307250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T17:52:58.650338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T17:53:02.488144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f684ee5-eeee-454a-a00b-e5d6d2901572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6803","nonce":1957398328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6805","nonce":1957398328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6809","nonce":1957398328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6807","nonce":1957398328}]},"public_addr":"172.21.15.31:6803/1957398328","cluster_addr":"172.21.15.31:6805/1957398328","heartbeat_back_addr":"172.21.15.31:6809/1957398328","heartbeat_front_addr":"172.21.15.31:6807/1957398328","state":["exists","up"]},{"osd":1,"uuid":"08be6b26-cb60-41f7-be8b-01a0c93ebafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6801","nonce":4209866427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6803","nonce":4209866427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6807","nonce":4209866427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6805","nonce":4209866427}]},"public_addr":"172.21.15.97:6801/4209866427","cluster_addr":"172.21.15.97:6803/4209866427","heartbeat_back_addr":"172.21.15.97:6807/4209866427","heartbeat_front_addr":"172.21.15.97:6805/4209866427","state":["exists","up"]},{"osd":2,"uuid":"498216b1-a0ac-4923-ade8-ce50fb97d472","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6801","nonce":223535147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6803","nonce":223535147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6807","nonce":223535147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6805","nonce":223535147}]},"public_addr":"172.21.15.186:6801/223535147","cluster_addr":"172.21.15.186:6803/223535147","heartbeat_back_addr":"172.21.15.186:6807/223535147","heartbeat_front_addr":"172.21.15.186:6805/223535147","state":["exists","up"]},{"osd":3,"uuid":"568a352d-a430-4f0f-9a49-6f992e0ceae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6811","nonce":2640112341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6813","nonce":2640112341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6817","nonce":2640112341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6815","nonce":2640112341}]},"public_addr":"172.21.15.31:6811/2640112341","cluster_addr":"172.21.15.31:6813/2640112341","heartbeat_back_addr":"172.21.15.31:6817/2640112341","heartbeat_front_addr":"172.21.15.31:6815/2640112341","state":["exists","up"]},{"osd":4,"uuid":"112ccbf7-24e8-473c-9852-2e7167dd3108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6809","nonce":1359263143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6811","nonce":1359263143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6815","nonce":1359263143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6813","nonce":1359263143}]},"public_addr":"172.21.15.97:6809/1359263143","cluster_addr":"172.21.15.97:6811/1359263143","heartbeat_back_addr":"172.21.15.97:6815/1359263143","heartbeat_front_addr":"172.21.15.97:6813/1359263143","state":["exists","up"]},{"osd":5,"uuid":"eb17759e-1ad8-4bda-9f61-54d2a9781535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6809","nonce":2006134626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6811","nonce":2006134626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6815","nonce":2006134626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6813","nonce":2006134626}]},"public_addr":"172.21.15.186:6809/2006134626","cluster_addr":"172.21.15.186:6811/2006134626","heartbeat_back_addr":"172.21.15.186:6815/2006134626","heartbeat_front_addr":"172.21.15.186:6813/2006134626","state":["exists","up"]},{"osd":6,"uuid":"f1d11e38-40e9-4db4-868f-145f1ffc0d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6819","nonce":2955759395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6821","nonce":2955759395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6825","nonce":2955759395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6823","nonce":2955759395}]},"public_addr":"172.21.15.31:6819/2955759395","cluster_addr":"172.21.15.31:6821/2955759395","heartbeat_back_addr":"172.21.15.31:6825/2955759395","heartbeat_front_addr":"172.21.15.31:6823/2955759395","state":["exists","up"]},{"osd":7,"uuid":"4e95f728-23ff-4bca-9b94-afde2662a5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6817","nonce":3940083075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6819","nonce":3940083075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6823","nonce":3940083075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6821","nonce":3940083075}]},"public_addr":"172.21.15.97:6817/3940083075","cluster_addr":"172.21.15.97:6819/3940083075","heartbeat_back_addr":"172.21.15.97:6823/3940083075","heartbeat_front_addr":"172.21.15.97:6821/3940083075","state":["exists","up"]},{"osd":8,"uuid":"5f91c915-2568-4d20-b645-00dad2efe952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6817","nonce":3193558170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6819","nonce":3193558170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6823","nonce":3193558170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6821","nonce":3193558170}]},"public_addr":"172.21.15.186:6817/3193558170","cluster_addr":"172.21.15.186:6819/3193558170","heartbeat_back_addr":"172.21.15.186:6823/3193558170","heartbeat_front_addr":"172.21.15.186:6821/3193558170","state":["exists","up"]},{"osd":9,"uuid":"12fda87d-830a-461f-a6cb-4699c52dc5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6827","nonce":3660801905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6829","nonce":3660801905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6833","nonce":3660801905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6831","nonce":3660801905}]},"public_addr":"172.21.15.31:6827/3660801905","cluster_addr":"172.21.15.31:6829/3660801905","heartbeat_back_addr":"172.21.15.31:6833/3660801905","heartbeat_front_addr":"172.21.15.31:6831/3660801905","state":["exists","up"]},{"osd":10,"uuid":"92a639b3-918b-485c-bd30-258b4224993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6825","nonce":4097344903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6827","nonce":4097344903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6831","nonce":4097344903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6829","nonce":4097344903}]},"public_addr":"172.21.15.97:6825/4097344903","cluster_addr":"172.21.15.97:6827/4097344903","heartbeat_back_addr":"172.21.15.97:6831/4097344903","heartbeat_front_addr":"172.21.15.97:6829/4097344903","state":["exists","up"]},{"osd":11,"uuid":"842aa599-d21d-4e4a-b872-e9ac20d8c0ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6825","nonce":1063973352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6827","nonce":1063973352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6831","nonce":1063973352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6829","nonce":1063973352}]},"public_addr":"172.21.15.186:6825/1063973352","cluster_addr":"172.21.15.186:6827/1063973352","heartbeat_back_addr":"172.21.15.186:6831/1063973352","heartbeat_front_addr":"172.21.15.186:6829/1063973352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:21.305331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:44.877932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:08.784665+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:34.381762+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:58.999358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:23.620237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:49.961719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:15.363702+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:40.599703+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:07.391645+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:32.800851+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:59.384148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/718966776":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/2035551671":"2024-04-27T17:45:48.218816+0000","172.21.15.31:6801/153341213":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/810461143":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/3148082955":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6801/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/288292084":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/3101596726":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/153341213":"2024-04-27T17:45:48.218816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T17:53:11.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.793+0000 7f2145bef700 1 -- 172.21.15.31:0/2303297083 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f21280a3c40 msgr2=0x7f21280a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:11.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.793+0000 7f2145bef700 1 --2- 172.21.15.31:0/2303297083 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f21280a3c40 0x7f21280a6100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2134005ea0 tx=0x7f2134005e30 comp rx=0 tx=0).stop 2024-04-26T17:53:11.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.793+0000 7f2145bef700 1 -- 172.21.15.31:0/2303297083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2140102320 msgr2=0x7f214019a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:11.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.793+0000 7f2145bef700 1 --2- 172.21.15.31:0/2303297083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2140102320 0x7f214019a950 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2130009f70 tx=0x7f2130005640 comp rx=0 tx=0).stop 2024-04-26T17:53:11.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.794+0000 7f2145bef700 1 -- 172.21.15.31:0/2303297083 shutdown_connections 2024-04-26T17:53:11.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.794+0000 7f2145bef700 1 --2- 172.21.15.31:0/2303297083 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f21280a3c40 0x7f21280a6100 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:11.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.794+0000 7f2145bef700 1 --2- 172.21.15.31:0/2303297083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2140108390 0x7f21401a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:11.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.794+0000 7f2145bef700 1 --2- 172.21.15.31:0/2303297083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2140102320 0x7f214019a950 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:11.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.794+0000 7f2145bef700 1 --2- 172.21.15.31:0/2303297083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2140102ce0 0x7f214019ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:11.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.794+0000 7f2145bef700 1 -- 172.21.15.31:0/2303297083 >> 172.21.15.31:0/2303297083 conn(0x7f21400792d0 msgr2=0x7f2140100fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:11.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.794+0000 7f2145bef700 1 -- 172.21.15.31:0/2303297083 shutdown_connections 2024-04-26T17:53:11.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:11.794+0000 7f2145bef700 1 -- 172.21.15.31:0/2303297083 wait complete. 2024-04-26T17:53:12.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:11 smithi186 ceph-mon[32797]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-26T17:53:12.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:11 smithi097 ceph-mon[32419]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-26T17:53:12.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-04-26T17:53:12.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:12 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2303297083' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:53:12.838 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:13.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:12 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2303297083' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:53:13.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:12 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2303297083' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:53:13.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:13 smithi031 ceph-mon[29164]: pgmap v300: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T17:53:14.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:13 smithi186 ceph-mon[32797]: pgmap v300: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T17:53:14.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:13 smithi097 ceph-mon[32419]: pgmap v300: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T17:53:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.524+0000 7fb37c359700 1 -- 172.21.15.31:0/134513218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb374101970 msgr2=0x7fb374101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.524+0000 7fb37c359700 1 --2- 172.21.15.31:0/134513218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb374101970 0x7fb374101df0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb3640099c0 tx=0x7fb364009cd0 comp rx=0 tx=0).stop 2024-04-26T17:53:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.524+0000 7fb37c359700 1 -- 172.21.15.31:0/134513218 shutdown_connections 2024-04-26T17:53:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.524+0000 7fb37c359700 1 --2- 172.21.15.31:0/134513218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb374102330 0x7fb3741102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.524+0000 7fb37c359700 1 --2- 172.21.15.31:0/134513218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3741094d0 0x7fb3741098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.524+0000 7fb37c359700 1 --2- 172.21.15.31:0/134513218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb374101970 0x7fb374101df0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.524+0000 7fb37c359700 1 -- 172.21.15.31:0/134513218 >> 172.21.15.31:0/134513218 conn(0x7fb374078920 msgr2=0x7fb3740ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.524+0000 7fb37c359700 1 -- 172.21.15.31:0/134513218 shutdown_connections 2024-04-26T17:53:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.524+0000 7fb37c359700 1 -- 172.21.15.31:0/134513218 wait complete. 2024-04-26T17:53:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.524+0000 7fb37c359700 1 Processor -- start 2024-04-26T17:53:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.524+0000 7fb37c359700 1 -- start start 2024-04-26T17:53:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37c359700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb374101970 0x7fb374102b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37c359700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb374102330 0x7fb374103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37c359700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3741094d0 0x7fb3741035d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37c359700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb374113650 con 0x7fb374101970 2024-04-26T17:53:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37c359700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb3741134d0 con 0x7fb374102330 2024-04-26T17:53:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37c359700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb3741137d0 con 0x7fb3741094d0 2024-04-26T17:53:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37a0f5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb374101970 0x7fb374102b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37a0f5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb374101970 0x7fb374102b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41558/0 (socket says 172.21.15.31:41558) 2024-04-26T17:53:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37a0f5700 1 -- 172.21.15.31:0/3968348418 learned_addr learned my addr 172.21.15.31:0/3968348418 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb3798f4700 1 --2- 172.21.15.31:0/3968348418 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb374102330 0x7fb374103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37a8f6700 1 --2- 172.21.15.31:0/3968348418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3741094d0 0x7fb3741035d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37a0f5700 1 -- 172.21.15.31:0/3968348418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3741094d0 msgr2=0x7fb3741035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37a0f5700 1 --2- 172.21.15.31:0/3968348418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3741094d0 0x7fb3741035d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37a0f5700 1 -- 172.21.15.31:0/3968348418 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb374102330 msgr2=0x7fb374103090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37a0f5700 1 --2- 172.21.15.31:0/3968348418 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb374102330 0x7fb374103090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37a0f5700 1 -- 172.21.15.31:0/3968348418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3741a8150 con 0x7fb374101970 2024-04-26T17:53:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb37a0f5700 1 --2- 172.21.15.31:0/3968348418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb374101970 0x7fb374102b50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb37000d980 tx=0x7fb37000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.525+0000 7fb36b7fe700 1 -- 172.21.15.31:0/3968348418 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb370017070 con 0x7fb374101970 2024-04-26T17:53:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.526+0000 7fb36b7fe700 1 -- 172.21.15.31:0/3968348418 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb370005020 con 0x7fb374101970 2024-04-26T17:53:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.526+0000 7fb37c359700 1 -- 172.21.15.31:0/3968348418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3741a8440 con 0x7fb374101970 2024-04-26T17:53:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.526+0000 7fb36b7fe700 1 -- 172.21.15.31:0/3968348418 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37000fa10 con 0x7fb374101970 2024-04-26T17:53:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.526+0000 7fb37c359700 1 -- 172.21.15.31:0/3968348418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3741a8900 con 0x7fb374101970 2024-04-26T17:53:14.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.527+0000 7fb37c359700 1 -- 172.21.15.31:0/3968348418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb37406a110 con 0x7fb374101970 2024-04-26T17:53:14.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.528+0000 7fb36b7fe700 1 -- 172.21.15.31:0/3968348418 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb370004a30 con 0x7fb374101970 2024-04-26T17:53:14.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.529+0000 7fb36b7fe700 1 --2- 172.21.15.31:0/3968348418 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb3600a3c40 0x7fb3600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:14.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.529+0000 7fb3798f4700 1 --2- 172.21.15.31:0/3968348418 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb3600a3c40 0x7fb3600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:14.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.529+0000 7fb36b7fe700 1 -- 172.21.15.31:0/3968348418 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fb3700d04a0 con 0x7fb374101970 2024-04-26T17:53:14.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.529+0000 7fb3798f4700 1 --2- 172.21.15.31:0/3968348418 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb3600a3c40 0x7fb3600a6100 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb36400b580 tx=0x7fb364005a90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:14.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.532+0000 7fb36b7fe700 1 -- 172.21.15.31:0/3968348418 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb370020020 con 0x7fb374101970 2024-04-26T17:53:14.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:14.792+0000 7fb37c359700 1 -- 172.21.15.31:0/3968348418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb37402e490 con 0x7fb374101970 2024-04-26T17:53:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:15.649+0000 7fb36b7fe700 1 -- 172.21.15.31:0/3968348418 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7fb37009cfd0 con 0x7fb374101970 2024-04-26T17:53:15.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:15.652+0000 7fb37c359700 1 -- 172.21.15.31:0/3968348418 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb3600a3c40 msgr2=0x7fb3600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:15.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:15.652+0000 7fb37c359700 1 --2- 172.21.15.31:0/3968348418 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb3600a3c40 0x7fb3600a6100 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb36400b580 tx=0x7fb364005a90 comp rx=0 tx=0).stop 2024-04-26T17:53:15.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:15.652+0000 7fb37c359700 1 -- 172.21.15.31:0/3968348418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb374101970 msgr2=0x7fb374102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:15.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:15.652+0000 7fb37c359700 1 --2- 172.21.15.31:0/3968348418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb374101970 0x7fb374102b50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb37000d980 tx=0x7fb37000dc90 comp rx=0 tx=0).stop 2024-04-26T17:53:15.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:15.652+0000 7fb37c359700 1 -- 172.21.15.31:0/3968348418 shutdown_connections 2024-04-26T17:53:15.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:15.652+0000 7fb37c359700 1 --2- 172.21.15.31:0/3968348418 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb3600a3c40 0x7fb3600a6100 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:15.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:15.652+0000 7fb37c359700 1 --2- 172.21.15.31:0/3968348418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3741094d0 0x7fb3741035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:15.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:15.652+0000 7fb37c359700 1 --2- 172.21.15.31:0/3968348418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb374101970 0x7fb374102b50 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:15.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:15.652+0000 7fb37c359700 1 --2- 172.21.15.31:0/3968348418 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb374102330 0x7fb374103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:15.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:15.652+0000 7fb37c359700 1 -- 172.21.15.31:0/3968348418 >> 172.21.15.31:0/3968348418 conn(0x7fb374078920 msgr2=0x7fb374100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:15.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:15.653+0000 7fb37c359700 1 -- 172.21.15.31:0/3968348418 shutdown_connections 2024-04-26T17:53:15.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:15.653+0000 7fb37c359700 1 -- 172.21.15.31:0/3968348418 wait complete. 2024-04-26T17:53:15.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:15 smithi031 ceph-mon[29164]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T17:53:15.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:15 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3968348418' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T17:53:16.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:15 smithi186 ceph-mon[32797]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T17:53:16.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:15 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3968348418' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T17:53:16.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:15 smithi097 ceph-mon[32419]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T17:53:16.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:15 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3968348418' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T17:53:16.380 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-26T17:53:16.381 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-04-26T17:53:16.638 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:16.998 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:16 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3968348418' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T17:53:16.998 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:16 smithi031 ceph-mon[29164]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T17:53:17.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:16 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3968348418' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T17:53:17.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:16 smithi186 ceph-mon[32797]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T17:53:17.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:16 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3968348418' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T17:53:17.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:16 smithi097 ceph-mon[32419]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T17:53:17.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:17 smithi031 ceph-mon[29164]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T17:53:18.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:17 smithi186 ceph-mon[32797]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T17:53:18.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:17 smithi097 ceph-mon[32419]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T17:53:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.299+0000 7f994a32a700 1 -- 172.21.15.31:0/1221594660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99441094f0 msgr2=0x7f99441098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.299+0000 7f994a32a700 1 --2- 172.21.15.31:0/1221594660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99441094f0 0x7f99441098d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9934009990 tx=0x7f9934009ca0 comp rx=0 tx=0).stop 2024-04-26T17:53:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.299+0000 7f994a32a700 1 -- 172.21.15.31:0/1221594660 shutdown_connections 2024-04-26T17:53:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.299+0000 7f994a32a700 1 --2- 172.21.15.31:0/1221594660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9944103e90 0x7f99441102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.299+0000 7f994a32a700 1 --2- 172.21.15.31:0/1221594660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99441094f0 0x7f99441098d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.299+0000 7f994a32a700 1 --2- 172.21.15.31:0/1221594660 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99441034d0 0x7f9944103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.299+0000 7f994a32a700 1 -- 172.21.15.31:0/1221594660 >> 172.21.15.31:0/1221594660 conn(0x7f994407a500 msgr2=0x7f994407a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:18.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.299+0000 7f994a32a700 1 -- 172.21.15.31:0/1221594660 shutdown_connections 2024-04-26T17:53:18.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.299+0000 7f994a32a700 1 -- 172.21.15.31:0/1221594660 wait complete. 2024-04-26T17:53:18.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.300+0000 7f994a32a700 1 Processor -- start 2024-04-26T17:53:18.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.300+0000 7f994a32a700 1 -- start start 2024-04-26T17:53:18.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.300+0000 7f994a32a700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99441034d0 0x7f9944075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:18.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.300+0000 7f994a32a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9944103e90 0x7f9944075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:18.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.300+0000 7f994a32a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99441094f0 0x7f99440761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:18.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.300+0000 7f994a32a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f99441135f0 con 0x7f99441094f0 2024-04-26T17:53:18.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.300+0000 7f994a32a700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f9944113470 con 0x7f99441034d0 2024-04-26T17:53:18.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.300+0000 7f994a32a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9944113770 con 0x7f9944103e90 2024-04-26T17:53:18.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.300+0000 7f99488c7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99441094f0 0x7f99440761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:18.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.300+0000 7f99488c7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99441094f0 0x7f99440761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44964/0 (socket says 172.21.15.31:44964) 2024-04-26T17:53:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.300+0000 7f99488c7700 1 -- 172.21.15.31:0/2534296277 learned_addr learned my addr 172.21.15.31:0/2534296277 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.300+0000 7f9943fff700 1 --2- 172.21.15.31:0/2534296277 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99441034d0 0x7f9944075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.300+0000 7f99437fe700 1 --2- 172.21.15.31:0/2534296277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9944103e90 0x7f9944075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.301+0000 7f99488c7700 1 -- 172.21.15.31:0/2534296277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9944103e90 msgr2=0x7f9944075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.301+0000 7f99488c7700 1 --2- 172.21.15.31:0/2534296277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9944103e90 0x7f9944075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.301+0000 7f99488c7700 1 -- 172.21.15.31:0/2534296277 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99441034d0 msgr2=0x7f9944075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.301+0000 7f99488c7700 1 --2- 172.21.15.31:0/2534296277 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99441034d0 0x7f9944075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.301+0000 7f99488c7700 1 -- 172.21.15.31:0/2534296277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99441a8070 con 0x7f99441094f0 2024-04-26T17:53:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.301+0000 7f99488c7700 1 --2- 172.21.15.31:0/2534296277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99441094f0 0x7f99440761a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f993c007ee0 tx=0x7f993c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.301+0000 7f99417fa700 1 -- 172.21.15.31:0/2534296277 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c004bb0 con 0x7f99441094f0 2024-04-26T17:53:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.301+0000 7f994a32a700 1 -- 172.21.15.31:0/2534296277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99441a8360 con 0x7f99441094f0 2024-04-26T17:53:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.301+0000 7f99417fa700 1 -- 172.21.15.31:0/2534296277 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f993c004d50 con 0x7f99441094f0 2024-04-26T17:53:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.301+0000 7f99417fa700 1 -- 172.21.15.31:0/2534296277 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c005920 con 0x7f99441094f0 2024-04-26T17:53:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.301+0000 7f994a32a700 1 -- 172.21.15.31:0/2534296277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99441a8850 con 0x7f99441094f0 2024-04-26T17:53:18.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.302+0000 7f994a32a700 1 -- 172.21.15.31:0/2534296277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f994406a110 con 0x7f99441094f0 2024-04-26T17:53:18.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.303+0000 7f99417fa700 1 -- 172.21.15.31:0/2534296277 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f993c01a020 con 0x7f99441094f0 2024-04-26T17:53:18.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.304+0000 7f99417fa700 1 --2- 172.21.15.31:0/2534296277 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f992c0a3c40 0x7f992c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:18.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.304+0000 7f99417fa700 1 -- 172.21.15.31:0/2534296277 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f993c0cff70 con 0x7f99441094f0 2024-04-26T17:53:18.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.304+0000 7f9943fff700 1 --2- 172.21.15.31:0/2534296277 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f992c0a3c40 0x7f992c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:18.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.304+0000 7f9943fff700 1 --2- 172.21.15.31:0/2534296277 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f992c0a3c40 0x7f992c0a6100 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f993400b5c0 tx=0x7f9934005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:18.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.308+0000 7f99417fa700 1 -- 172.21.15.31:0/2534296277 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f993c010070 con 0x7f99441094f0 2024-04-26T17:53:18.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.563+0000 7f994a32a700 1 -- 172.21.15.31:0/2534296277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f994402e490 con 0x7f99441094f0 2024-04-26T17:53:18.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.661+0000 7f99417fa700 1 -- 172.21.15.31:0/2534296277 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f993c09caa0 con 0x7f99441094f0 2024-04-26T17:53:18.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.663+0000 7f994a32a700 1 -- 172.21.15.31:0/2534296277 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f992c0a3c40 msgr2=0x7f992c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:18.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.663+0000 7f994a32a700 1 --2- 172.21.15.31:0/2534296277 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f992c0a3c40 0x7f992c0a6100 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f993400b5c0 tx=0x7f9934005960 comp rx=0 tx=0).stop 2024-04-26T17:53:18.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.663+0000 7f994a32a700 1 -- 172.21.15.31:0/2534296277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99441094f0 msgr2=0x7f99440761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:18.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.663+0000 7f994a32a700 1 --2- 172.21.15.31:0/2534296277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99441094f0 0x7f99440761a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f993c007ee0 tx=0x7f993c00d3b0 comp rx=0 tx=0).stop 2024-04-26T17:53:18.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.663+0000 7f994a32a700 1 -- 172.21.15.31:0/2534296277 shutdown_connections 2024-04-26T17:53:18.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.663+0000 7f994a32a700 1 --2- 172.21.15.31:0/2534296277 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f992c0a3c40 0x7f992c0a6100 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:18.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.663+0000 7f994a32a700 1 --2- 172.21.15.31:0/2534296277 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f99441034d0 0x7f9944075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:18.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.664+0000 7f994a32a700 1 --2- 172.21.15.31:0/2534296277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99441094f0 0x7f99440761a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:18.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.664+0000 7f994a32a700 1 --2- 172.21.15.31:0/2534296277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9944103e90 0x7f9944075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:18.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.664+0000 7f994a32a700 1 -- 172.21.15.31:0/2534296277 >> 172.21.15.31:0/2534296277 conn(0x7f994407a500 msgr2=0x7f9944102250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:18.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.664+0000 7f994a32a700 1 -- 172.21.15.31:0/2534296277 shutdown_connections 2024-04-26T17:53:18.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:18.664+0000 7f994a32a700 1 -- 172.21.15.31:0/2534296277 wait complete. 2024-04-26T17:53:19.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:18 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2534296277' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T17:53:19.045 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:18 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2534296277' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T17:53:19.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:18 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2534296277' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T17:53:19.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-04-26T17:53:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:19.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:19 smithi031 ceph-mon[29164]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 3 op/s 2024-04-26T17:53:19.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:19 smithi031 ceph-mon[29164]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T17:53:19.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:19 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2534296277' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T17:53:19.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:19 smithi031 ceph-mon[29164]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T17:53:19.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:19 smithi031 ceph-mon[29164]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T17:53:19.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:19 smithi031 ceph-mon[29164]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-26T17:53:19.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:19 smithi031 ceph-mon[29164]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-26T17:53:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:19 smithi186 ceph-mon[32797]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 3 op/s 2024-04-26T17:53:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:19 smithi186 ceph-mon[32797]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T17:53:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:19 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2534296277' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T17:53:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:19 smithi186 ceph-mon[32797]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T17:53:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:19 smithi186 ceph-mon[32797]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T17:53:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:19 smithi186 ceph-mon[32797]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-26T17:53:20.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:19 smithi186 ceph-mon[32797]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-26T17:53:20.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:19 smithi097 ceph-mon[32419]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 3 op/s 2024-04-26T17:53:20.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:19 smithi097 ceph-mon[32419]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T17:53:20.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:19 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2534296277' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T17:53:20.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:19 smithi097 ceph-mon[32419]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T17:53:20.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:19 smithi097 ceph-mon[32419]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T17:53:20.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:19 smithi097 ceph-mon[32419]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-26T17:53:20.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:19 smithi097 ceph-mon[32419]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-26T17:53:21.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:20 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] up:active 2024-04-26T17:53:21.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:20 smithi186 ceph-mon[32797]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T17:53:21.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:20 smithi186 ceph-mon[32797]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T17:53:21.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:20 smithi186 ceph-mon[32797]: Cluster is now healthy 2024-04-26T17:53:21.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:20 smithi186 ceph-mon[32797]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-26T17:53:21.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:20 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-26T17:53:21.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:20 smithi186 ceph-mon[32797]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-26T17:53:21.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:20 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] up:active 2024-04-26T17:53:21.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:20 smithi031 ceph-mon[29164]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T17:53:21.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:20 smithi031 ceph-mon[29164]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T17:53:21.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:20 smithi031 ceph-mon[29164]: Cluster is now healthy 2024-04-26T17:53:21.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:20 smithi031 ceph-mon[29164]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-26T17:53:21.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:20 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-26T17:53:21.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:20 smithi031 ceph-mon[29164]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-26T17:53:21.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:20 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] up:active 2024-04-26T17:53:21.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:20 smithi097 ceph-mon[32419]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T17:53:21.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:20 smithi097 ceph-mon[32419]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T17:53:21.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:20 smithi097 ceph-mon[32419]: Cluster is now healthy 2024-04-26T17:53:21.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:20 smithi097 ceph-mon[32419]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-26T17:53:21.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:20 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-26T17:53:21.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:20 smithi097 ceph-mon[32419]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-26T17:53:21.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.497+0000 7f113769d700 1 -- 172.21.15.31:0/2936404345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1130103e30 msgr2=0x7f1130110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:21.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.497+0000 7f113769d700 1 --2- 172.21.15.31:0/2936404345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1130103e30 0x7f1130110290 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f112800b0a0 tx=0x7f112800b3b0 comp rx=0 tx=0).stop 2024-04-26T17:53:21.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.497+0000 7f113769d700 1 -- 172.21.15.31:0/2936404345 shutdown_connections 2024-04-26T17:53:21.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.497+0000 7f113769d700 1 --2- 172.21.15.31:0/2936404345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1130103e30 0x7f1130110290 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:21.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.497+0000 7f113769d700 1 --2- 172.21.15.31:0/2936404345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11301094c0 0x7f11301098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:21.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.497+0000 7f113769d700 1 --2- 172.21.15.31:0/2936404345 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1130103470 0x7f11301038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:21.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.497+0000 7f113769d700 1 -- 172.21.15.31:0/2936404345 >> 172.21.15.31:0/2936404345 conn(0x7f113007a580 msgr2=0x7f113007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:21.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.497+0000 7f113769d700 1 -- 172.21.15.31:0/2936404345 shutdown_connections 2024-04-26T17:53:21.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.497+0000 7f113769d700 1 -- 172.21.15.31:0/2936404345 wait complete. 2024-04-26T17:53:21.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.498+0000 7f113769d700 1 Processor -- start 2024-04-26T17:53:21.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.498+0000 7f113769d700 1 -- start start 2024-04-26T17:53:21.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.498+0000 7f113769d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1130103470 0x7f11301a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:21.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.498+0000 7f113769d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1130103e30 0x7f11301a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:21.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.498+0000 7f113769d700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11301094c0 0x7f113019bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:21.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.498+0000 7f113769d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f11301135e0 con 0x7f1130103470 2024-04-26T17:53:21.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.498+0000 7f113769d700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f1130113460 con 0x7f11301094c0 2024-04-26T17:53:21.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.498+0000 7f113769d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1130113760 con 0x7f1130103e30 2024-04-26T17:53:21.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.498+0000 7f1135439700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1130103470 0x7f11301a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f1135439700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1130103470 0x7f11301a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44986/0 (socket says 172.21.15.31:44986) 2024-04-26T17:53:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f1135439700 1 -- 172.21.15.31:0/3575535746 learned_addr learned my addr 172.21.15.31:0/3575535746 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f1135c3a700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11301094c0 0x7f113019bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f1134c38700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1130103e30 0x7f11301a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f1135c3a700 1 -- 172.21.15.31:0/3575535746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1130103e30 msgr2=0x7f11301a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f1135c3a700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1130103e30 0x7f11301a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f1135c3a700 1 -- 172.21.15.31:0/3575535746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1130103470 msgr2=0x7f11301a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f1135c3a700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1130103470 0x7f11301a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f1135c3a700 1 -- 172.21.15.31:0/3575535746 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f113019c210 con 0x7f11301094c0 2024-04-26T17:53:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f1135439700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1130103470 0x7f11301a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:53:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f1135c3a700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11301094c0 0x7f113019bab0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f112800d010 tx=0x7f1128004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f11227fc700 1 -- 172.21.15.31:0/3575535746 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128021070 con 0x7f11301094c0 2024-04-26T17:53:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f113769d700 1 -- 172.21.15.31:0/3575535746 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f113019c4a0 con 0x7f11301094c0 2024-04-26T17:53:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f11227fc700 1 -- 172.21.15.31:0/3575535746 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1128007e30 con 0x7f11301094c0 2024-04-26T17:53:21.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f11227fc700 1 -- 172.21.15.31:0/3575535746 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11280045a0 con 0x7f11301094c0 2024-04-26T17:53:21.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.499+0000 7f113769d700 1 -- 172.21.15.31:0/3575535746 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11301a87c0 con 0x7f11301094c0 2024-04-26T17:53:21.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.500+0000 7f113769d700 1 -- 172.21.15.31:0/3575535746 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f113006a110 con 0x7f11301094c0 2024-04-26T17:53:21.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.501+0000 7f11227fc700 1 -- 172.21.15.31:0/3575535746 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1128004740 con 0x7f11301094c0 2024-04-26T17:53:21.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.502+0000 7f11227fc700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f111c0a3c40 0x7f111c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:21.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.502+0000 7f1135439700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f111c0a3c40 0x7f111c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:21.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.502+0000 7f11227fc700 1 -- 172.21.15.31:0/3575535746 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f112800f020 con 0x7f11301094c0 2024-04-26T17:53:21.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.502+0000 7f1135439700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f111c0a3c40 0x7f111c0a6100 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f112c005950 tx=0x7f112c00a3b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:21.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.506+0000 7f11227fc700 1 -- 172.21.15.31:0/3575535746 <== mon.2 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f112802a020 con 0x7f11301094c0 2024-04-26T17:53:21.634 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 17:53:21 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T17:53:21.138+0000 7f83a0ee8700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T17:53:21.634 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 17:53:21 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T17:53:21.138+0000 7f83a0ee8700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T17:53:21.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:21.764+0000 7f113769d700 1 -- 172.21.15.31:0/3575535746 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f11301a8990 con 0x7f11301094c0 2024-04-26T17:53:22.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:21 smithi186 ceph-mon[32797]: pgmap v304: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-04-26T17:53:22.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:21 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] up:active 2024-04-26T17:53:22.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:21 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-26T17:53:22.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:21 smithi031 ceph-mon[29164]: pgmap v304: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-04-26T17:53:22.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:21 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] up:active 2024-04-26T17:53:22.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:21 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-26T17:53:22.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:21 smithi097 ceph-mon[32419]: pgmap v304: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-04-26T17:53:22.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:21 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] up:active 2024-04-26T17:53:22.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:21 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-26T17:53:22.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:22.686+0000 7f11227fc700 1 -- 172.21.15.31:0/3575535746 <== mon.2 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f112809e340 con 0x7f11301094c0 2024-04-26T17:53:22.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:22.689+0000 7f113769d700 1 -- 172.21.15.31:0/3575535746 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f111c0a3c40 msgr2=0x7f111c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:22.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:22.689+0000 7f113769d700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f111c0a3c40 0x7f111c0a6100 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f112c005950 tx=0x7f112c00a3b0 comp rx=0 tx=0).stop 2024-04-26T17:53:22.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:22.689+0000 7f113769d700 1 -- 172.21.15.31:0/3575535746 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11301094c0 msgr2=0x7f113019bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:22.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:22.689+0000 7f113769d700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11301094c0 0x7f113019bab0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f112800d010 tx=0x7f1128004d00 comp rx=0 tx=0).stop 2024-04-26T17:53:22.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:22.689+0000 7f113769d700 1 -- 172.21.15.31:0/3575535746 shutdown_connections 2024-04-26T17:53:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:22.689+0000 7f113769d700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f111c0a3c40 0x7f111c0a6100 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:22.689+0000 7f113769d700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11301094c0 0x7f113019bab0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:22.689+0000 7f113769d700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1130103470 0x7f11301a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:22.689+0000 7f113769d700 1 --2- 172.21.15.31:0/3575535746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1130103e30 0x7f11301a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:22.689+0000 7f113769d700 1 -- 172.21.15.31:0/3575535746 >> 172.21.15.31:0/3575535746 conn(0x7f113007a580 msgr2=0x7f113010e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:22.689+0000 7f113769d700 1 -- 172.21.15.31:0/3575535746 shutdown_connections 2024-04-26T17:53:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:22.689+0000 7f113769d700 1 -- 172.21.15.31:0/3575535746 wait complete. 2024-04-26T17:53:22.998 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:22 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3575535746' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T17:53:22.998 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:22 smithi031 ceph-mon[29164]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T17:53:23.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:22 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3575535746' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T17:53:23.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:22 smithi186 ceph-mon[32797]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T17:53:23.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:22 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3575535746' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T17:53:23.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:22 smithi097 ceph-mon[32419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T17:53:23.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-04-26T17:53:23.704 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:23.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:23 smithi031 ceph-mon[29164]: pgmap v305: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-04-26T17:53:23.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:23 smithi031 ceph-mon[29164]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T17:53:23.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:23 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-26T17:53:23.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:23 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:53:24.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:23 smithi186 ceph-mon[32797]: pgmap v305: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-04-26T17:53:24.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:23 smithi186 ceph-mon[32797]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T17:53:24.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:23 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-26T17:53:24.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:23 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:53:24.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:23 smithi097 ceph-mon[32419]: pgmap v305: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-04-26T17:53:24.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:23 smithi097 ceph-mon[32419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T17:53:24.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:23 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-26T17:53:24.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:23 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:53:25.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.482+0000 7f13afaea700 1 -- 172.21.15.31:0/1695250126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a81094e0 msgr2=0x7f13a81098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:25.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.482+0000 7f13afaea700 1 --2- 172.21.15.31:0/1695250126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a81094e0 0x7f13a81098c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f139800b0a0 tx=0x7f139800b3b0 comp rx=0 tx=0).stop 2024-04-26T17:53:25.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.482+0000 7f13afaea700 1 -- 172.21.15.31:0/1695250126 shutdown_connections 2024-04-26T17:53:25.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.482+0000 7f13afaea700 1 --2- 172.21.15.31:0/1695250126 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13a8103e60 0x7f13a81102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:25.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.482+0000 7f13afaea700 1 --2- 172.21.15.31:0/1695250126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a81094e0 0x7f13a81098c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:25.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.482+0000 7f13afaea700 1 --2- 172.21.15.31:0/1695250126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13a81034a0 0x7f13a8103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:25.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.482+0000 7f13afaea700 1 -- 172.21.15.31:0/1695250126 >> 172.21.15.31:0/1695250126 conn(0x7f13a807a590 msgr2=0x7f13a807a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:25.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.482+0000 7f13afaea700 1 -- 172.21.15.31:0/1695250126 shutdown_connections 2024-04-26T17:53:25.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.482+0000 7f13afaea700 1 -- 172.21.15.31:0/1695250126 wait complete. 2024-04-26T17:53:25.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.482+0000 7f13afaea700 1 Processor -- start 2024-04-26T17:53:25.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.482+0000 7f13afaea700 1 -- start start 2024-04-26T17:53:25.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13afaea700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a81034a0 0x7f13a8075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:25.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13afaea700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13a8103e60 0x7f13a8075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:25.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13afaea700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13a81094e0 0x7f13a80761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:25.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13afaea700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f13a81136b0 con 0x7f13a81034a0 2024-04-26T17:53:25.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13afaea700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f13a8113530 con 0x7f13a81094e0 2024-04-26T17:53:25.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13afaea700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f13a8113830 con 0x7f13a8103e60 2024-04-26T17:53:25.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13ad886700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a81034a0 0x7f13a8075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:25.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13ad886700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a81034a0 0x7f13a8075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43602/0 (socket says 172.21.15.31:43602) 2024-04-26T17:53:25.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13ad886700 1 -- 172.21.15.31:0/313940710 learned_addr learned my addr 172.21.15.31:0/313940710 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:25.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13ae087700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13a81094e0 0x7f13a80761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:25.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13ad085700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13a8103e60 0x7f13a8075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:25.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13ad886700 1 -- 172.21.15.31:0/313940710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13a8103e60 msgr2=0x7f13a8075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:25.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13ad886700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13a8103e60 0x7f13a8075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:25.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13ad886700 1 -- 172.21.15.31:0/313940710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13a81094e0 msgr2=0x7f13a80761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:25.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13ad886700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13a81094e0 0x7f13a80761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:25.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13ad886700 1 -- 172.21.15.31:0/313940710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a81a8110 con 0x7f13a81034a0 2024-04-26T17:53:25.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13ad085700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13a8103e60 0x7f13a8075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:53:25.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13ae087700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13a81094e0 0x7f13a80761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:53:25.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13ad886700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a81034a0 0x7f13a8075720 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f139800b070 tx=0x7f1398004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:25.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.483+0000 7f13a6ffd700 1 -- 172.21.15.31:0/313940710 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1398021070 con 0x7f13a81034a0 2024-04-26T17:53:25.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.484+0000 7f13a6ffd700 1 -- 172.21.15.31:0/313940710 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1398007db0 con 0x7f13a81034a0 2024-04-26T17:53:25.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.484+0000 7f13afaea700 1 -- 172.21.15.31:0/313940710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a81a83a0 con 0x7f13a81034a0 2024-04-26T17:53:25.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.484+0000 7f13a6ffd700 1 -- 172.21.15.31:0/313940710 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139801d450 con 0x7f13a81034a0 2024-04-26T17:53:25.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.484+0000 7f13afaea700 1 -- 172.21.15.31:0/313940710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a81a8930 con 0x7f13a81034a0 2024-04-26T17:53:25.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.485+0000 7f13afaea700 1 -- 172.21.15.31:0/313940710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13a806a110 con 0x7f13a81034a0 2024-04-26T17:53:25.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.486+0000 7f13a6ffd700 1 -- 172.21.15.31:0/313940710 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f13980040b0 con 0x7f13a81034a0 2024-04-26T17:53:25.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.487+0000 7f13a6ffd700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f13900a3cc0 0x7f13900a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:25.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.487+0000 7f13ad085700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f13900a3cc0 0x7f13900a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:25.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.487+0000 7f13a6ffd700 1 -- 172.21.15.31:0/313940710 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f139800f020 con 0x7f13a81034a0 2024-04-26T17:53:25.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.488+0000 7f13ad085700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f13900a3cc0 0x7f13900a6180 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f13a807ac70 tx=0x7f139c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:25.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.491+0000 7f13a6ffd700 1 -- 172.21.15.31:0/313940710 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f139802a020 con 0x7f13a81034a0 2024-04-26T17:53:25.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:25.753+0000 7f13afaea700 1 -- 172.21.15.31:0/313940710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f13a8076c50 con 0x7f13a81034a0 2024-04-26T17:53:25.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:25 smithi031 ceph-mon[29164]: pgmap v306: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-26T17:53:26.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:25 smithi186 ceph-mon[32797]: pgmap v306: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-26T17:53:26.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:25 smithi097 ceph-mon[32419]: pgmap v306: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-26T17:53:26.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:26.697+0000 7f13a6ffd700 1 -- 172.21.15.31:0/313940710 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v25) ==== 115+0+0 (secure 0 0 0) 0x7f139809e960 con 0x7f13a81034a0 2024-04-26T17:53:26.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:26.700+0000 7f13afaea700 1 -- 172.21.15.31:0/313940710 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f13900a3cc0 msgr2=0x7f13900a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:26.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:26.700+0000 7f13afaea700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f13900a3cc0 0x7f13900a6180 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f13a807ac70 tx=0x7f139c008040 comp rx=0 tx=0).stop 2024-04-26T17:53:26.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:26.700+0000 7f13afaea700 1 -- 172.21.15.31:0/313940710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a81034a0 msgr2=0x7f13a8075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:26.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:26.700+0000 7f13afaea700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a81034a0 0x7f13a8075720 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f139800b070 tx=0x7f1398004ef0 comp rx=0 tx=0).stop 2024-04-26T17:53:26.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:26.700+0000 7f13afaea700 1 -- 172.21.15.31:0/313940710 shutdown_connections 2024-04-26T17:53:26.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:26.700+0000 7f13afaea700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f13900a3cc0 0x7f13900a6180 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:26.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:26.700+0000 7f13afaea700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f13a81094e0 0x7f13a80761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:26.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:26.700+0000 7f13afaea700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a81034a0 0x7f13a8075720 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:26.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:26.700+0000 7f13afaea700 1 --2- 172.21.15.31:0/313940710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13a8103e60 0x7f13a8075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:26.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:26.700+0000 7f13afaea700 1 -- 172.21.15.31:0/313940710 >> 172.21.15.31:0/313940710 conn(0x7f13a807a590 msgr2=0x7f13a81022d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:26.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:26.700+0000 7f13afaea700 1 -- 172.21.15.31:0/313940710 shutdown_connections 2024-04-26T17:53:26.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:26.700+0000 7f13afaea700 1 -- 172.21.15.31:0/313940710 wait complete. 2024-04-26T17:53:27.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:26 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/313940710' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T17:53:27.033 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:26 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/313940710' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T17:53:27.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:26 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/313940710' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T17:53:27.521 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-04-26T17:53:27.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-04-26T17:53:27.782 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:27.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:27 smithi031 ceph-mon[29164]: pgmap v307: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-26T17:53:27.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:27 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/313940710' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T17:53:27.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:27 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:53:28.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:27 smithi186 ceph-mon[32797]: pgmap v307: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-26T17:53:28.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:27 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/313940710' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T17:53:28.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:27 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:53:28.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:27 smithi097 ceph-mon[32419]: pgmap v307: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-26T17:53:28.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:27 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/313940710' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T17:53:28.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:27 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:53:29.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.532+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3373591263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ec1094c0 msgr2=0x7ff7ec1098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:29.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.532+0000 7ff7f3e0c700 1 --2- 172.21.15.31:0/3373591263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ec1094c0 0x7ff7ec1098a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4009990 tx=0x7ff7e4009ca0 comp rx=0 tx=0).stop 2024-04-26T17:53:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.532+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3373591263 shutdown_connections 2024-04-26T17:53:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.532+0000 7ff7f3e0c700 1 --2- 172.21.15.31:0/3373591263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7ec103e30 0x7ff7ec110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.532+0000 7ff7f3e0c700 1 --2- 172.21.15.31:0/3373591263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ec1094c0 0x7ff7ec1098a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.532+0000 7ff7f3e0c700 1 --2- 172.21.15.31:0/3373591263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7ec103470 0x7ff7ec1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.532+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3373591263 >> 172.21.15.31:0/3373591263 conn(0x7ff7ec07a580 msgr2=0x7ff7ec07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.532+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3373591263 shutdown_connections 2024-04-26T17:53:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.532+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3373591263 wait complete. 2024-04-26T17:53:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f3e0c700 1 Processor -- start 2024-04-26T17:53:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f3e0c700 1 -- start start 2024-04-26T17:53:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f3e0c700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7ec103470 0x7ff7ec075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f3e0c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7ec103e30 0x7ff7ec075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f3e0c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ec1094c0 0x7ff7ec0761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f3e0c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec1136c0 con 0x7ff7ec1094c0 2024-04-26T17:53:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f3e0c700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec113540 con 0x7ff7ec103470 2024-04-26T17:53:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f3e0c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec113840 con 0x7ff7ec103e30 2024-04-26T17:53:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f23a9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ec1094c0 0x7ff7ec0761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f23a9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ec1094c0 0x7ff7ec0761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43622/0 (socket says 172.21.15.31:43622) 2024-04-26T17:53:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f23a9700 1 -- 172.21.15.31:0/3698511095 learned_addr learned my addr 172.21.15.31:0/3698511095 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f1ba8700 1 --2- 172.21.15.31:0/3698511095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7ec103470 0x7ff7ec075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f13a7700 1 --2- 172.21.15.31:0/3698511095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7ec103e30 0x7ff7ec075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f23a9700 1 -- 172.21.15.31:0/3698511095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7ec103e30 msgr2=0x7ff7ec075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f23a9700 1 --2- 172.21.15.31:0/3698511095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7ec103e30 0x7ff7ec075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f23a9700 1 -- 172.21.15.31:0/3698511095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7ec103470 msgr2=0x7ff7ec075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f23a9700 1 --2- 172.21.15.31:0/3698511095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7ec103470 0x7ff7ec075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.533+0000 7ff7f23a9700 1 -- 172.21.15.31:0/3698511095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ec1a8260 con 0x7ff7ec1094c0 2024-04-26T17:53:29.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.534+0000 7ff7f23a9700 1 --2- 172.21.15.31:0/3698511095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ec1094c0 0x7ff7ec0761a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc00b970 tx=0x7ff7dc00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:29.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.534+0000 7ff7eaffd700 1 -- 172.21.15.31:0/3698511095 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc00d860 con 0x7ff7ec1094c0 2024-04-26T17:53:29.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.534+0000 7ff7eaffd700 1 -- 172.21.15.31:0/3698511095 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7dc00e430 con 0x7ff7ec1094c0 2024-04-26T17:53:29.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.534+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3698511095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7ec1a8550 con 0x7ff7ec1094c0 2024-04-26T17:53:29.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.534+0000 7ff7eaffd700 1 -- 172.21.15.31:0/3698511095 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc017660 con 0x7ff7ec1094c0 2024-04-26T17:53:29.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.534+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3698511095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7ec1a8a90 con 0x7ff7ec1094c0 2024-04-26T17:53:29.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.535+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3698511095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7ec06a110 con 0x7ff7ec1094c0 2024-04-26T17:53:29.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.536+0000 7ff7eaffd700 1 -- 172.21.15.31:0/3698511095 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff7dc00da00 con 0x7ff7ec1094c0 2024-04-26T17:53:29.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.537+0000 7ff7eaffd700 1 --2- 172.21.15.31:0/3698511095 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff7d40a3cc0 0x7ff7d40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:29.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.537+0000 7ff7f1ba8700 1 --2- 172.21.15.31:0/3698511095 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff7d40a3cc0 0x7ff7d40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:29.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.537+0000 7ff7eaffd700 1 -- 172.21.15.31:0/3698511095 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7ff7dc0cfe70 con 0x7ff7ec1094c0 2024-04-26T17:53:29.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.537+0000 7ff7f1ba8700 1 --2- 172.21.15.31:0/3698511095 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff7d40a3cc0 0x7ff7d40a6180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff7e400b5c0 tx=0x7ff7e4005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:29.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.542+0000 7ff7eaffd700 1 -- 172.21.15.31:0/3698511095 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff7dc013030 con 0x7ff7ec1094c0 2024-04-26T17:53:29.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.784+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3698511095 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff7ec063860 con 0x7ff7d40a3cc0 2024-04-26T17:53:29.784 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:29 smithi031 ceph-mon[29164]: pgmap v308: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T17:53:29.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.808+0000 7ff7eaffd700 1 -- 172.21.15.31:0/3698511095 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff7ec063860 con 0x7ff7d40a3cc0 2024-04-26T17:53:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.810+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3698511095 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff7d40a3cc0 msgr2=0x7ff7d40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.810+0000 7ff7f3e0c700 1 --2- 172.21.15.31:0/3698511095 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff7d40a3cc0 0x7ff7d40a6180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff7e400b5c0 tx=0x7ff7e4005960 comp rx=0 tx=0).stop 2024-04-26T17:53:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.810+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3698511095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ec1094c0 msgr2=0x7ff7ec0761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.810+0000 7ff7f3e0c700 1 --2- 172.21.15.31:0/3698511095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ec1094c0 0x7ff7ec0761a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc00b970 tx=0x7ff7dc00bc80 comp rx=0 tx=0).stop 2024-04-26T17:53:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.810+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3698511095 shutdown_connections 2024-04-26T17:53:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.810+0000 7ff7f3e0c700 1 --2- 172.21.15.31:0/3698511095 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff7d40a3cc0 0x7ff7d40a6180 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.810+0000 7ff7f3e0c700 1 --2- 172.21.15.31:0/3698511095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7ec103470 0x7ff7ec075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.810+0000 7ff7f3e0c700 1 --2- 172.21.15.31:0/3698511095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ec1094c0 0x7ff7ec0761a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.810+0000 7ff7f3e0c700 1 --2- 172.21.15.31:0/3698511095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7ec103e30 0x7ff7ec075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.810+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3698511095 >> 172.21.15.31:0/3698511095 conn(0x7ff7ec07a580 msgr2=0x7ff7ec102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.810+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3698511095 shutdown_connections 2024-04-26T17:53:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:29.811+0000 7ff7f3e0c700 1 -- 172.21.15.31:0/3698511095 wait complete. 2024-04-26T17:53:30.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:29 smithi186 ceph-mon[32797]: pgmap v308: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T17:53:30.060 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:53:29 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:53:29.790+0000 7fe597964700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:53:30.060 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:53:29 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:53:29.790+0000 7fe597964700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:53:30.060 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:53:29 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:53:29.790+0000 7fe597964700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:53:30.060 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:53:29 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:53:29.790+0000 7fe597964700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:53:30.060 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:53:29 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:53:29.790+0000 7fe597964700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:53:30.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:29 smithi097 ceph-mon[32419]: pgmap v308: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T17:53:30.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-26T17:53:30.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:30 smithi031 ceph-mon[29164]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:30.834 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:31.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:30 smithi186 ceph-mon[32797]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:31.035 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:30 smithi097 ceph-mon[32419]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:32.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:31 smithi186 ceph-mon[32797]: pgmap v309: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T17:53:32.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:31 smithi186 ceph-mon[32797]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-26T17:53:32.037 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:31 smithi031 ceph-mon[29164]: pgmap v309: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T17:53:32.038 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:31 smithi031 ceph-mon[29164]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-26T17:53:32.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:31 smithi097 ceph-mon[32419]: pgmap v309: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T17:53:32.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:31 smithi097 ceph-mon[32419]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-26T17:53:32.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.602+0000 7f9ed7585700 1 -- 172.21.15.31:0/2880823723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed01094e0 msgr2=0x7f9ed01098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:32.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.602+0000 7f9ed7585700 1 --2- 172.21.15.31:0/2880823723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed01094e0 0x7f9ed01098c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009990 tx=0x7f9ecc009ca0 comp rx=0 tx=0).stop 2024-04-26T17:53:32.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.602+0000 7f9ed7585700 1 -- 172.21.15.31:0/2880823723 shutdown_connections 2024-04-26T17:53:32.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.602+0000 7f9ed7585700 1 --2- 172.21.15.31:0/2880823723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ed006c510 0x7f9ed01102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:32.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.602+0000 7f9ed7585700 1 --2- 172.21.15.31:0/2880823723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed01094e0 0x7f9ed01098c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:32.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.602+0000 7f9ed7585700 1 --2- 172.21.15.31:0/2880823723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ed006bb50 0x7f9ed006bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:32.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.602+0000 7f9ed7585700 1 -- 172.21.15.31:0/2880823723 >> 172.21.15.31:0/2880823723 conn(0x7f9ed0078190 msgr2=0x7f9ed00785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:32.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.602+0000 7f9ed7585700 1 -- 172.21.15.31:0/2880823723 shutdown_connections 2024-04-26T17:53:32.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.602+0000 7f9ed7585700 1 -- 172.21.15.31:0/2880823723 wait complete. 2024-04-26T17:53:32.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.602+0000 7f9ed7585700 1 Processor -- start 2024-04-26T17:53:32.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.602+0000 7f9ed7585700 1 -- start start 2024-04-26T17:53:32.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed7585700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed006bb50 0x7f9ed0102740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:32.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed7585700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ed006c510 0x7f9ed0102c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:32.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed7585700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ed01094e0 0x7f9ed01031c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:32.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed7585700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed0113630 con 0x7f9ed006bb50 2024-04-26T17:53:32.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed7585700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01134b0 con 0x7f9ed006c510 2024-04-26T17:53:32.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed7585700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01137b0 con 0x7f9ed01094e0 2024-04-26T17:53:32.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed5321700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed006bb50 0x7f9ed0102740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:32.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed5321700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed006bb50 0x7f9ed0102740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43666/0 (socket says 172.21.15.31:43666) 2024-04-26T17:53:32.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed5321700 1 -- 172.21.15.31:0/2346880970 learned_addr learned my addr 172.21.15.31:0/2346880970 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:32.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed5b22700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ed01094e0 0x7f9ed01031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:32.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed4b20700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ed006c510 0x7f9ed0102c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:32.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed5b22700 1 -- 172.21.15.31:0/2346880970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ed006c510 msgr2=0x7f9ed0102c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:32.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed5b22700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ed006c510 0x7f9ed0102c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:32.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed5b22700 1 -- 172.21.15.31:0/2346880970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed006bb50 msgr2=0x7f9ed0102740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:32.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed5b22700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed006bb50 0x7f9ed0102740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:32.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed5b22700 1 -- 172.21.15.31:0/2346880970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed01a80b0 con 0x7f9ed01094e0 2024-04-26T17:53:32.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed4b20700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ed006c510 0x7f9ed0102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:53:32.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed5321700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed006bb50 0x7f9ed0102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:53:32.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.603+0000 7f9ed5b22700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ed01094e0 0x7f9ed01031c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8007f10 tx=0x7f9ec800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:32.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.604+0000 7f9ec67fc700 1 -- 172.21.15.31:0/2346880970 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec800ddd0 con 0x7f9ed01094e0 2024-04-26T17:53:32.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.604+0000 7f9ed7585700 1 -- 172.21.15.31:0/2346880970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ed01a83a0 con 0x7f9ed01094e0 2024-04-26T17:53:32.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.604+0000 7f9ec67fc700 1 -- 172.21.15.31:0/2346880970 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ec8004bb0 con 0x7f9ed01094e0 2024-04-26T17:53:32.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.604+0000 7f9ec67fc700 1 -- 172.21.15.31:0/2346880970 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec80044f0 con 0x7f9ed01094e0 2024-04-26T17:53:32.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.604+0000 7f9ed7585700 1 -- 172.21.15.31:0/2346880970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ed01a8890 con 0x7f9ed01094e0 2024-04-26T17:53:32.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.605+0000 7f9ed7585700 1 -- 172.21.15.31:0/2346880970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ed0104b80 con 0x7f9ed01094e0 2024-04-26T17:53:32.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.606+0000 7f9ec67fc700 1 -- 172.21.15.31:0/2346880970 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9ec801c020 con 0x7f9ed01094e0 2024-04-26T17:53:32.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.606+0000 7f9ec67fc700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9ebc0a3cd0 0x7f9ebc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:32.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.606+0000 7f9ec67fc700 1 -- 172.21.15.31:0/2346880970 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f9ec80cfeb0 con 0x7f9ed01094e0 2024-04-26T17:53:32.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.606+0000 7f9ed5321700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9ebc0a3cd0 0x7f9ebc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:32.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.607+0000 7f9ed5321700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9ebc0a3cd0 0x7f9ebc0a6190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc006010 tx=0x7f9ecc01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:32.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.610+0000 7f9ec67fc700 1 -- 172.21.15.31:0/2346880970 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ec8010070 con 0x7f9ed01094e0 2024-04-26T17:53:32.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.851+0000 7f9ed7585700 1 -- 172.21.15.31:0/2346880970 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9ed0063860 con 0x7f9ebc0a3cd0 2024-04-26T17:53:32.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.877+0000 7f9ec67fc700 1 -- 172.21.15.31:0/2346880970 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9ed0063860 con 0x7f9ebc0a3cd0 2024-04-26T17:53:32.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.879+0000 7f9ed7585700 1 -- 172.21.15.31:0/2346880970 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9ebc0a3cd0 msgr2=0x7f9ebc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:32.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.879+0000 7f9ed7585700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9ebc0a3cd0 0x7f9ebc0a6190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc006010 tx=0x7f9ecc01a040 comp rx=0 tx=0).stop 2024-04-26T17:53:32.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.879+0000 7f9ed7585700 1 -- 172.21.15.31:0/2346880970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ed01094e0 msgr2=0x7f9ed01031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:32.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.879+0000 7f9ed7585700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ed01094e0 0x7f9ed01031c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8007f10 tx=0x7f9ec800d3b0 comp rx=0 tx=0).stop 2024-04-26T17:53:32.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.879+0000 7f9ed7585700 1 -- 172.21.15.31:0/2346880970 shutdown_connections 2024-04-26T17:53:32.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.879+0000 7f9ed7585700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9ebc0a3cd0 0x7f9ebc0a6190 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:32.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.879+0000 7f9ed7585700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ed01094e0 0x7f9ed01031c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:32.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.879+0000 7f9ed7585700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed006bb50 0x7f9ed0102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:32.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.879+0000 7f9ed7585700 1 --2- 172.21.15.31:0/2346880970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ed006c510 0x7f9ed0102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:32.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.879+0000 7f9ed7585700 1 -- 172.21.15.31:0/2346880970 >> 172.21.15.31:0/2346880970 conn(0x7f9ed0078190 msgr2=0x7f9ed0101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:32.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.879+0000 7f9ed7585700 1 -- 172.21.15.31:0/2346880970 shutdown_connections 2024-04-26T17:53:32.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:32.879+0000 7f9ed7585700 1 -- 172.21.15.31:0/2346880970 wait complete. 2024-04-26T17:53:33.678 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-26T17:53:33.928 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:34.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:33 smithi186 ceph-mon[32797]: pgmap v310: 97 pgs: 97 active+clean; 588 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-04-26T17:53:34.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:33 smithi186 ceph-mon[32797]: from='client.24523 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:34.050 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:33 smithi031 ceph-mon[29164]: pgmap v310: 97 pgs: 97 active+clean; 588 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-04-26T17:53:34.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:33 smithi031 ceph-mon[29164]: from='client.24523 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:34.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:33 smithi097 ceph-mon[32419]: pgmap v310: 97 pgs: 97 active+clean; 588 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-04-26T17:53:34.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:33 smithi097 ceph-mon[32419]: from='client.24523 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.724+0000 7f34c4a0e700 1 -- 172.21.15.31:0/2493166318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34c0109560 msgr2=0x7f34c0109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.724+0000 7f34c4a0e700 1 --2- 172.21.15.31:0/2493166318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34c0109560 0x7f34c0109940 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f34b40099c0 tx=0x7f34b4009cd0 comp rx=0 tx=0).stop 2024-04-26T17:53:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.724+0000 7f34c4a0e700 1 -- 172.21.15.31:0/2493166318 shutdown_connections 2024-04-26T17:53:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.724+0000 7f34c4a0e700 1 --2- 172.21.15.31:0/2493166318 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34c0103e80 0x7f34c0110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.724+0000 7f34c4a0e700 1 --2- 172.21.15.31:0/2493166318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34c0109560 0x7f34c0109940 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.724+0000 7f34c4a0e700 1 --2- 172.21.15.31:0/2493166318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34c01034c0 0x7f34c0103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.724+0000 7f34c4a0e700 1 -- 172.21.15.31:0/2493166318 >> 172.21.15.31:0/2493166318 conn(0x7f34c007a5d0 msgr2=0x7f34c007a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.724+0000 7f34c4a0e700 1 -- 172.21.15.31:0/2493166318 shutdown_connections 2024-04-26T17:53:35.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.724+0000 7f34c4a0e700 1 -- 172.21.15.31:0/2493166318 wait complete. 2024-04-26T17:53:35.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.725+0000 7f34c4a0e700 1 Processor -- start 2024-04-26T17:53:35.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.725+0000 7f34c4a0e700 1 -- start start 2024-04-26T17:53:35.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.725+0000 7f34c4a0e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34c01034c0 0x7f34c019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:35.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.725+0000 7f34c4a0e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34c0103e80 0x7f34c019fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:35.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.725+0000 7f34c4a0e700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34c0109560 0x7f34c01999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:35.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.725+0000 7f34c4a0e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01136c0 con 0x7f34c01034c0 2024-04-26T17:53:35.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.725+0000 7f34c4a0e700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f34c0113540 con 0x7f34c0109560 2024-04-26T17:53:35.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.725+0000 7f34c4a0e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f34c0113840 con 0x7f34c0103e80 2024-04-26T17:53:35.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.725+0000 7f34be59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34c01034c0 0x7f34c019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:35.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.725+0000 7f34be59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34c01034c0 0x7f34c019f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44414/0 (socket says 172.21.15.31:44414) 2024-04-26T17:53:35.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.725+0000 7f34be59c700 1 -- 172.21.15.31:0/1976885664 learned_addr learned my addr 172.21.15.31:0/1976885664 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:35.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.725+0000 7f34bdd9b700 1 --2- 172.21.15.31:0/1976885664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34c0103e80 0x7f34c019fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:35.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.725+0000 7f34bed9d700 1 --2- 172.21.15.31:0/1976885664 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34c0109560 0x7f34c01999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:35.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.726+0000 7f34be59c700 1 -- 172.21.15.31:0/1976885664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34c0103e80 msgr2=0x7f34c019fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:35.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.726+0000 7f34be59c700 1 --2- 172.21.15.31:0/1976885664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34c0103e80 0x7f34c019fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:35.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.726+0000 7f34be59c700 1 -- 172.21.15.31:0/1976885664 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34c0109560 msgr2=0x7f34c01999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:35.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.726+0000 7f34be59c700 1 --2- 172.21.15.31:0/1976885664 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34c0109560 0x7f34c01999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:35.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.726+0000 7f34be59c700 1 -- 172.21.15.31:0/1976885664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c019a100 con 0x7f34c01034c0 2024-04-26T17:53:35.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.726+0000 7f34be59c700 1 --2- 172.21.15.31:0/1976885664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34c01034c0 0x7f34c019f790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f34b4005910 tx=0x7f34b40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:35.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:35 smithi031 ceph-mon[29164]: pgmap v311: 97 pgs: 97 active+clean; 588 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-26T17:53:35.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.726+0000 7f34af7fe700 1 -- 172.21.15.31:0/1976885664 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b4026070 con 0x7f34c01034c0 2024-04-26T17:53:35.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.726+0000 7f34af7fe700 1 -- 172.21.15.31:0/1976885664 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34b400f460 con 0x7f34c01034c0 2024-04-26T17:53:35.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.726+0000 7f34c4a0e700 1 -- 172.21.15.31:0/1976885664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c019a390 con 0x7f34c01034c0 2024-04-26T17:53:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.726+0000 7f34af7fe700 1 -- 172.21.15.31:0/1976885664 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b4009e60 con 0x7f34c01034c0 2024-04-26T17:53:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.726+0000 7f34c4a0e700 1 -- 172.21.15.31:0/1976885664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c01a6660 con 0x7f34c01034c0 2024-04-26T17:53:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.728+0000 7f34c4a0e700 1 -- 172.21.15.31:0/1976885664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34c010b9f0 con 0x7f34c01034c0 2024-04-26T17:53:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.729+0000 7f34af7fe700 1 -- 172.21.15.31:0/1976885664 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f34b4029070 con 0x7f34c01034c0 2024-04-26T17:53:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.730+0000 7f34af7fe700 1 --2- 172.21.15.31:0/1976885664 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f34a80a3d50 0x7f34a80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.730+0000 7f34bdd9b700 1 --2- 172.21.15.31:0/1976885664 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f34a80a3d50 0x7f34a80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.730+0000 7f34af7fe700 1 -- 172.21.15.31:0/1976885664 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f34b40d0b00 con 0x7f34c01034c0 2024-04-26T17:53:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.731+0000 7f34bdd9b700 1 --2- 172.21.15.31:0/1976885664 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f34a80a3d50 0x7f34a80a6210 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f34c007acb0 tx=0x7f34b0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:35.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.735+0000 7f34af7fe700 1 -- 172.21.15.31:0/1976885664 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f34b405dff0 con 0x7f34c01034c0 2024-04-26T17:53:35.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.980+0000 7f34c4a0e700 1 -- 172.21.15.31:0/1976885664 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f34c019ae80 con 0x7f34a80a3d50 2024-04-26T17:53:35.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.985+0000 7f34af7fe700 1 -- 172.21.15.31:0/1976885664 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f34c019ae80 con 0x7f34a80a3d50 2024-04-26T17:53:35.985 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/qa/sv_0/d273f893-9259-4d21-a670-d83f5a5b9ee0 2024-04-26T17:53:35.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.987+0000 7f34c4a0e700 1 -- 172.21.15.31:0/1976885664 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f34a80a3d50 msgr2=0x7f34a80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:35.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.987+0000 7f34c4a0e700 1 --2- 172.21.15.31:0/1976885664 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f34a80a3d50 0x7f34a80a6210 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f34c007acb0 tx=0x7f34b0009260 comp rx=0 tx=0).stop 2024-04-26T17:53:35.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.987+0000 7f34c4a0e700 1 -- 172.21.15.31:0/1976885664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34c01034c0 msgr2=0x7f34c019f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:35.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.987+0000 7f34c4a0e700 1 --2- 172.21.15.31:0/1976885664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34c01034c0 0x7f34c019f790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f34b4005910 tx=0x7f34b40045e0 comp rx=0 tx=0).stop 2024-04-26T17:53:35.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.987+0000 7f34c4a0e700 1 -- 172.21.15.31:0/1976885664 shutdown_connections 2024-04-26T17:53:35.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.987+0000 7f34c4a0e700 1 --2- 172.21.15.31:0/1976885664 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f34a80a3d50 0x7f34a80a6210 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:35.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.987+0000 7f34c4a0e700 1 --2- 172.21.15.31:0/1976885664 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34c0109560 0x7f34c01999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:35.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.987+0000 7f34c4a0e700 1 --2- 172.21.15.31:0/1976885664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34c01034c0 0x7f34c019f790 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:35.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.987+0000 7f34c4a0e700 1 --2- 172.21.15.31:0/1976885664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34c0103e80 0x7f34c019fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:35.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.987+0000 7f34c4a0e700 1 -- 172.21.15.31:0/1976885664 >> 172.21.15.31:0/1976885664 conn(0x7f34c007a5d0 msgr2=0x7f34c0102240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:35.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.987+0000 7f34c4a0e700 1 -- 172.21.15.31:0/1976885664 shutdown_connections 2024-04-26T17:53:35.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:35.987+0000 7f34c4a0e700 1 -- 172.21.15.31:0/1976885664 wait complete. 2024-04-26T17:53:36.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:35 smithi186 ceph-mon[32797]: pgmap v311: 97 pgs: 97 active+clean; 588 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-26T17:53:36.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:35 smithi097 ceph-mon[32419]: pgmap v311: 97 pgs: 97 active+clean; 588 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-26T17:53:36.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-26T17:53:37.009 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:37.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:36 smithi186 ceph-mon[32797]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:37.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:36 smithi031 ceph-mon[29164]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:37.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:36 smithi097 ceph-mon[32419]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:37.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:37 smithi031 ceph-mon[29164]: pgmap v312: 97 pgs: 97 active+clean; 608 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-26T17:53:38.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:37 smithi186 ceph-mon[32797]: pgmap v312: 97 pgs: 97 active+clean; 608 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-26T17:53:38.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:37 smithi097 ceph-mon[32419]: pgmap v312: 97 pgs: 97 active+clean; 608 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-26T17:53:38.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.666+0000 7efeff83f700 1 -- 172.21.15.31:0/1194470857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef81027a0 msgr2=0x7efef810e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:38.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.666+0000 7efeff83f700 1 --2- 172.21.15.31:0/1194470857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef81027a0 0x7efef810e9b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efef000b0a0 tx=0x7efef000b3b0 comp rx=0 tx=0).stop 2024-04-26T17:53:38.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.666+0000 7efeff83f700 1 -- 172.21.15.31:0/1194470857 shutdown_connections 2024-04-26T17:53:38.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.666+0000 7efeff83f700 1 --2- 172.21.15.31:0/1194470857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef81027a0 0x7efef810e9b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:38.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.666+0000 7efeff83f700 1 --2- 172.21.15.31:0/1194470857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef8107c20 0x7efef8108000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:38.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.666+0000 7efeff83f700 1 --2- 172.21.15.31:0/1194470857 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efef8101de0 0x7efef8102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:38.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.666+0000 7efeff83f700 1 -- 172.21.15.31:0/1194470857 >> 172.21.15.31:0/1194470857 conn(0x7efef8077640 msgr2=0x7efef8077a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:38.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.666+0000 7efeff83f700 1 -- 172.21.15.31:0/1194470857 shutdown_connections 2024-04-26T17:53:38.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.666+0000 7efeff83f700 1 -- 172.21.15.31:0/1194470857 wait complete. 2024-04-26T17:53:38.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efeff83f700 1 Processor -- start 2024-04-26T17:53:38.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efeff83f700 1 -- start start 2024-04-26T17:53:38.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efeff83f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef8101de0 0x7efef819f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:38.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efeff83f700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efef81027a0 0x7efef819fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:38.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efeff83f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef8107c20 0x7efef8199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:38.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efeff83f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efef8112580 con 0x7efef8107c20 2024-04-26T17:53:38.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efeff83f700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7efef8112400 con 0x7efef81027a0 2024-04-26T17:53:38.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efeff83f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efef8112700 con 0x7efef8101de0 2024-04-26T17:53:38.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efefdddc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef8107c20 0x7efef8199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:38.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efefdddc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef8107c20 0x7efef8199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44422/0 (socket says 172.21.15.31:44422) 2024-04-26T17:53:38.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efefdddc700 1 -- 172.21.15.31:0/3709267218 learned_addr learned my addr 172.21.15.31:0/3709267218 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:38.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efefcdda700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efef81027a0 0x7efef819fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:38.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efefd5db700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef8101de0 0x7efef819f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:38.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efefdddc700 1 -- 172.21.15.31:0/3709267218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef8101de0 msgr2=0x7efef819f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:38.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efefdddc700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef8101de0 0x7efef819f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:38.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efefdddc700 1 -- 172.21.15.31:0/3709267218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efef81027a0 msgr2=0x7efef819fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:38.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efefdddc700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efef81027a0 0x7efef819fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:38.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efefdddc700 1 -- 172.21.15.31:0/3709267218 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef8199f70 con 0x7efef8107c20 2024-04-26T17:53:38.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.667+0000 7efefd5db700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef8101de0 0x7efef819f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:53:38.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.668+0000 7efefdddc700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef8107c20 0x7efef8199790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efef000b070 tx=0x7efef000ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:38.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.668+0000 7efeee7fc700 1 -- 172.21.15.31:0/3709267218 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef0021070 con 0x7efef8107c20 2024-04-26T17:53:38.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.668+0000 7efeff83f700 1 -- 172.21.15.31:0/3709267218 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef819a200 con 0x7efef8107c20 2024-04-26T17:53:38.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.668+0000 7efeee7fc700 1 -- 172.21.15.31:0/3709267218 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efef0007c80 con 0x7efef8107c20 2024-04-26T17:53:38.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.668+0000 7efeee7fc700 1 -- 172.21.15.31:0/3709267218 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef0005440 con 0x7efef8107c20 2024-04-26T17:53:38.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.668+0000 7efeff83f700 1 -- 172.21.15.31:0/3709267218 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef81a6470 con 0x7efef8107c20 2024-04-26T17:53:38.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.669+0000 7efeff83f700 1 -- 172.21.15.31:0/3709267218 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef81031e0 con 0x7efef8107c20 2024-04-26T17:53:38.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.670+0000 7efeee7fc700 1 -- 172.21.15.31:0/3709267218 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7efef0024070 con 0x7efef8107c20 2024-04-26T17:53:38.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.670+0000 7efeee7fc700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7efee40a3cd0 0x7efee40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:38.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.671+0000 7efefd5db700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7efee40a3cd0 0x7efee40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:38.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.671+0000 7efeee7fc700 1 -- 172.21.15.31:0/3709267218 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7efef000f020 con 0x7efef8107c20 2024-04-26T17:53:38.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.671+0000 7efefd5db700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7efee40a3cd0 0x7efee40a6190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efef4005fd0 tx=0x7efef4005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:38.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.675+0000 7efeee7fc700 1 -- 172.21.15.31:0/3709267218 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efef002b020 con 0x7efef8107c20 2024-04-26T17:53:38.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.920+0000 7efeff83f700 1 -- 172.21.15.31:0/3709267218 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7efef8104d90 con 0x7efee40a3cd0 2024-04-26T17:53:38.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.946+0000 7efeee7fc700 1 -- 172.21.15.31:0/3709267218 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7efef8104d90 con 0x7efee40a3cd0 2024-04-26T17:53:38.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.948+0000 7efeff83f700 1 -- 172.21.15.31:0/3709267218 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7efee40a3cd0 msgr2=0x7efee40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:38.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.948+0000 7efeff83f700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7efee40a3cd0 0x7efee40a6190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efef4005fd0 tx=0x7efef4005e30 comp rx=0 tx=0).stop 2024-04-26T17:53:38.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.948+0000 7efeff83f700 1 -- 172.21.15.31:0/3709267218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef8107c20 msgr2=0x7efef8199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:38.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.948+0000 7efeff83f700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef8107c20 0x7efef8199790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efef000b070 tx=0x7efef000ce50 comp rx=0 tx=0).stop 2024-04-26T17:53:38.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.948+0000 7efeff83f700 1 -- 172.21.15.31:0/3709267218 shutdown_connections 2024-04-26T17:53:38.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.948+0000 7efeff83f700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7efee40a3cd0 0x7efee40a6190 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:38.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.948+0000 7efeff83f700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef8107c20 0x7efef8199790 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:38.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.948+0000 7efeff83f700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef8101de0 0x7efef819f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:38.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.948+0000 7efeff83f700 1 --2- 172.21.15.31:0/3709267218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efef81027a0 0x7efef819fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:38.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.948+0000 7efeff83f700 1 -- 172.21.15.31:0/3709267218 >> 172.21.15.31:0/3709267218 conn(0x7efef8077640 msgr2=0x7efef8103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:38.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.948+0000 7efeff83f700 1 -- 172.21.15.31:0/3709267218 shutdown_connections 2024-04-26T17:53:38.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:38.948+0000 7efeff83f700 1 -- 172.21.15.31:0/3709267218 wait complete. 2024-04-26T17:53:39.592 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-26T17:53:39.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:39 smithi031 ceph-mon[29164]: pgmap v313: 97 pgs: 97 active+clean; 608 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-26T17:53:39.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:39 smithi031 ceph-mon[29164]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:39.846 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:40.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:39 smithi186 ceph-mon[32797]: pgmap v313: 97 pgs: 97 active+clean; 608 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-26T17:53:40.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:39 smithi186 ceph-mon[32797]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:40.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:39 smithi097 ceph-mon[32419]: pgmap v313: 97 pgs: 97 active+clean; 608 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-26T17:53:40.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:39 smithi097 ceph-mon[32419]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.499+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2118513263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39cc103470 msgr2=0x7f39cc1038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.499+0000 7f39d1dd6700 1 --2- 172.21.15.31:0/2118513263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39cc103470 0x7f39cc1038f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f39c0009990 tx=0x7f39c0009ca0 comp rx=0 tx=0).stop 2024-04-26T17:53:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.499+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2118513263 shutdown_connections 2024-04-26T17:53:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.499+0000 7f39d1dd6700 1 --2- 172.21.15.31:0/2118513263 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc1094c0 0x7f39cc1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.499+0000 7f39d1dd6700 1 --2- 172.21.15.31:0/2118513263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39cc103e30 0x7f39cc110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.499+0000 7f39d1dd6700 1 --2- 172.21.15.31:0/2118513263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39cc103470 0x7f39cc1038f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.499+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2118513263 >> 172.21.15.31:0/2118513263 conn(0x7f39cc07a580 msgr2=0x7f39cc07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.499+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2118513263 shutdown_connections 2024-04-26T17:53:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.499+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2118513263 wait complete. 2024-04-26T17:53:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.500+0000 7f39d1dd6700 1 Processor -- start 2024-04-26T17:53:41.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.500+0000 7f39d1dd6700 1 -- start start 2024-04-26T17:53:41.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.500+0000 7f39d1dd6700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc103470 0x7f39cc075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:41.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.500+0000 7f39d1dd6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39cc103e30 0x7f39cc075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:41.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.500+0000 7f39d1dd6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39cc1094c0 0x7f39cc0761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:41.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.500+0000 7f39d1dd6700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc1135f0 con 0x7f39cc1094c0 2024-04-26T17:53:41.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.500+0000 7f39d1dd6700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc113470 con 0x7f39cc103470 2024-04-26T17:53:41.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.500+0000 7f39d1dd6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc113770 con 0x7f39cc103e30 2024-04-26T17:53:41.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.500+0000 7f39cbfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39cc1094c0 0x7f39cc0761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:41.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.500+0000 7f39cbfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39cc1094c0 0x7f39cc0761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44438/0 (socket says 172.21.15.31:44438) 2024-04-26T17:53:41.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.500+0000 7f39cbfff700 1 -- 172.21.15.31:0/2999498224 learned_addr learned my addr 172.21.15.31:0/2999498224 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:41.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.500+0000 7f39cb7fe700 1 --2- 172.21.15.31:0/2999498224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc103470 0x7f39cc075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:41.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.501+0000 7f39caffd700 1 --2- 172.21.15.31:0/2999498224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39cc103e30 0x7f39cc075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:41.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.501+0000 7f39cbfff700 1 -- 172.21.15.31:0/2999498224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39cc103e30 msgr2=0x7f39cc075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:41.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.501+0000 7f39cbfff700 1 --2- 172.21.15.31:0/2999498224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39cc103e30 0x7f39cc075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:41.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.501+0000 7f39cbfff700 1 -- 172.21.15.31:0/2999498224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc103470 msgr2=0x7f39cc075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:41.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.501+0000 7f39cbfff700 1 --2- 172.21.15.31:0/2999498224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc103470 0x7f39cc075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:41.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.501+0000 7f39cbfff700 1 -- 172.21.15.31:0/2999498224 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39cc1a8070 con 0x7f39cc1094c0 2024-04-26T17:53:41.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.501+0000 7f39cbfff700 1 --2- 172.21.15.31:0/2999498224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39cc1094c0 0x7f39cc0761a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f39c4007ee0 tx=0x7f39c400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:41.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.501+0000 7f39c8ff9700 1 -- 172.21.15.31:0/2999498224 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c4004bb0 con 0x7f39cc1094c0 2024-04-26T17:53:41.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.501+0000 7f39c8ff9700 1 -- 172.21.15.31:0/2999498224 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39c4004d50 con 0x7f39cc1094c0 2024-04-26T17:53:41.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.501+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2999498224 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39cc1a8360 con 0x7f39cc1094c0 2024-04-26T17:53:41.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.501+0000 7f39c8ff9700 1 -- 172.21.15.31:0/2999498224 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c4005920 con 0x7f39cc1094c0 2024-04-26T17:53:41.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.501+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2999498224 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39cc1a8870 con 0x7f39cc1094c0 2024-04-26T17:53:41.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.502+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2999498224 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39cc10b930 con 0x7f39cc1094c0 2024-04-26T17:53:41.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.503+0000 7f39c8ff9700 1 -- 172.21.15.31:0/2999498224 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f39c401a020 con 0x7f39cc1094c0 2024-04-26T17:53:41.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.504+0000 7f39c8ff9700 1 --2- 172.21.15.31:0/2999498224 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f39b40a3cd0 0x7f39b40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:41.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.504+0000 7f39c8ff9700 1 -- 172.21.15.31:0/2999498224 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f39c40cfa30 con 0x7f39cc1094c0 2024-04-26T17:53:41.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.504+0000 7f39cb7fe700 1 --2- 172.21.15.31:0/2999498224 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f39b40a3cd0 0x7f39b40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:41.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.505+0000 7f39cb7fe700 1 --2- 172.21.15.31:0/2999498224 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f39b40a3cd0 0x7f39b40a6190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f39bc00abd0 tx=0x7f39bc00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:41.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.509+0000 7f39c8ff9700 1 -- 172.21.15.31:0/2999498224 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f39c4010070 con 0x7f39cc1094c0 2024-04-26T17:53:41.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.754+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2999498224 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f39cc063860 con 0x7f39b40a3cd0 2024-04-26T17:53:41.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:41 smithi031 ceph-mon[29164]: pgmap v314: 97 pgs: 97 active+clean; 608 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-26T17:53:41.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.759+0000 7f39c8ff9700 1 -- 172.21.15.31:0/2999498224 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f39cc063860 con 0x7f39b40a3cd0 2024-04-26T17:53:41.758 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/qa/sv_1/26ef2745-8372-4260-b242-0000b032cf66 2024-04-26T17:53:41.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.761+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2999498224 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f39b40a3cd0 msgr2=0x7f39b40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:41.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.761+0000 7f39d1dd6700 1 --2- 172.21.15.31:0/2999498224 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f39b40a3cd0 0x7f39b40a6190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f39bc00abd0 tx=0x7f39bc00a380 comp rx=0 tx=0).stop 2024-04-26T17:53:41.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.761+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2999498224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39cc1094c0 msgr2=0x7f39cc0761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:41.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.761+0000 7f39d1dd6700 1 --2- 172.21.15.31:0/2999498224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39cc1094c0 0x7f39cc0761a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f39c4007ee0 tx=0x7f39c400d3b0 comp rx=0 tx=0).stop 2024-04-26T17:53:41.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.761+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2999498224 shutdown_connections 2024-04-26T17:53:41.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.761+0000 7f39d1dd6700 1 --2- 172.21.15.31:0/2999498224 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f39b40a3cd0 0x7f39b40a6190 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:41.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.761+0000 7f39d1dd6700 1 --2- 172.21.15.31:0/2999498224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc103470 0x7f39cc075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:41.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.761+0000 7f39d1dd6700 1 --2- 172.21.15.31:0/2999498224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39cc1094c0 0x7f39cc0761a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:41.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.761+0000 7f39d1dd6700 1 --2- 172.21.15.31:0/2999498224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39cc103e30 0x7f39cc075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:41.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.761+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2999498224 >> 172.21.15.31:0/2999498224 conn(0x7f39cc07a580 msgr2=0x7f39cc102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:41.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.761+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2999498224 shutdown_connections 2024-04-26T17:53:41.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:41.761+0000 7f39d1dd6700 1 -- 172.21.15.31:0/2999498224 wait complete. 2024-04-26T17:53:42.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:41 smithi186 ceph-mon[32797]: pgmap v314: 97 pgs: 97 active+clean; 608 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-26T17:53:42.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:41 smithi097 ceph-mon[32419]: pgmap v314: 97 pgs: 97 active+clean; 608 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-26T17:53:42.460 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:53:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:43.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:42 smithi186 ceph-mon[32797]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:43.046 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:42 smithi031 ceph-mon[29164]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:43.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:42 smithi097 ceph-mon[32419]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:53:43.794 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:43 smithi031 ceph-mon[29164]: pgmap v315: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-04-26T17:53:44.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:43 smithi186 ceph-mon[32797]: pgmap v315: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-04-26T17:53:44.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:43 smithi097 ceph-mon[32419]: pgmap v315: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-04-26T17:53:44.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.335+0000 7f8c52a06700 1 -- 172.21.15.31:0/3224883364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c4c06c4f0 msgr2=0x7f8c4c1102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:44.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.335+0000 7f8c52a06700 1 --2- 172.21.15.31:0/3224883364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c4c06c4f0 0x7f8c4c1102e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8c4400b0a0 tx=0x7f8c4400b3b0 comp rx=0 tx=0).stop 2024-04-26T17:53:44.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.335+0000 7f8c52a06700 1 -- 172.21.15.31:0/3224883364 shutdown_connections 2024-04-26T17:53:44.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.335+0000 7f8c52a06700 1 --2- 172.21.15.31:0/3224883364 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c4c1094a0 0x7f8c4c109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:44.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.335+0000 7f8c52a06700 1 --2- 172.21.15.31:0/3224883364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c4c06c4f0 0x7f8c4c1102e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:44.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.335+0000 7f8c52a06700 1 --2- 172.21.15.31:0/3224883364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c4c06bb50 0x7f8c4c06bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:44.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.335+0000 7f8c52a06700 1 -- 172.21.15.31:0/3224883364 >> 172.21.15.31:0/3224883364 conn(0x7f8c4c0792d0 msgr2=0x7f8c4c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:44.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.335+0000 7f8c52a06700 1 -- 172.21.15.31:0/3224883364 shutdown_connections 2024-04-26T17:53:44.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.335+0000 7f8c52a06700 1 -- 172.21.15.31:0/3224883364 wait complete. 2024-04-26T17:53:44.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.336+0000 7f8c52a06700 1 Processor -- start 2024-04-26T17:53:44.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.336+0000 7f8c52a06700 1 -- start start 2024-04-26T17:53:44.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.336+0000 7f8c52a06700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c4c06bb50 0x7f8c4c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:44.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.336+0000 7f8c52a06700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c4c06c4f0 0x7f8c4c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:44.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.336+0000 7f8c52a06700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c4c1094a0 0x7f8c4c1998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:44.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.336+0000 7f8c52a06700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c1135a0 con 0x7f8c4c1094a0 2024-04-26T17:53:44.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.336+0000 7f8c52a06700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c113420 con 0x7f8c4c06bb50 2024-04-26T17:53:44.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.336+0000 7f8c52a06700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c113720 con 0x7f8c4c06c4f0 2024-04-26T17:53:44.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c50fa3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c4c1094a0 0x7f8c4c1998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:44.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c50fa3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c4c1094a0 0x7f8c4c1998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44464/0 (socket says 172.21.15.31:44464) 2024-04-26T17:53:44.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c50fa3700 1 -- 172.21.15.31:0/1327444487 learned_addr learned my addr 172.21.15.31:0/1327444487 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:44.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c4b7fe700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c4c06c4f0 0x7f8c4c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:44.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c4bfff700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c4c06bb50 0x7f8c4c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:44.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c50fa3700 1 -- 172.21.15.31:0/1327444487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c4c06c4f0 msgr2=0x7f8c4c19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:44.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c50fa3700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c4c06c4f0 0x7f8c4c19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:44.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c50fa3700 1 -- 172.21.15.31:0/1327444487 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c4c06bb50 msgr2=0x7f8c4c19f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:44.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c50fa3700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c4c06bb50 0x7f8c4c19f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:44.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c50fa3700 1 -- 172.21.15.31:0/1327444487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c19a040 con 0x7f8c4c1094a0 2024-04-26T17:53:44.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c4bfff700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c4c06bb50 0x7f8c4c19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:53:44.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c50fa3700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c4c1094a0 0x7f8c4c1998e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8c440062a0 tx=0x7f8c44004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:44.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c497fa700 1 -- 172.21.15.31:0/1327444487 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c44021070 con 0x7f8c4c1094a0 2024-04-26T17:53:44.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c497fa700 1 -- 172.21.15.31:0/1327444487 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c44007c80 con 0x7f8c4c1094a0 2024-04-26T17:53:44.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c52a06700 1 -- 172.21.15.31:0/1327444487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c4c19a2d0 con 0x7f8c4c1094a0 2024-04-26T17:53:44.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c497fa700 1 -- 172.21.15.31:0/1327444487 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4401d450 con 0x7f8c4c1094a0 2024-04-26T17:53:44.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.337+0000 7f8c52a06700 1 -- 172.21.15.31:0/1327444487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c4c1a65f0 con 0x7f8c4c1094a0 2024-04-26T17:53:44.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.338+0000 7f8c52a06700 1 -- 172.21.15.31:0/1327444487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c4c06a110 con 0x7f8c4c1094a0 2024-04-26T17:53:44.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.339+0000 7f8c497fa700 1 -- 172.21.15.31:0/1327444487 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8c44004020 con 0x7f8c4c1094a0 2024-04-26T17:53:44.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.340+0000 7f8c497fa700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8c340a3cd0 0x7f8c340a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:44.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.340+0000 7f8c4bfff700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8c340a3cd0 0x7f8c340a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:44.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.340+0000 7f8c497fa700 1 -- 172.21.15.31:0/1327444487 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f8c4400f020 con 0x7f8c4c1094a0 2024-04-26T17:53:44.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.341+0000 7f8c4bfff700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8c340a3cd0 0x7f8c340a6190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c005e20 tx=0x7f8c3c005db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:44.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.345+0000 7f8c497fa700 1 -- 172.21.15.31:0/1327444487 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c4402a020 con 0x7f8c4c1094a0 2024-04-26T17:53:44.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.606+0000 7f8c52a06700 1 -- 172.21.15.31:0/1327444487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c4c02e490 con 0x7f8c4c1094a0 2024-04-26T17:53:44.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.607+0000 7f8c497fa700 1 -- 172.21.15.31:0/1327444487 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11871 (secure 0 0 0) 0x7f8c4409e580 con 0x7f8c4c1094a0 2024-04-26T17:53:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:53:44.606 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T17:53:25.754649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24457,24473,24446]},"id":1}]} 2024-04-26T17:53:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.609+0000 7f8c52a06700 1 -- 172.21.15.31:0/1327444487 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8c340a3cd0 msgr2=0x7f8c340a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.609+0000 7f8c52a06700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8c340a3cd0 0x7f8c340a6190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c005e20 tx=0x7f8c3c005db0 comp rx=0 tx=0).stop 2024-04-26T17:53:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.609+0000 7f8c52a06700 1 -- 172.21.15.31:0/1327444487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c4c1094a0 msgr2=0x7f8c4c1998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.609+0000 7f8c52a06700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c4c1094a0 0x7f8c4c1998e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8c440062a0 tx=0x7f8c44004e20 comp rx=0 tx=0).stop 2024-04-26T17:53:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.609+0000 7f8c52a06700 1 -- 172.21.15.31:0/1327444487 shutdown_connections 2024-04-26T17:53:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.609+0000 7f8c52a06700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8c340a3cd0 0x7f8c340a6190 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.609+0000 7f8c52a06700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c4c06bb50 0x7f8c4c19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.609+0000 7f8c52a06700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c4c1094a0 0x7f8c4c1998e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.609+0000 7f8c52a06700 1 --2- 172.21.15.31:0/1327444487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c4c06c4f0 0x7f8c4c19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:44.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.609+0000 7f8c52a06700 1 -- 172.21.15.31:0/1327444487 >> 172.21.15.31:0/1327444487 conn(0x7f8c4c0792d0 msgr2=0x7f8c4c100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:44.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.609+0000 7f8c52a06700 1 -- 172.21.15.31:0/1327444487 shutdown_connections 2024-04-26T17:53:44.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:44.609+0000 7f8c52a06700 1 -- 172.21.15.31:0/1327444487 wait complete. 2024-04-26T17:53:44.609 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-04-26T17:53:44.968 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:44 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1327444487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:53:45.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:44 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1327444487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:53:45.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:44 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1327444487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:53:45.309 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T17:53:45.562 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:45.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:45 smithi031 ceph-mon[29164]: pgmap v316: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.4 KiB/s wr, 7 op/s 2024-04-26T17:53:46.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:45 smithi186 ceph-mon[32797]: pgmap v316: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.4 KiB/s wr, 7 op/s 2024-04-26T17:53:46.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:45 smithi097 ceph-mon[32419]: pgmap v316: 97 pgs: 97 active+clean; 632 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.4 KiB/s wr, 7 op/s 2024-04-26T17:53:47.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.249+0000 7f3056724700 1 -- 172.21.15.31:0/1388577647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3050107680 msgr2=0x7f305010f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:47.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.249+0000 7f3056724700 1 --2- 172.21.15.31:0/1388577647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3050107680 0x7f305010f250 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f304800b0d0 tx=0x7f304800b3e0 comp rx=0 tx=0).stop 2024-04-26T17:53:47.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.249+0000 7f3056724700 1 -- 172.21.15.31:0/1388577647 shutdown_connections 2024-04-26T17:53:47.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.249+0000 7f3056724700 1 --2- 172.21.15.31:0/1388577647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3050107680 0x7f305010f250 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:47.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.250+0000 7f3056724700 1 --2- 172.21.15.31:0/1388577647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30501025a0 0x7f3050102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:47.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.250+0000 7f3056724700 1 --2- 172.21.15.31:0/1388577647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3050102f50 0x7f3050106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:47.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.250+0000 7f3056724700 1 -- 172.21.15.31:0/1388577647 >> 172.21.15.31:0/1388577647 conn(0x7f30500793c0 msgr2=0x7f30500797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:47.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.250+0000 7f3056724700 1 -- 172.21.15.31:0/1388577647 shutdown_connections 2024-04-26T17:53:47.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.250+0000 7f3056724700 1 -- 172.21.15.31:0/1388577647 wait complete. 2024-04-26T17:53:47.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.250+0000 7f3056724700 1 Processor -- start 2024-04-26T17:53:47.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.250+0000 7f3056724700 1 -- start start 2024-04-26T17:53:47.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f3056724700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30501025a0 0x7f305019bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:47.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f3056724700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3050102f50 0x7f305019c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:47.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f3056724700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3050107680 0x7f30501a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:47.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f3056724700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3050112590 con 0x7f3050102f50 2024-04-26T17:53:47.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f3056724700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3050112410 con 0x7f30501025a0 2024-04-26T17:53:47.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f3056724700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3050112710 con 0x7f3050107680 2024-04-26T17:53:47.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f304f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3050102f50 0x7f305019c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:47.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f304ffff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30501025a0 0x7f305019bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:47.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f304f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3050102f50 0x7f305019c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57468/0 (socket says 172.21.15.31:57468) 2024-04-26T17:53:47.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f304ffff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30501025a0 0x7f305019bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:38840/0 (socket says 172.21.15.31:38840) 2024-04-26T17:53:47.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f304f7fe700 1 -- 172.21.15.31:0/2335259935 learned_addr learned my addr 172.21.15.31:0/2335259935 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:47.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f3054cc1700 1 --2- 172.21.15.31:0/2335259935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3050107680 0x7f30501a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:47.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f304f7fe700 1 -- 172.21.15.31:0/2335259935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3050107680 msgr2=0x7f30501a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:47.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f304f7fe700 1 --2- 172.21.15.31:0/2335259935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3050107680 0x7f30501a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:47.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f304f7fe700 1 -- 172.21.15.31:0/2335259935 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30501025a0 msgr2=0x7f305019bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:47.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f304f7fe700 1 --2- 172.21.15.31:0/2335259935 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30501025a0 0x7f305019bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:47.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f304f7fe700 1 -- 172.21.15.31:0/2335259935 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30501a0d30 con 0x7f3050102f50 2024-04-26T17:53:47.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.251+0000 7f304ffff700 1 --2- 172.21.15.31:0/2335259935 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30501025a0 0x7f305019bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:53:47.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.252+0000 7f304f7fe700 1 --2- 172.21.15.31:0/2335259935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3050102f50 0x7f305019c260 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f304400c8f0 tx=0x7f304400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:47.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.252+0000 7f304d7fa700 1 -- 172.21.15.31:0/2335259935 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3044007940 con 0x7f3050102f50 2024-04-26T17:53:47.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.252+0000 7f304d7fa700 1 -- 172.21.15.31:0/2335259935 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f304400f450 con 0x7f3050102f50 2024-04-26T17:53:47.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.252+0000 7f3056724700 1 -- 172.21.15.31:0/2335259935 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30501a1020 con 0x7f3050102f50 2024-04-26T17:53:47.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.252+0000 7f304d7fa700 1 -- 172.21.15.31:0/2335259935 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3044018660 con 0x7f3050102f50 2024-04-26T17:53:47.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.252+0000 7f3056724700 1 -- 172.21.15.31:0/2335259935 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30501a88a0 con 0x7f3050102f50 2024-04-26T17:53:47.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.253+0000 7f3056724700 1 -- 172.21.15.31:0/2335259935 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f305006a110 con 0x7f3050102f50 2024-04-26T17:53:47.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.254+0000 7f304d7fa700 1 -- 172.21.15.31:0/2335259935 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f304400f890 con 0x7f3050102f50 2024-04-26T17:53:47.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.255+0000 7f304d7fa700 1 --2- 172.21.15.31:0/2335259935 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f30380a3d50 0x7f30380a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:47.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.255+0000 7f304ffff700 1 --2- 172.21.15.31:0/2335259935 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f30380a3d50 0x7f30380a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:47.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.255+0000 7f304d7fa700 1 -- 172.21.15.31:0/2335259935 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f30440d0390 con 0x7f3050102f50 2024-04-26T17:53:47.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.256+0000 7f304ffff700 1 --2- 172.21.15.31:0/2335259935 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f30380a3d50 0x7f30380a6210 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3040009de0 tx=0x7f3040009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:47.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.259+0000 7f304d7fa700 1 -- 172.21.15.31:0/2335259935 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3044014030 con 0x7f3050102f50 2024-04-26T17:53:47.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.471+0000 7f3056724700 1 -- 172.21.15.31:0/2335259935 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f305002e490 con 0x7f3050102f50 2024-04-26T17:53:47.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.473+0000 7f304d7fa700 1 -- 172.21.15.31:0/2335259935 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22190 (secure 0 0 0) 0x7f304409cdd0 con 0x7f3050102f50 2024-04-26T17:53:47.472 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:53:47.472 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":88,"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","created":"2024-04-26T17:44:48.618250+0000","modified":"2024-04-26T17:53:05.425953+0000","last_up_change":"2024-04-26T17:52:00.798886+0000","last_in_change":"2024-04-26T17:51:40.613593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T17:48:12.307250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T17:52:58.650338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T17:53:02.488144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f684ee5-eeee-454a-a00b-e5d6d2901572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6803","nonce":1957398328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6805","nonce":1957398328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6809","nonce":1957398328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6807","nonce":1957398328}]},"public_addr":"172.21.15.31:6803/1957398328","cluster_addr":"172.21.15.31:6805/1957398328","heartbeat_back_addr":"172.21.15.31:6809/1957398328","heartbeat_front_addr":"172.21.15.31:6807/1957398328","state":["exists","up"]},{"osd":1,"uuid":"08be6b26-cb60-41f7-be8b-01a0c93ebafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6801","nonce":4209866427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6803","nonce":4209866427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6807","nonce":4209866427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6805","nonce":4209866427}]},"public_addr":"172.21.15.97:6801/4209866427","cluster_addr":"172.21.15.97:6803/4209866427","heartbeat_back_addr":"172.21.15.97:6807/4209866427","heartbeat_front_addr":"172.21.15.97:6805/4209866427","state":["exists","up"]},{"osd":2,"uuid":"498216b1-a0ac-4923-ade8-ce50fb97d472","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6801","nonce":223535147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6803","nonce":223535147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6807","nonce":223535147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6805","nonce":223535147}]},"public_addr":"172.21.15.186:6801/223535147","cluster_addr":"172.21.15.186:6803/223535147","heartbeat_back_addr":"172.21.15.186:6807/223535147","heartbeat_front_addr":"172.21.15.186:6805/223535147","state":["exists","up"]},{"osd":3,"uuid":"568a352d-a430-4f0f-9a49-6f992e0ceae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6811","nonce":2640112341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6813","nonce":2640112341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6817","nonce":2640112341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6815","nonce":2640112341}]},"public_addr":"172.21.15.31:6811/2640112341","cluster_addr":"172.21.15.31:6813/2640112341","heartbeat_back_addr":"172.21.15.31:6817/2640112341","heartbeat_front_addr":"172.21.15.31:6815/2640112341","state":["exists","up"]},{"osd":4,"uuid":"112ccbf7-24e8-473c-9852-2e7167dd3108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6809","nonce":1359263143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6811","nonce":1359263143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6815","nonce":1359263143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6813","nonce":1359263143}]},"public_addr":"172.21.15.97:6809/1359263143","cluster_addr":"172.21.15.97:6811/1359263143","heartbeat_back_addr":"172.21.15.97:6815/1359263143","heartbeat_front_addr":"172.21.15.97:6813/1359263143","state":["exists","up"]},{"osd":5,"uuid":"eb17759e-1ad8-4bda-9f61-54d2a9781535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6809","nonce":2006134626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6811","nonce":2006134626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6815","nonce":2006134626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6813","nonce":2006134626}]},"public_addr":"172.21.15.186:6809/2006134626","cluster_addr":"172.21.15.186:6811/2006134626","heartbeat_back_addr":"172.21.15.186:6815/2006134626","heartbeat_front_addr":"172.21.15.186:6813/2006134626","state":["exists","up"]},{"osd":6,"uuid":"f1d11e38-40e9-4db4-868f-145f1ffc0d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6819","nonce":2955759395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6821","nonce":2955759395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6825","nonce":2955759395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6823","nonce":2955759395}]},"public_addr":"172.21.15.31:6819/2955759395","cluster_addr":"172.21.15.31:6821/2955759395","heartbeat_back_addr":"172.21.15.31:6825/2955759395","heartbeat_front_addr":"172.21.15.31:6823/2955759395","state":["exists","up"]},{"osd":7,"uuid":"4e95f728-23ff-4bca-9b94-afde2662a5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6817","nonce":3940083075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6819","nonce":3940083075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6823","nonce":3940083075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6821","nonce":3940083075}]},"public_addr":"172.21.15.97:6817/3940083075","cluster_addr":"172.21.15.97:6819/3940083075","heartbeat_back_addr":"172.21.15.97:6823/3940083075","heartbeat_front_addr":"172.21.15.97:6821/3940083075","state":["exists","up"]},{"osd":8,"uuid":"5f91c915-2568-4d20-b645-00dad2efe952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6817","nonce":3193558170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6819","nonce":3193558170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6823","nonce":3193558170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6821","nonce":3193558170}]},"public_addr":"172.21.15.186:6817/3193558170","cluster_addr":"172.21.15.186:6819/3193558170","heartbeat_back_addr":"172.21.15.186:6823/3193558170","heartbeat_front_addr":"172.21.15.186:6821/3193558170","state":["exists","up"]},{"osd":9,"uuid":"12fda87d-830a-461f-a6cb-4699c52dc5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6827","nonce":3660801905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6829","nonce":3660801905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6833","nonce":3660801905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6831","nonce":3660801905}]},"public_addr":"172.21.15.31:6827/3660801905","cluster_addr":"172.21.15.31:6829/3660801905","heartbeat_back_addr":"172.21.15.31:6833/3660801905","heartbeat_front_addr":"172.21.15.31:6831/3660801905","state":["exists","up"]},{"osd":10,"uuid":"92a639b3-918b-485c-bd30-258b4224993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6825","nonce":4097344903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6827","nonce":4097344903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6831","nonce":4097344903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6829","nonce":4097344903}]},"public_addr":"172.21.15.97:6825/4097344903","cluster_addr":"172.21.15.97:6827/4097344903","heartbeat_back_addr":"172.21.15.97:6831/4097344903","heartbeat_front_addr":"172.21.15.97:6829/4097344903","state":["exists","up"]},{"osd":11,"uuid":"842aa599-d21d-4e4a-b872-e9ac20d8c0ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6825","nonce":1063973352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6827","nonce":1063973352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6831","nonce":1063973352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6829","nonce":1063973352}]},"public_addr":"172.21.15.186:6825/1063973352","cluster_addr":"172.21.15.186:6827/1063973352","heartbeat_back_addr":"172.21.15.186:6831/1063973352","heartbeat_front_addr":"172.21.15.186:6829/1063973352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:21.305331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:44.877932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:08.784665+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:34.381762+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:58.999358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:23.620237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:49.961719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:15.363702+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:40.599703+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:07.391645+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:32.800851+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:59.384148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/718966776":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/2035551671":"2024-04-27T17:45:48.218816+0000","172.21.15.31:6801/153341213":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/810461143":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/3148082955":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6801/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/288292084":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/3101596726":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/153341213":"2024-04-27T17:45:48.218816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T17:53:47.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.475+0000 7f3056724700 1 -- 172.21.15.31:0/2335259935 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f30380a3d50 msgr2=0x7f30380a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:47.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.475+0000 7f3056724700 1 --2- 172.21.15.31:0/2335259935 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f30380a3d50 0x7f30380a6210 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3040009de0 tx=0x7f3040009190 comp rx=0 tx=0).stop 2024-04-26T17:53:47.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.475+0000 7f3056724700 1 -- 172.21.15.31:0/2335259935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3050102f50 msgr2=0x7f305019c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:47.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.475+0000 7f3056724700 1 --2- 172.21.15.31:0/2335259935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3050102f50 0x7f305019c260 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f304400c8f0 tx=0x7f304400cc00 comp rx=0 tx=0).stop 2024-04-26T17:53:47.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.475+0000 7f3056724700 1 -- 172.21.15.31:0/2335259935 shutdown_connections 2024-04-26T17:53:47.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.475+0000 7f3056724700 1 --2- 172.21.15.31:0/2335259935 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f30380a3d50 0x7f30380a6210 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:47.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.475+0000 7f3056724700 1 --2- 172.21.15.31:0/2335259935 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30501025a0 0x7f305019bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:47.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.475+0000 7f3056724700 1 --2- 172.21.15.31:0/2335259935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3050107680 0x7f30501a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:47.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.475+0000 7f3056724700 1 --2- 172.21.15.31:0/2335259935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3050102f50 0x7f305019c260 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:47.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.475+0000 7f3056724700 1 -- 172.21.15.31:0/2335259935 >> 172.21.15.31:0/2335259935 conn(0x7f30500793c0 msgr2=0x7f305006c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:47.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.475+0000 7f3056724700 1 -- 172.21.15.31:0/2335259935 shutdown_connections 2024-04-26T17:53:47.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:47.475+0000 7f3056724700 1 -- 172.21.15.31:0/2335259935 wait complete. 2024-04-26T17:53:47.837 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:47 smithi031 ceph-mon[29164]: pgmap v317: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-04-26T17:53:47.837 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:47 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2335259935' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:53:48.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:47 smithi186 ceph-mon[32797]: pgmap v317: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-04-26T17:53:48.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:47 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2335259935' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:53:48.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:47 smithi097 ceph-mon[32419]: pgmap v317: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-04-26T17:53:48.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:47 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2335259935' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:53:48.168 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T17:53:48.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T17:53:48.420 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:48.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-26T17:53:48.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-26T17:53:48.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-26T17:53:48.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-04-26T17:53:48.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-04-26T17:53:48.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-04-26T17:53:48.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:48 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-26T17:53:49.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-26T17:53:49.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-26T17:53:49.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-26T17:53:49.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-04-26T17:53:49.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-04-26T17:53:49.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-04-26T17:53:49.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:48 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-26T17:53:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-26T17:53:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-26T17:53:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-26T17:53:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-04-26T17:53:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-04-26T17:53:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-04-26T17:53:49.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:48 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-26T17:53:49.880 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:49 smithi031 ceph-mon[29164]: pgmap v318: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-04-26T17:53:49.880 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-26T17:53:49.880 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-26T17:53:49.880 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-26T17:53:49.880 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-04-26T17:53:49.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-04-26T17:53:49.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-04-26T17:53:49.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:49 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-26T17:53:49.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:49 smithi031 ceph-mon[29164]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T17:53:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:49 smithi186 ceph-mon[32797]: pgmap v318: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-04-26T17:53:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-26T17:53:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-26T17:53:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-26T17:53:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-04-26T17:53:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-04-26T17:53:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-04-26T17:53:50.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:49 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-26T17:53:50.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:49 smithi186 ceph-mon[32797]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T17:53:50.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.113+0000 7fa1e583a700 1 -- 172.21.15.31:0/73655489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1e006bbb0 msgr2=0x7fa1e006c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:50.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.113+0000 7fa1e583a700 1 --2- 172.21.15.31:0/73655489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1e006bbb0 0x7fa1e006c010 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa1d40099c0 tx=0x7fa1d4009cd0 comp rx=0 tx=0).stop 2024-04-26T17:53:50.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.113+0000 7fa1e583a700 1 -- 172.21.15.31:0/73655489 shutdown_connections 2024-04-26T17:53:50.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.113+0000 7fa1e583a700 1 --2- 172.21.15.31:0/73655489 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e006c550 0x7fa1e01102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:50.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.113+0000 7fa1e583a700 1 --2- 172.21.15.31:0/73655489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1e01094b0 0x7fa1e0109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:50.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.113+0000 7fa1e583a700 1 --2- 172.21.15.31:0/73655489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1e006bbb0 0x7fa1e006c010 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:50.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.113+0000 7fa1e583a700 1 -- 172.21.15.31:0/73655489 >> 172.21.15.31:0/73655489 conn(0x7fa1e0079250 msgr2=0x7fa1e0079660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:50.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.113+0000 7fa1e583a700 1 -- 172.21.15.31:0/73655489 shutdown_connections 2024-04-26T17:53:50.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.113+0000 7fa1e583a700 1 -- 172.21.15.31:0/73655489 wait complete. 2024-04-26T17:53:50.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.114+0000 7fa1e583a700 1 Processor -- start 2024-04-26T17:53:50.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.114+0000 7fa1e583a700 1 -- start start 2024-04-26T17:53:50.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.114+0000 7fa1e583a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1e006bbb0 0x7fa1e019f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.114+0000 7fa1e583a700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e006c550 0x7fa1e019fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.114+0000 7fa1e583a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1e01094b0 0x7fa1e01999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.114+0000 7fa1e583a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e01136a0 con 0x7fa1e01094b0 2024-04-26T17:53:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.114+0000 7fa1e583a700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e0113520 con 0x7fa1e006c550 2024-04-26T17:53:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.114+0000 7fa1e583a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e0113820 con 0x7fa1e006bbb0 2024-04-26T17:53:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.114+0000 7fa1df7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1e01094b0 0x7fa1e01999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.114+0000 7fa1df7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1e01094b0 0x7fa1e01999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57484/0 (socket says 172.21.15.31:57484) 2024-04-26T17:53:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.114+0000 7fa1df7fe700 1 -- 172.21.15.31:0/2117562946 learned_addr learned my addr 172.21.15.31:0/2117562946 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1de7fc700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e006c550 0x7fa1e019fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1deffd700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1e006bbb0 0x7fa1e019f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1df7fe700 1 -- 172.21.15.31:0/2117562946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1e006bbb0 msgr2=0x7fa1e019f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1df7fe700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1e006bbb0 0x7fa1e019f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1df7fe700 1 -- 172.21.15.31:0/2117562946 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e006c550 msgr2=0x7fa1e019fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1df7fe700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e006c550 0x7fa1e019fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1df7fe700 1 -- 172.21.15.31:0/2117562946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1e019a140 con 0x7fa1e01094b0 2024-04-26T17:53:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1de7fc700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e006c550 0x7fa1e019fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:53:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1deffd700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1e006bbb0 0x7fa1e019f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:53:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1df7fe700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1e01094b0 0x7fa1e01999e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800b970 tx=0x7fa1d800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1e4838700 1 -- 172.21.15.31:0/2117562946 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8019070 con 0x7fa1e01094b0 2024-04-26T17:53:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1e583a700 1 -- 172.21.15.31:0/2117562946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1e019a430 con 0x7fa1e01094b0 2024-04-26T17:53:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1e4838700 1 -- 172.21.15.31:0/2117562946 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1d8014440 con 0x7fa1e01094b0 2024-04-26T17:53:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1e4838700 1 -- 172.21.15.31:0/2117562946 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d800ce60 con 0x7fa1e01094b0 2024-04-26T17:53:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.115+0000 7fa1e583a700 1 -- 172.21.15.31:0/2117562946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1e01a66f0 con 0x7fa1e01094b0 2024-04-26T17:53:50.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.116+0000 7fa1e583a700 1 -- 172.21.15.31:0/2117562946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1e006a110 con 0x7fa1e01094b0 2024-04-26T17:53:50.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.118+0000 7fa1e4838700 1 -- 172.21.15.31:0/2117562946 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa1d801c070 con 0x7fa1e01094b0 2024-04-26T17:53:50.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.118+0000 7fa1e4838700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa1c80a3cd0 0x7fa1c80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:50.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.119+0000 7fa1deffd700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa1c80a3cd0 0x7fa1c80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:50.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.119+0000 7fa1e4838700 1 -- 172.21.15.31:0/2117562946 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fa1d80cfdc0 con 0x7fa1e01094b0 2024-04-26T17:53:50.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.119+0000 7fa1deffd700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa1c80a3cd0 0x7fa1c80a6190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa1e019ab20 tx=0x7fa1d0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:50.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.122+0000 7fa1e4838700 1 -- 172.21.15.31:0/2117562946 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa1d809c780 con 0x7fa1e01094b0 2024-04-26T17:53:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:49 smithi097 ceph-mon[32419]: pgmap v318: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-04-26T17:53:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-26T17:53:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-26T17:53:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-26T17:53:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-04-26T17:53:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-04-26T17:53:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-04-26T17:53:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:49 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-26T17:53:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:49 smithi097 ceph-mon[32419]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T17:53:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.325+0000 7fa1e583a700 1 -- 172.21.15.31:0/2117562946 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa1e0063860 con 0x7fa1c80a3cd0 2024-04-26T17:53:50.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.332+0000 7fa1e4838700 1 -- 172.21.15.31:0/2117562946 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258711 (secure 0 0 0) 0x7fa1e0063860 con 0x7fa1c80a3cd0 2024-04-26T17:53:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:53:50.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.335+0000 7fa1e583a700 1 -- 172.21.15.31:0/2117562946 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa1c80a3cd0 msgr2=0x7fa1c80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.335+0000 7fa1e583a700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa1c80a3cd0 0x7fa1c80a6190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa1e019ab20 tx=0x7fa1d0008040 comp rx=0 tx=0).stop 2024-04-26T17:53:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.335+0000 7fa1e583a700 1 -- 172.21.15.31:0/2117562946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1e01094b0 msgr2=0x7fa1e01999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.335+0000 7fa1e583a700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1e01094b0 0x7fa1e01999e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800b970 tx=0x7fa1d800bd30 comp rx=0 tx=0).stop 2024-04-26T17:53:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.335+0000 7fa1e583a700 1 -- 172.21.15.31:0/2117562946 shutdown_connections 2024-04-26T17:53:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.335+0000 7fa1e583a700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa1c80a3cd0 0x7fa1c80a6190 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.335+0000 7fa1e583a700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1e01094b0 0x7fa1e01999e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.335+0000 7fa1e583a700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1e006bbb0 0x7fa1e019f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.335+0000 7fa1e583a700 1 --2- 172.21.15.31:0/2117562946 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1e006c550 0x7fa1e019fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.335+0000 7fa1e583a700 1 -- 172.21.15.31:0/2117562946 >> 172.21.15.31:0/2117562946 conn(0x7fa1e0079250 msgr2=0x7fa1e0100510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.335+0000 7fa1e583a700 1 -- 172.21.15.31:0/2117562946 shutdown_connections 2024-04-26T17:53:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:50.335+0000 7fa1e583a700 1 -- 172.21.15.31:0/2117562946 wait complete. 2024-04-26T17:53:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-26T17:53:51.010 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-04-26T17:53:50.317530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652681,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":307,"num_read_kb":266,"num_write":347,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360864,"kb_used_data":11708,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580960,"statfs":{"total":1151940427776,"available":1151570903040,"internally_reserved":0,"allocated":11988992,"data_stored":7502298,"data_compressed":106863,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5395,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":31,"num_write":11,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002225"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.616857+0000","last_change":"2024-04-26T17:53:03.616592+0000","last_active":"2024-04-26T17:53:05.616857+0000","last_peered":"2024-04-26T17:53:05.616857+0000","last_clean":"2024-04-26T17:53:05.616857+0000","last_became_active":"2024-04-26T17:53:03.616255+0000","last_became_peered":"2024-04-26T17:53:03.616255+0000","last_unstale":"2024-04-26T17:53:05.616857+0000","last_undegraded":"2024-04-26T17:53:05.616857+0000","last_fullsized":"2024-04-26T17:53:05.616857+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:43:31.183679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.759387+0000","last_change":"2024-04-26T17:53:03.615402+0000","last_active":"2024-04-26T17:53:48.759387+0000","last_peered":"2024-04-26T17:53:48.759387+0000","last_clean":"2024-04-26T17:53:48.759387+0000","last_became_active":"2024-04-26T17:53:03.615048+0000","last_became_peered":"2024-04-26T17:53:03.615048+0000","last_unstale":"2024-04-26T17:53:48.759387+0000","last_undegraded":"2024-04-26T17:53:48.759387+0000","last_fullsized":"2024-04-26T17:53:48.759387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:05:12.454770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.759086+0000","last_change":"2024-04-26T17:53:03.622625+0000","last_active":"2024-04-26T17:53:48.759086+0000","last_peered":"2024-04-26T17:53:48.759086+0000","last_clean":"2024-04-26T17:53:48.759086+0000","last_became_active":"2024-04-26T17:53:03.622305+0000","last_became_peered":"2024-04-26T17:53:03.622305+0000","last_unstale":"2024-04-26T17:53:48.759086+0000","last_undegraded":"2024-04-26T17:53:48.759086+0000","last_fullsized":"2024-04-26T17:53:48.759086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:06.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.628525+0000","last_change":"2024-04-26T17:53:03.617650+0000","last_active":"2024-04-26T17:53:05.628525+0000","last_peered":"2024-04-26T17:53:05.628525+0000","last_clean":"2024-04-26T17:53:05.628525+0000","last_became_active":"2024-04-26T17:53:03.617220+0000","last_became_peered":"2024-04-26T17:53:03.617220+0000","last_unstale":"2024-04-26T17:53:05.628525+0000","last_undegraded":"2024-04-26T17:53:05.628525+0000","last_fullsized":"2024-04-26T17:53:05.628525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:31.765577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.624153+0000","last_change":"2024-04-26T17:53:03.615429+0000","last_active":"2024-04-26T17:53:05.624153+0000","last_peered":"2024-04-26T17:53:05.624153+0000","last_clean":"2024-04-26T17:53:05.624153+0000","last_became_active":"2024-04-26T17:53:03.615079+0000","last_became_peered":"2024-04-26T17:53:03.615079+0000","last_unstale":"2024-04-26T17:53:05.624153+0000","last_undegraded":"2024-04-26T17:53:05.624153+0000","last_fullsized":"2024-04-26T17:53:05.624153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:20.994016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.617071+0000","last_change":"2024-04-26T17:53:03.624619+0000","last_active":"2024-04-26T17:53:05.617071+0000","last_peered":"2024-04-26T17:53:05.617071+0000","last_clean":"2024-04-26T17:53:05.617071+0000","last_became_active":"2024-04-26T17:53:03.624251+0000","last_became_peered":"2024-04-26T17:53:03.624251+0000","last_unstale":"2024-04-26T17:53:05.617071+0000","last_undegraded":"2024-04-26T17:53:05.617071+0000","last_fullsized":"2024-04-26T17:53:05.617071+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:50:17.535235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623022+0000","last_change":"2024-04-26T17:53:03.619325+0000","last_active":"2024-04-26T17:53:05.623022+0000","last_peered":"2024-04-26T17:53:05.623022+0000","last_clean":"2024-04-26T17:53:05.623022+0000","last_became_active":"2024-04-26T17:53:03.618989+0000","last_became_peered":"2024-04-26T17:53:03.618989+0000","last_unstale":"2024-04-26T17:53:05.623022+0000","last_undegraded":"2024-04-26T17:53:05.623022+0000","last_fullsized":"2024-04-26T17:53:05.623022+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:10.195742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.625958+0000","last_change":"2024-04-26T17:53:03.609346+0000","last_active":"2024-04-26T17:53:05.625958+0000","last_peered":"2024-04-26T17:53:05.625958+0000","last_clean":"2024-04-26T17:53:05.625958+0000","last_became_active":"2024-04-26T17:53:03.609021+0000","last_became_peered":"2024-04-26T17:53:03.609021+0000","last_unstale":"2024-04-26T17:53:05.625958+0000","last_undegraded":"2024-04-26T17:53:05.625958+0000","last_fullsized":"2024-04-26T17:53:05.625958+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:44.328599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.624266+0000","last_change":"2024-04-26T17:53:03.618573+0000","last_active":"2024-04-26T17:53:05.624266+0000","last_peered":"2024-04-26T17:53:05.624266+0000","last_clean":"2024-04-26T17:53:05.624266+0000","last_became_active":"2024-04-26T17:53:03.618223+0000","last_became_peered":"2024-04-26T17:53:03.618223+0000","last_unstale":"2024-04-26T17:53:05.624266+0000","last_undegraded":"2024-04-26T17:53:05.624266+0000","last_fullsized":"2024-04-26T17:53:05.624266+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:16.749205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.617278+0000","last_change":"2024-04-26T17:53:03.621530+0000","last_active":"2024-04-26T17:53:05.617278+0000","last_peered":"2024-04-26T17:53:05.617278+0000","last_clean":"2024-04-26T17:53:05.617278+0000","last_became_active":"2024-04-26T17:53:03.621174+0000","last_became_peered":"2024-04-26T17:53:03.621174+0000","last_unstale":"2024-04-26T17:53:05.617278+0000","last_undegraded":"2024-04-26T17:53:05.617278+0000","last_fullsized":"2024-04-26T17:53:05.617278+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:31.388232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.624033+0000","last_change":"2024-04-26T17:53:03.617682+0000","last_active":"2024-04-26T17:53:05.624033+0000","last_peered":"2024-04-26T17:53:05.624033+0000","last_clean":"2024-04-26T17:53:05.624033+0000","last_became_active":"2024-04-26T17:53:03.617348+0000","last_became_peered":"2024-04-26T17:53:03.617348+0000","last_unstale":"2024-04-26T17:53:05.624033+0000","last_undegraded":"2024-04-26T17:53:05.624033+0000","last_fullsized":"2024-04-26T17:53:05.624033+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:21.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.628330+0000","last_change":"2024-04-26T17:53:03.617955+0000","last_active":"2024-04-26T17:53:05.628330+0000","last_peered":"2024-04-26T17:53:05.628330+0000","last_clean":"2024-04-26T17:53:05.628330+0000","last_became_active":"2024-04-26T17:53:03.616765+0000","last_became_peered":"2024-04-26T17:53:03.616765+0000","last_unstale":"2024-04-26T17:53:05.628330+0000","last_undegraded":"2024-04-26T17:53:05.628330+0000","last_fullsized":"2024-04-26T17:53:05.628330+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:59.292277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.622873+0000","last_change":"2024-04-26T17:53:03.613070+0000","last_active":"2024-04-26T17:53:05.622873+0000","last_peered":"2024-04-26T17:53:05.622873+0000","last_clean":"2024-04-26T17:53:05.622873+0000","last_became_active":"2024-04-26T17:53:03.612263+0000","last_became_peered":"2024-04-26T17:53:03.612263+0000","last_unstale":"2024-04-26T17:53:05.622873+0000","last_undegraded":"2024-04-26T17:53:05.622873+0000","last_fullsized":"2024-04-26T17:53:05.622873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:55.329158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.759305+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:53:48.759305+0000","last_peered":"2024-04-26T17:53:48.759305+0000","last_clean":"2024-04-26T17:53:48.759305+0000","last_became_active":"2024-04-26T17:53:03.616623+0000","last_became_peered":"2024-04-26T17:53:03.616623+0000","last_unstale":"2024-04-26T17:53:48.759305+0000","last_undegraded":"2024-04-26T17:53:48.759305+0000","last_fullsized":"2024-04-26T17:53:48.759305+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:04.231549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.758706+0000","last_change":"2024-04-26T17:53:03.622147+0000","last_active":"2024-04-26T17:53:48.758706+0000","last_peered":"2024-04-26T17:53:48.758706+0000","last_clean":"2024-04-26T17:53:48.758706+0000","last_became_active":"2024-04-26T17:53:03.621542+0000","last_became_peered":"2024-04-26T17:53:03.621542+0000","last_unstale":"2024-04-26T17:53:48.758706+0000","last_undegraded":"2024-04-26T17:53:48.758706+0000","last_fullsized":"2024-04-26T17:53:48.758706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:01.746954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623200+0000","last_change":"2024-04-26T17:53:03.624433+0000","last_active":"2024-04-26T17:53:05.623200+0000","last_peered":"2024-04-26T17:53:05.623200+0000","last_clean":"2024-04-26T17:53:05.623200+0000","last_became_active":"2024-04-26T17:53:03.624135+0000","last_became_peered":"2024-04-26T17:53:03.624135+0000","last_unstale":"2024-04-26T17:53:05.623200+0000","last_undegraded":"2024-04-26T17:53:05.623200+0000","last_fullsized":"2024-04-26T17:53:05.623200+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:32:20.521917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.617023+0000","last_change":"2024-04-26T17:53:03.623607+0000","last_active":"2024-04-26T17:53:05.617023+0000","last_peered":"2024-04-26T17:53:05.617023+0000","last_clean":"2024-04-26T17:53:05.617023+0000","last_became_active":"2024-04-26T17:53:03.621783+0000","last_became_peered":"2024-04-26T17:53:03.621783+0000","last_unstale":"2024-04-26T17:53:05.617023+0000","last_undegraded":"2024-04-26T17:53:05.617023+0000","last_fullsized":"2024-04-26T17:53:05.617023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:38:59.872868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.756636+0000","last_change":"2024-04-26T17:53:03.614974+0000","last_active":"2024-04-26T17:53:48.756636+0000","last_peered":"2024-04-26T17:53:48.756636+0000","last_clean":"2024-04-26T17:53:48.756636+0000","last_became_active":"2024-04-26T17:53:03.614619+0000","last_became_peered":"2024-04-26T17:53:03.614619+0000","last_unstale":"2024-04-26T17:53:48.756636+0000","last_undegraded":"2024-04-26T17:53:48.756636+0000","last_fullsized":"2024-04-26T17:53:48.756636+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:50.729531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623336+0000","last_change":"2024-04-26T17:53:03.611140+0000","last_active":"2024-04-26T17:53:05.623336+0000","last_peered":"2024-04-26T17:53:05.623336+0000","last_clean":"2024-04-26T17:53:05.623336+0000","last_became_active":"2024-04-26T17:53:03.610549+0000","last_became_peered":"2024-04-26T17:53:03.610549+0000","last_unstale":"2024-04-26T17:53:05.623336+0000","last_undegraded":"2024-04-26T17:53:05.623336+0000","last_fullsized":"2024-04-26T17:53:05.623336+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:46:17.178598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.756562+0000","last_change":"2024-04-26T17:53:00.585971+0000","last_active":"2024-04-26T17:53:48.756562+0000","last_peered":"2024-04-26T17:53:48.756562+0000","last_clean":"2024-04-26T17:53:48.756562+0000","last_became_active":"2024-04-26T17:53:00.585648+0000","last_became_peered":"2024-04-26T17:53:00.585648+0000","last_unstale":"2024-04-26T17:53:48.756562+0000","last_undegraded":"2024-04-26T17:53:48.756562+0000","last_fullsized":"2024-04-26T17:53:48.756562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:00:24.082773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":2,"reported_epoch":89,"state":"peering","last_fresh":"2024-04-26T17:53:03.598208+0000","last_change":"2024-04-26T17:53:49.721525+0000","last_active":"2024-04-26T17:53:02.586248+0000","last_peered":"2024-04-26T17:53:02.586248+0000","last_clean":"2024-04-26T17:53:02.586248+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T17:53:03.598208+0000","last_undegraded":"2024-04-26T17:53:03.598208+0000","last_fullsized":"2024-04-26T17:53:03.598208+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.633194+0000","last_change":"2024-04-26T17:53:01.145438+0000","last_active":"2024-04-26T17:53:05.633194+0000","last_peered":"2024-04-26T17:53:05.633194+0000","last_clean":"2024-04-26T17:53:05.633194+0000","last_became_active":"2024-04-26T17:53:01.145030+0000","last_became_peered":"2024-04-26T17:53:01.145030+0000","last_unstale":"2024-04-26T17:53:05.633194+0000","last_undegraded":"2024-04-26T17:53:05.633194+0000","last_fullsized":"2024-04-26T17:53:05.633194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:15.650247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.759766+0000","last_change":"2024-04-26T17:53:03.622647+0000","last_active":"2024-04-26T17:53:48.759766+0000","last_peered":"2024-04-26T17:53:48.759766+0000","last_clean":"2024-04-26T17:53:48.759766+0000","last_became_active":"2024-04-26T17:53:03.622298+0000","last_became_peered":"2024-04-26T17:53:03.622298+0000","last_unstale":"2024-04-26T17:53:48.759766+0000","last_undegraded":"2024-04-26T17:53:48.759766+0000","last_fullsized":"2024-04-26T17:53:48.759766+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:37.230539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:19.687587+0000","last_change":"2024-04-26T17:53:00.588285+0000","last_active":"2024-04-26T17:53:19.687587+0000","last_peered":"2024-04-26T17:53:19.687587+0000","last_clean":"2024-04-26T17:53:19.687587+0000","last_became_active":"2024-04-26T17:53:00.586194+0000","last_became_peered":"2024-04-26T17:53:00.586194+0000","last_unstale":"2024-04-26T17:53:19.687587+0000","last_undegraded":"2024-04-26T17:53:19.687587+0000","last_fullsized":"2024-04-26T17:53:19.687587+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:55:28.871587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.624551+0000","last_change":"2024-04-26T17:53:03.617462+0000","last_active":"2024-04-26T17:53:05.624551+0000","last_peered":"2024-04-26T17:53:05.624551+0000","last_clean":"2024-04-26T17:53:05.624551+0000","last_became_active":"2024-04-26T17:53:03.617115+0000","last_became_peered":"2024-04-26T17:53:03.617115+0000","last_unstale":"2024-04-26T17:53:05.624551+0000","last_undegraded":"2024-04-26T17:53:05.624551+0000","last_fullsized":"2024-04-26T17:53:05.624551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:11.406146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.632834+0000","last_change":"2024-04-26T17:53:00.583378+0000","last_active":"2024-04-26T17:53:05.632834+0000","last_peered":"2024-04-26T17:53:05.632834+0000","last_clean":"2024-04-26T17:53:05.632834+0000","last_became_active":"2024-04-26T17:53:00.583020+0000","last_became_peered":"2024-04-26T17:53:00.583020+0000","last_unstale":"2024-04-26T17:53:05.632834+0000","last_undegraded":"2024-04-26T17:53:05.632834+0000","last_fullsized":"2024-04-26T17:53:05.632834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:24.089996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.629325+0000","last_change":"2024-04-26T17:53:03.609557+0000","last_active":"2024-04-26T17:53:05.629325+0000","last_peered":"2024-04-26T17:53:05.629325+0000","last_clean":"2024-04-26T17:53:05.629325+0000","last_became_active":"2024-04-26T17:53:03.609028+0000","last_became_peered":"2024-04-26T17:53:03.609028+0000","last_unstale":"2024-04-26T17:53:05.629325+0000","last_undegraded":"2024-04-26T17:53:05.629325+0000","last_fullsized":"2024-04-26T17:53:05.629325+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:39:22.860877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:43.729868+0000","last_change":"2024-04-26T17:53:00.586683+0000","last_active":"2024-04-26T17:53:43.729868+0000","last_peered":"2024-04-26T17:53:43.729868+0000","last_clean":"2024-04-26T17:53:43.729868+0000","last_became_active":"2024-04-26T17:53:00.586272+0000","last_became_peered":"2024-04-26T17:53:00.586272+0000","last_unstale":"2024-04-26T17:53:43.729868+0000","last_undegraded":"2024-04-26T17:53:43.729868+0000","last_fullsized":"2024-04-26T17:53:43.729868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:20:29.068607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623469+0000","last_change":"2024-04-26T17:53:03.611073+0000","last_active":"2024-04-26T17:53:05.623469+0000","last_peered":"2024-04-26T17:53:05.623469+0000","last_clean":"2024-04-26T17:53:05.623469+0000","last_became_active":"2024-04-26T17:53:03.610701+0000","last_became_peered":"2024-04-26T17:53:03.610701+0000","last_unstale":"2024-04-26T17:53:05.623469+0000","last_undegraded":"2024-04-26T17:53:05.623469+0000","last_fullsized":"2024-04-26T17:53:05.623469+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:01:00.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:19.688687+0000","last_change":"2024-04-26T17:53:00.586082+0000","last_active":"2024-04-26T17:53:19.688687+0000","last_peered":"2024-04-26T17:53:19.688687+0000","last_clean":"2024-04-26T17:53:19.688687+0000","last_became_active":"2024-04-26T17:53:00.585678+0000","last_became_peered":"2024-04-26T17:53:00.585678+0000","last_unstale":"2024-04-26T17:53:19.688687+0000","last_undegraded":"2024-04-26T17:53:19.688687+0000","last_fullsized":"2024-04-26T17:53:19.688687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:55:25.523846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.628134+0000","last_change":"2024-04-26T17:53:03.600230+0000","last_active":"2024-04-26T17:53:05.628134+0000","last_peered":"2024-04-26T17:53:05.628134+0000","last_clean":"2024-04-26T17:53:05.628134+0000","last_became_active":"2024-04-26T17:53:03.599713+0000","last_became_peered":"2024-04-26T17:53:03.599713+0000","last_unstale":"2024-04-26T17:53:05.628134+0000","last_undegraded":"2024-04-26T17:53:05.628134+0000","last_fullsized":"2024-04-26T17:53:05.628134+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:45.822561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:19.687458+0000","last_change":"2024-04-26T17:53:00.584588+0000","last_active":"2024-04-26T17:53:19.687458+0000","last_peered":"2024-04-26T17:53:19.687458+0000","last_clean":"2024-04-26T17:53:19.687458+0000","last_became_active":"2024-04-26T17:53:00.584284+0000","last_became_peered":"2024-04-26T17:53:00.584284+0000","last_unstale":"2024-04-26T17:53:19.687458+0000","last_undegraded":"2024-04-26T17:53:19.687458+0000","last_fullsized":"2024-04-26T17:53:19.687458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:08.428904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.626727+0000","last_change":"2024-04-26T17:53:03.615237+0000","last_active":"2024-04-26T17:53:05.626727+0000","last_peered":"2024-04-26T17:53:05.626727+0000","last_clean":"2024-04-26T17:53:05.626727+0000","last_became_active":"2024-04-26T17:53:03.614704+0000","last_became_peered":"2024-04-26T17:53:03.614704+0000","last_unstale":"2024-04-26T17:53:05.626727+0000","last_undegraded":"2024-04-26T17:53:05.626727+0000","last_fullsized":"2024-04-26T17:53:05.626727+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:27.251839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.628186+0000","last_change":"2024-04-26T17:53:00.589287+0000","last_active":"2024-04-26T17:53:05.628186+0000","last_peered":"2024-04-26T17:53:05.628186+0000","last_clean":"2024-04-26T17:53:05.628186+0000","last_became_active":"2024-04-26T17:53:00.588119+0000","last_became_peered":"2024-04-26T17:53:00.588119+0000","last_unstale":"2024-04-26T17:53:05.628186+0000","last_undegraded":"2024-04-26T17:53:05.628186+0000","last_fullsized":"2024-04-26T17:53:05.628186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:25.389777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.756432+0000","last_change":"2024-04-26T17:53:03.608061+0000","last_active":"2024-04-26T17:53:48.756432+0000","last_peered":"2024-04-26T17:53:48.756432+0000","last_clean":"2024-04-26T17:53:48.756432+0000","last_became_active":"2024-04-26T17:53:03.607624+0000","last_became_peered":"2024-04-26T17:53:03.607624+0000","last_unstale":"2024-04-26T17:53:48.756432+0000","last_undegraded":"2024-04-26T17:53:48.756432+0000","last_fullsized":"2024-04-26T17:53:48.756432+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:15.111011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:44.730528+0000","last_change":"2024-04-26T17:53:00.588621+0000","last_active":"2024-04-26T17:53:44.730528+0000","last_peered":"2024-04-26T17:53:44.730528+0000","last_clean":"2024-04-26T17:53:44.730528+0000","last_became_active":"2024-04-26T17:53:00.587330+0000","last_became_peered":"2024-04-26T17:53:00.587330+0000","last_unstale":"2024-04-26T17:53:44.730528+0000","last_undegraded":"2024-04-26T17:53:44.730528+0000","last_fullsized":"2024-04-26T17:53:44.730528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:01:12.178610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.624899+0000","last_change":"2024-04-26T17:53:03.615723+0000","last_active":"2024-04-26T17:53:05.624899+0000","last_peered":"2024-04-26T17:53:05.624899+0000","last_clean":"2024-04-26T17:53:05.624899+0000","last_became_active":"2024-04-26T17:53:03.614931+0000","last_became_peered":"2024-04-26T17:53:03.614931+0000","last_unstale":"2024-04-26T17:53:05.624899+0000","last_undegraded":"2024-04-26T17:53:05.624899+0000","last_fullsized":"2024-04-26T17:53:05.624899+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:09.663876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:18.678893+0000","last_change":"2024-04-26T17:53:00.589232+0000","last_active":"2024-04-26T17:53:18.678893+0000","last_peered":"2024-04-26T17:53:18.678893+0000","last_clean":"2024-04-26T17:53:18.678893+0000","last_became_active":"2024-04-26T17:53:00.588863+0000","last_became_peered":"2024-04-26T17:53:00.588863+0000","last_unstale":"2024-04-26T17:53:18.678893+0000","last_undegraded":"2024-04-26T17:53:18.678893+0000","last_fullsized":"2024-04-26T17:53:18.678893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:14.360782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.759664+0000","last_change":"2024-04-26T17:53:03.624923+0000","last_active":"2024-04-26T17:53:48.759664+0000","last_peered":"2024-04-26T17:53:48.759664+0000","last_clean":"2024-04-26T17:53:48.759664+0000","last_became_active":"2024-04-26T17:53:03.624620+0000","last_became_peered":"2024-04-26T17:53:03.624620+0000","last_unstale":"2024-04-26T17:53:48.759664+0000","last_undegraded":"2024-04-26T17:53:48.759664+0000","last_fullsized":"2024-04-26T17:53:48.759664+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:50:51.259563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:18.678708+0000","last_change":"2024-04-26T17:53:01.146950+0000","last_active":"2024-04-26T17:53:18.678708+0000","last_peered":"2024-04-26T17:53:18.678708+0000","last_clean":"2024-04-26T17:53:18.678708+0000","last_became_active":"2024-04-26T17:53:01.146591+0000","last_became_peered":"2024-04-26T17:53:01.146591+0000","last_unstale":"2024-04-26T17:53:18.678708+0000","last_undegraded":"2024-04-26T17:53:18.678708+0000","last_fullsized":"2024-04-26T17:53:18.678708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:17.756660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.625037+0000","last_change":"2024-04-26T17:53:03.626944+0000","last_active":"2024-04-26T17:53:05.625037+0000","last_peered":"2024-04-26T17:53:05.625037+0000","last_clean":"2024-04-26T17:53:05.625037+0000","last_became_active":"2024-04-26T17:53:03.626594+0000","last_became_peered":"2024-04-26T17:53:03.626594+0000","last_unstale":"2024-04-26T17:53:05.625037+0000","last_undegraded":"2024-04-26T17:53:05.625037+0000","last_fullsized":"2024-04-26T17:53:05.625037+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:52:42.997251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'29","reported_seq":106,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:45.755214+0000","last_change":"2024-04-26T17:53:00.588353+0000","last_active":"2024-04-26T17:53:45.755214+0000","last_peered":"2024-04-26T17:53:45.755214+0000","last_clean":"2024-04-26T17:53:45.755214+0000","last_became_active":"2024-04-26T17:53:00.586351+0000","last_became_peered":"2024-04-26T17:53:00.586351+0000","last_unstale":"2024-04-26T17:53:45.755214+0000","last_undegraded":"2024-04-26T17:53:45.755214+0000","last_fullsized":"2024-04-26T17:53:45.755214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:41.660525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58078,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":55,"num_read_kb":78,"num_write":30,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623493+0000","last_change":"2024-04-26T17:53:03.612052+0000","last_active":"2024-04-26T17:53:05.623493+0000","last_peered":"2024-04-26T17:53:05.623493+0000","last_clean":"2024-04-26T17:53:05.623493+0000","last_became_active":"2024-04-26T17:53:03.611733+0000","last_became_peered":"2024-04-26T17:53:03.611733+0000","last_unstale":"2024-04-26T17:53:05.623493+0000","last_undegraded":"2024-04-26T17:53:05.623493+0000","last_fullsized":"2024-04-26T17:53:05.623493+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:05:23.958317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:19.688521+0000","last_change":"2024-04-26T17:53:00.588284+0000","last_active":"2024-04-26T17:53:19.688521+0000","last_peered":"2024-04-26T17:53:19.688521+0000","last_clean":"2024-04-26T17:53:19.688521+0000","last_became_active":"2024-04-26T17:53:00.587942+0000","last_became_peered":"2024-04-26T17:53:00.587942+0000","last_unstale":"2024-04-26T17:53:19.688521+0000","last_undegraded":"2024-04-26T17:53:19.688521+0000","last_fullsized":"2024-04-26T17:53:19.688521+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:43.782971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.622973+0000","last_change":"2024-04-26T17:53:03.613092+0000","last_active":"2024-04-26T17:53:05.622973+0000","last_peered":"2024-04-26T17:53:05.622973+0000","last_clean":"2024-04-26T17:53:05.622973+0000","last_became_active":"2024-04-26T17:53:03.612773+0000","last_became_peered":"2024-04-26T17:53:03.612773+0000","last_unstale":"2024-04-26T17:53:05.622973+0000","last_undegraded":"2024-04-26T17:53:05.622973+0000","last_fullsized":"2024-04-26T17:53:05.622973+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:41:21.340375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:18.679718+0000","last_change":"2024-04-26T17:53:00.588777+0000","last_active":"2024-04-26T17:53:18.679718+0000","last_peered":"2024-04-26T17:53:18.679718+0000","last_clean":"2024-04-26T17:53:18.679718+0000","last_became_active":"2024-04-26T17:53:00.588380+0000","last_became_peered":"2024-04-26T17:53:00.588380+0000","last_unstale":"2024-04-26T17:53:18.679718+0000","last_undegraded":"2024-04-26T17:53:18.679718+0000","last_fullsized":"2024-04-26T17:53:18.679718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:44.587381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.624491+0000","last_change":"2024-04-26T17:53:03.620344+0000","last_active":"2024-04-26T17:53:05.624491+0000","last_peered":"2024-04-26T17:53:05.624491+0000","last_clean":"2024-04-26T17:53:05.624491+0000","last_became_active":"2024-04-26T17:53:03.619944+0000","last_became_peered":"2024-04-26T17:53:03.619944+0000","last_unstale":"2024-04-26T17:53:05.624491+0000","last_undegraded":"2024-04-26T17:53:05.624491+0000","last_fullsized":"2024-04-26T17:53:05.624491+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:08:31.801342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:44.730351+0000","last_change":"2024-04-26T17:53:00.587733+0000","last_active":"2024-04-26T17:53:44.730351+0000","last_peered":"2024-04-26T17:53:44.730351+0000","last_clean":"2024-04-26T17:53:44.730351+0000","last_became_active":"2024-04-26T17:53:00.587373+0000","last_became_peered":"2024-04-26T17:53:00.587373+0000","last_unstale":"2024-04-26T17:53:44.730351+0000","last_undegraded":"2024-04-26T17:53:44.730351+0000","last_fullsized":"2024-04-26T17:53:44.730351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:10.978944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.622730+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:53:05.622730+0000","last_peered":"2024-04-26T17:53:05.622730+0000","last_clean":"2024-04-26T17:53:05.622730+0000","last_became_active":"2024-04-26T17:53:03.616588+0000","last_became_peered":"2024-04-26T17:53:03.616588+0000","last_unstale":"2024-04-26T17:53:05.622730+0000","last_undegraded":"2024-04-26T17:53:05.622730+0000","last_fullsized":"2024-04-26T17:53:05.622730+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:46:38.419613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:19.689077+0000","last_change":"2024-04-26T17:53:01.146922+0000","last_active":"2024-04-26T17:53:19.689077+0000","last_peered":"2024-04-26T17:53:19.689077+0000","last_clean":"2024-04-26T17:53:19.689077+0000","last_became_active":"2024-04-26T17:53:01.146520+0000","last_became_peered":"2024-04-26T17:53:01.146520+0000","last_unstale":"2024-04-26T17:53:19.689077+0000","last_undegraded":"2024-04-26T17:53:19.689077+0000","last_fullsized":"2024-04-26T17:53:19.689077+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:12:49.238012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.627582+0000","last_change":"2024-04-26T17:53:03.608513+0000","last_active":"2024-04-26T17:53:05.627582+0000","last_peered":"2024-04-26T17:53:05.627582+0000","last_clean":"2024-04-26T17:53:05.627582+0000","last_became_active":"2024-04-26T17:53:03.608167+0000","last_became_peered":"2024-04-26T17:53:03.608167+0000","last_unstale":"2024-04-26T17:53:05.627582+0000","last_undegraded":"2024-04-26T17:53:05.627582+0000","last_fullsized":"2024-04-26T17:53:05.627582+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:03.002756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:20.683384+0000","last_change":"2024-04-26T17:53:00.583878+0000","last_active":"2024-04-26T17:53:20.683384+0000","last_peered":"2024-04-26T17:53:20.683384+0000","last_clean":"2024-04-26T17:53:20.683384+0000","last_became_active":"2024-04-26T17:53:00.583159+0000","last_became_peered":"2024-04-26T17:53:00.583159+0000","last_unstale":"2024-04-26T17:53:20.683384+0000","last_undegraded":"2024-04-26T17:53:20.683384+0000","last_fullsized":"2024-04-26T17:53:20.683384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:49.850942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.616800+0000","last_change":"2024-04-26T17:53:03.614467+0000","last_active":"2024-04-26T17:53:05.616800+0000","last_peered":"2024-04-26T17:53:05.616800+0000","last_clean":"2024-04-26T17:53:05.616800+0000","last_became_active":"2024-04-26T17:53:03.613972+0000","last_became_peered":"2024-04-26T17:53:03.613972+0000","last_unstale":"2024-04-26T17:53:05.616800+0000","last_undegraded":"2024-04-26T17:53:05.616800+0000","last_fullsized":"2024-04-26T17:53:05.616800+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:27.259777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"peering","last_fresh":"2024-04-26T17:53:05.613913+0000","last_change":"2024-04-26T17:53:48.756965+0000","last_active":"2024-04-26T17:53:48.756965+0000","last_peered":"2024-04-26T17:53:05.613913+0000","last_clean":"2024-04-26T17:53:05.613913+0000","last_became_active":"2024-04-26T17:53:01.145146+0000","last_became_peered":"2024-04-26T17:53:01.145146+0000","last_unstale":"2024-04-26T17:53:05.613913+0000","last_undegraded":"2024-04-26T17:53:05.613913+0000","last_fullsized":"2024-04-26T17:53:05.613913+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,11],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'1","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.804789+0000","last_change":"2024-04-26T17:53:03.613888+0000","last_active":"2024-04-26T17:53:05.804789+0000","last_peered":"2024-04-26T17:53:05.804789+0000","last_clean":"2024-04-26T17:53:05.804789+0000","last_became_active":"2024-04-26T17:53:03.613499+0000","last_became_peered":"2024-04-26T17:53:03.613499+0000","last_unstale":"2024-04-26T17:53:05.804789+0000","last_undegraded":"2024-04-26T17:53:05.804789+0000","last_fullsized":"2024-04-26T17:53:05.804789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:15.549419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.624425+0000","last_change":"2024-04-26T17:53:00.587803+0000","last_active":"2024-04-26T17:53:05.624425+0000","last_peered":"2024-04-26T17:53:05.624425+0000","last_clean":"2024-04-26T17:53:05.624425+0000","last_became_active":"2024-04-26T17:53:00.587382+0000","last_became_peered":"2024-04-26T17:53:00.587382+0000","last_unstale":"2024-04-26T17:53:05.624425+0000","last_undegraded":"2024-04-26T17:53:05.624425+0000","last_fullsized":"2024-04-26T17:53:05.624425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:38.249478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.617122+0000","last_change":"2024-04-26T17:53:03.624749+0000","last_active":"2024-04-26T17:53:05.617122+0000","last_peered":"2024-04-26T17:53:05.617122+0000","last_clean":"2024-04-26T17:53:05.617122+0000","last_became_active":"2024-04-26T17:53:03.624445+0000","last_became_peered":"2024-04-26T17:53:03.624445+0000","last_unstale":"2024-04-26T17:53:05.617122+0000","last_undegraded":"2024-04-26T17:53:05.617122+0000","last_fullsized":"2024-04-26T17:53:05.617122+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:58.362581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.628670+0000","last_change":"2024-04-26T17:53:03.612899+0000","last_active":"2024-04-26T17:53:05.628670+0000","last_peered":"2024-04-26T17:53:05.628670+0000","last_clean":"2024-04-26T17:53:05.628670+0000","last_became_active":"2024-04-26T17:53:03.612564+0000","last_became_peered":"2024-04-26T17:53:03.612564+0000","last_unstale":"2024-04-26T17:53:05.628670+0000","last_undegraded":"2024-04-26T17:53:05.628670+0000","last_fullsized":"2024-04-26T17:53:05.628670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:07.368256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":52,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:46.732353+0000","last_change":"2024-04-26T17:53:01.145471+0000","last_active":"2024-04-26T17:53:46.732353+0000","last_peered":"2024-04-26T17:53:46.732353+0000","last_clean":"2024-04-26T17:53:46.732353+0000","last_became_active":"2024-04-26T17:53:01.145040+0000","last_became_peered":"2024-04-26T17:53:01.145040+0000","last_unstale":"2024-04-26T17:53:46.732353+0000","last_undegraded":"2024-04-26T17:53:46.732353+0000","last_fullsized":"2024-04-26T17:53:46.732353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:46:27.437329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.627187+0000","last_change":"2024-04-26T17:53:03.618600+0000","last_active":"2024-04-26T17:53:05.627187+0000","last_peered":"2024-04-26T17:53:05.627187+0000","last_clean":"2024-04-26T17:53:05.627187+0000","last_became_active":"2024-04-26T17:53:03.618139+0000","last_became_peered":"2024-04-26T17:53:03.618139+0000","last_unstale":"2024-04-26T17:53:05.627187+0000","last_undegraded":"2024-04-26T17:53:05.627187+0000","last_fullsized":"2024-04-26T17:53:05.627187+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:56.007659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:22.694234+0000","last_change":"2024-04-26T17:53:00.586386+0000","last_active":"2024-04-26T17:53:22.694234+0000","last_peered":"2024-04-26T17:53:22.694234+0000","last_clean":"2024-04-26T17:53:22.694234+0000","last_became_active":"2024-04-26T17:53:00.585953+0000","last_became_peered":"2024-04-26T17:53:00.585953+0000","last_unstale":"2024-04-26T17:53:22.694234+0000","last_undegraded":"2024-04-26T17:53:22.694234+0000","last_fullsized":"2024-04-26T17:53:22.694234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:39:57.160983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.622884+0000","last_change":"2024-04-26T17:51:38.839853+0000","last_active":"2024-04-26T17:53:05.622884+0000","last_peered":"2024-04-26T17:53:05.622884+0000","last_clean":"2024-04-26T17:53:05.622884+0000","last_became_active":"2024-04-26T17:51:38.839318+0000","last_became_peered":"2024-04-26T17:51:38.839318+0000","last_unstale":"2024-04-26T17:53:05.622884+0000","last_undegraded":"2024-04-26T17:53:05.622884+0000","last_fullsized":"2024-04-26T17:53:05.622884+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_clean_scrub_stamp":"2024-04-26T17:48:12.577543+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:08:50.683663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.622208+0000","last_change":"2024-04-26T17:53:03.614170+0000","last_active":"2024-04-26T17:53:05.622208+0000","last_peered":"2024-04-26T17:53:05.622208+0000","last_clean":"2024-04-26T17:53:05.622208+0000","last_became_active":"2024-04-26T17:53:03.613836+0000","last_became_peered":"2024-04-26T17:53:03.613836+0000","last_unstale":"2024-04-26T17:53:05.622208+0000","last_undegraded":"2024-04-26T17:53:05.622208+0000","last_fullsized":"2024-04-26T17:53:05.622208+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:48.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:19.686104+0000","last_change":"2024-04-26T17:53:00.587319+0000","last_active":"2024-04-26T17:53:19.686104+0000","last_peered":"2024-04-26T17:53:19.686104+0000","last_clean":"2024-04-26T17:53:19.686104+0000","last_became_active":"2024-04-26T17:53:00.586983+0000","last_became_peered":"2024-04-26T17:53:00.586983+0000","last_unstale":"2024-04-26T17:53:19.686104+0000","last_undegraded":"2024-04-26T17:53:19.686104+0000","last_fullsized":"2024-04-26T17:53:19.686104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:03:43.470013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:18.680332+0000","last_change":"2024-04-26T17:53:01.145672+0000","last_active":"2024-04-26T17:53:18.680332+0000","last_peered":"2024-04-26T17:53:18.680332+0000","last_clean":"2024-04-26T17:53:18.680332+0000","last_became_active":"2024-04-26T17:53:01.145274+0000","last_became_peered":"2024-04-26T17:53:01.145274+0000","last_unstale":"2024-04-26T17:53:18.680332+0000","last_undegraded":"2024-04-26T17:53:18.680332+0000","last_fullsized":"2024-04-26T17:53:18.680332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:21.122352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623379+0000","last_change":"2024-04-26T17:53:03.611232+0000","last_active":"2024-04-26T17:53:05.623379+0000","last_peered":"2024-04-26T17:53:05.623379+0000","last_clean":"2024-04-26T17:53:05.623379+0000","last_became_active":"2024-04-26T17:53:03.610905+0000","last_became_peered":"2024-04-26T17:53:03.610905+0000","last_unstale":"2024-04-26T17:53:05.623379+0000","last_undegraded":"2024-04-26T17:53:05.623379+0000","last_fullsized":"2024-04-26T17:53:05.623379+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:31.426237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'11","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.756696+0000","last_change":"2024-04-26T17:53:00.580220+0000","last_active":"2024-04-26T17:53:48.756696+0000","last_peered":"2024-04-26T17:53:48.756696+0000","last_clean":"2024-04-26T17:53:48.756696+0000","last_became_active":"2024-04-26T17:53:00.579818+0000","last_became_peered":"2024-04-26T17:53:00.579818+0000","last_unstale":"2024-04-26T17:53:48.756696+0000","last_undegraded":"2024-04-26T17:53:48.756696+0000","last_fullsized":"2024-04-26T17:53:48.756696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:15.260489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.627175+0000","last_change":"2024-04-26T17:53:03.610882+0000","last_active":"2024-04-26T17:53:05.627175+0000","last_peered":"2024-04-26T17:53:05.627175+0000","last_clean":"2024-04-26T17:53:05.627175+0000","last_became_active":"2024-04-26T17:53:03.609979+0000","last_became_peered":"2024-04-26T17:53:03.609979+0000","last_unstale":"2024-04-26T17:53:05.627175+0000","last_undegraded":"2024-04-26T17:53:05.627175+0000","last_fullsized":"2024-04-26T17:53:05.627175+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:33.167579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:18.678024+0000","last_change":"2024-04-26T17:53:01.358555+0000","last_active":"2024-04-26T17:53:18.678024+0000","last_peered":"2024-04-26T17:53:18.678024+0000","last_clean":"2024-04-26T17:53:18.678024+0000","last_became_active":"2024-04-26T17:53:01.358138+0000","last_became_peered":"2024-04-26T17:53:01.358138+0000","last_unstale":"2024-04-26T17:53:18.678024+0000","last_undegraded":"2024-04-26T17:53:18.678024+0000","last_fullsized":"2024-04-26T17:53:18.678024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:14.886609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.759505+0000","last_change":"2024-04-26T17:53:03.618816+0000","last_active":"2024-04-26T17:53:48.759505+0000","last_peered":"2024-04-26T17:53:48.759505+0000","last_clean":"2024-04-26T17:53:48.759505+0000","last_became_active":"2024-04-26T17:53:03.618184+0000","last_became_peered":"2024-04-26T17:53:03.618184+0000","last_unstale":"2024-04-26T17:53:48.759505+0000","last_undegraded":"2024-04-26T17:53:48.759505+0000","last_fullsized":"2024-04-26T17:53:48.759505+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:07.514516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:18.679325+0000","last_change":"2024-04-26T17:53:01.147081+0000","last_active":"2024-04-26T17:53:18.679325+0000","last_peered":"2024-04-26T17:53:18.679325+0000","last_clean":"2024-04-26T17:53:18.679325+0000","last_became_active":"2024-04-26T17:53:01.146698+0000","last_became_peered":"2024-04-26T17:53:01.146698+0000","last_unstale":"2024-04-26T17:53:18.679325+0000","last_undegraded":"2024-04-26T17:53:18.679325+0000","last_fullsized":"2024-04-26T17:53:18.679325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:44:19.927316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623073+0000","last_change":"2024-04-26T17:53:03.612592+0000","last_active":"2024-04-26T17:53:05.623073+0000","last_peered":"2024-04-26T17:53:05.623073+0000","last_clean":"2024-04-26T17:53:05.623073+0000","last_became_active":"2024-04-26T17:53:03.612274+0000","last_became_peered":"2024-04-26T17:53:03.612274+0000","last_unstale":"2024-04-26T17:53:05.623073+0000","last_undegraded":"2024-04-26T17:53:05.623073+0000","last_fullsized":"2024-04-26T17:53:05.623073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:30:48.461947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:18.678673+0000","last_change":"2024-04-26T17:53:00.586430+0000","last_active":"2024-04-26T17:53:18.678673+0000","last_peered":"2024-04-26T17:53:18.678673+0000","last_clean":"2024-04-26T17:53:18.678673+0000","last_became_active":"2024-04-26T17:53:00.586004+0000","last_became_peered":"2024-04-26T17:53:00.586004+0000","last_unstale":"2024-04-26T17:53:18.678673+0000","last_undegraded":"2024-04-26T17:53:18.678673+0000","last_fullsized":"2024-04-26T17:53:18.678673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:47:18.224890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.616522+0000","last_change":"2024-04-26T17:53:03.623081+0000","last_active":"2024-04-26T17:53:05.616522+0000","last_peered":"2024-04-26T17:53:05.616522+0000","last_clean":"2024-04-26T17:53:05.616522+0000","last_became_active":"2024-04-26T17:53:03.622761+0000","last_became_peered":"2024-04-26T17:53:03.622761+0000","last_unstale":"2024-04-26T17:53:05.616522+0000","last_undegraded":"2024-04-26T17:53:05.616522+0000","last_fullsized":"2024-04-26T17:53:05.616522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:39.520075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623075+0000","last_change":"2024-04-26T17:53:00.583013+0000","last_active":"2024-04-26T17:53:05.623075+0000","last_peered":"2024-04-26T17:53:05.623075+0000","last_clean":"2024-04-26T17:53:05.623075+0000","last_became_active":"2024-04-26T17:53:00.582653+0000","last_became_peered":"2024-04-26T17:53:00.582653+0000","last_unstale":"2024-04-26T17:53:05.623075+0000","last_undegraded":"2024-04-26T17:53:05.623075+0000","last_fullsized":"2024-04-26T17:53:05.623075+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:09.448621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.626999+0000","last_change":"2024-04-26T17:53:03.618232+0000","last_active":"2024-04-26T17:53:05.626999+0000","last_peered":"2024-04-26T17:53:05.626999+0000","last_clean":"2024-04-26T17:53:05.626999+0000","last_became_active":"2024-04-26T17:53:03.617906+0000","last_became_peered":"2024-04-26T17:53:03.617906+0000","last_unstale":"2024-04-26T17:53:05.626999+0000","last_undegraded":"2024-04-26T17:53:05.626999+0000","last_fullsized":"2024-04-26T17:53:05.626999+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:17.357398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:18.681229+0000","last_change":"2024-04-26T17:53:00.586565+0000","last_active":"2024-04-26T17:53:18.681229+0000","last_peered":"2024-04-26T17:53:18.681229+0000","last_clean":"2024-04-26T17:53:18.681229+0000","last_became_active":"2024-04-26T17:53:00.586161+0000","last_became_peered":"2024-04-26T17:53:00.586161+0000","last_unstale":"2024-04-26T17:53:18.681229+0000","last_undegraded":"2024-04-26T17:53:18.681229+0000","last_fullsized":"2024-04-26T17:53:18.681229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:56:25.294055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623319+0000","last_change":"2024-04-26T17:53:03.620340+0000","last_active":"2024-04-26T17:53:05.623319+0000","last_peered":"2024-04-26T17:53:05.623319+0000","last_clean":"2024-04-26T17:53:05.623319+0000","last_became_active":"2024-04-26T17:53:03.619897+0000","last_became_peered":"2024-04-26T17:53:03.619897+0000","last_unstale":"2024-04-26T17:53:05.623319+0000","last_undegraded":"2024-04-26T17:53:05.623319+0000","last_fullsized":"2024-04-26T17:53:05.623319+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:12:40.662070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:19.689714+0000","last_change":"2024-04-26T17:53:01.358131+0000","last_active":"2024-04-26T17:53:19.689714+0000","last_peered":"2024-04-26T17:53:19.689714+0000","last_clean":"2024-04-26T17:53:19.689714+0000","last_became_active":"2024-04-26T17:53:01.357796+0000","last_became_peered":"2024-04-26T17:53:01.357796+0000","last_unstale":"2024-04-26T17:53:19.689714+0000","last_undegraded":"2024-04-26T17:53:19.689714+0000","last_fullsized":"2024-04-26T17:53:19.689714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:47.849873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.624290+0000","last_change":"2024-04-26T17:53:03.616133+0000","last_active":"2024-04-26T17:53:05.624290+0000","last_peered":"2024-04-26T17:53:05.624290+0000","last_clean":"2024-04-26T17:53:05.624290+0000","last_became_active":"2024-04-26T17:53:03.615797+0000","last_became_peered":"2024-04-26T17:53:03.615797+0000","last_unstale":"2024-04-26T17:53:05.624290+0000","last_undegraded":"2024-04-26T17:53:05.624290+0000","last_fullsized":"2024-04-26T17:53:05.624290+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:05.698489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.633654+0000","last_change":"2024-04-26T17:53:00.587696+0000","last_active":"2024-04-26T17:53:05.633654+0000","last_peered":"2024-04-26T17:53:05.633654+0000","last_clean":"2024-04-26T17:53:05.633654+0000","last_became_active":"2024-04-26T17:53:00.587305+0000","last_became_peered":"2024-04-26T17:53:00.587305+0000","last_unstale":"2024-04-26T17:53:05.633654+0000","last_undegraded":"2024-04-26T17:53:05.633654+0000","last_fullsized":"2024-04-26T17:53:05.633654+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:39.185753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'1","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:06.937817+0000","last_change":"2024-04-26T17:53:03.613076+0000","last_active":"2024-04-26T17:53:06.937817+0000","last_peered":"2024-04-26T17:53:06.937817+0000","last_clean":"2024-04-26T17:53:06.937817+0000","last_became_active":"2024-04-26T17:53:03.612746+0000","last_became_peered":"2024-04-26T17:53:03.612746+0000","last_unstale":"2024-04-26T17:53:06.937817+0000","last_undegraded":"2024-04-26T17:53:06.937817+0000","last_fullsized":"2024-04-26T17:53:06.937817+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:20.920154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.629072+0000","last_change":"2024-04-26T17:53:00.582575+0000","last_active":"2024-04-26T17:53:05.629072+0000","last_peered":"2024-04-26T17:53:05.629072+0000","last_clean":"2024-04-26T17:53:05.629072+0000","last_became_active":"2024-04-26T17:53:00.582215+0000","last_became_peered":"2024-04-26T17:53:00.582215+0000","last_unstale":"2024-04-26T17:53:05.629072+0000","last_undegraded":"2024-04-26T17:53:05.629072+0000","last_fullsized":"2024-04-26T17:53:05.629072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:35:12.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623833+0000","last_change":"2024-04-26T17:53:03.616269+0000","last_active":"2024-04-26T17:53:05.623833+0000","last_peered":"2024-04-26T17:53:05.623833+0000","last_clean":"2024-04-26T17:53:05.623833+0000","last_became_active":"2024-04-26T17:53:03.615896+0000","last_became_peered":"2024-04-26T17:53:03.615896+0000","last_unstale":"2024-04-26T17:53:05.623833+0000","last_undegraded":"2024-04-26T17:53:05.623833+0000","last_fullsized":"2024-04-26T17:53:05.623833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:12.111392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.616683+0000","last_change":"2024-04-26T17:53:03.621589+0000","last_active":"2024-04-26T17:53:05.616683+0000","last_peered":"2024-04-26T17:53:05.616683+0000","last_clean":"2024-04-26T17:53:05.616683+0000","last_became_active":"2024-04-26T17:53:03.621211+0000","last_became_peered":"2024-04-26T17:53:03.621211+0000","last_unstale":"2024-04-26T17:53:05.616683+0000","last_undegraded":"2024-04-26T17:53:05.616683+0000","last_fullsized":"2024-04-26T17:53:05.616683+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.487476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.622988+0000","last_change":"2024-04-26T17:53:03.624536+0000","last_active":"2024-04-26T17:53:05.622988+0000","last_peered":"2024-04-26T17:53:05.622988+0000","last_clean":"2024-04-26T17:53:05.622988+0000","last_became_active":"2024-04-26T17:53:03.624234+0000","last_became_peered":"2024-04-26T17:53:03.624234+0000","last_unstale":"2024-04-26T17:53:05.622988+0000","last_undegraded":"2024-04-26T17:53:05.622988+0000","last_fullsized":"2024-04-26T17:53:05.622988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:31:36.322688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.622903+0000","last_change":"2024-04-26T17:53:03.620096+0000","last_active":"2024-04-26T17:53:05.622903+0000","last_peered":"2024-04-26T17:53:05.622903+0000","last_clean":"2024-04-26T17:53:05.622903+0000","last_became_active":"2024-04-26T17:53:03.619729+0000","last_became_peered":"2024-04-26T17:53:03.619729+0000","last_unstale":"2024-04-26T17:53:05.622903+0000","last_undegraded":"2024-04-26T17:53:05.622903+0000","last_fullsized":"2024-04-26T17:53:05.622903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:17.987728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"peering","last_fresh":"2024-04-26T17:53:05.626877+0000","last_change":"2024-04-26T17:53:48.760704+0000","last_active":"2024-04-26T17:53:48.760704+0000","last_peered":"2024-04-26T17:53:05.626877+0000","last_clean":"2024-04-26T17:53:05.626877+0000","last_became_active":"2024-04-26T17:53:03.618062+0000","last_became_peered":"2024-04-26T17:53:03.618062+0000","last_unstale":"2024-04-26T17:53:05.626877+0000","last_undegraded":"2024-04-26T17:53:05.626877+0000","last_fullsized":"2024-04-26T17:53:05.626877+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.622725+0000","last_change":"2024-04-26T17:53:03.619193+0000","last_active":"2024-04-26T17:53:05.622725+0000","last_peered":"2024-04-26T17:53:05.622725+0000","last_clean":"2024-04-26T17:53:05.622725+0000","last_became_active":"2024-04-26T17:53:03.618776+0000","last_became_peered":"2024-04-26T17:53:03.618776+0000","last_unstale":"2024-04-26T17:53:05.622725+0000","last_undegraded":"2024-04-26T17:53:05.622725+0000","last_fullsized":"2024-04-26T17:53:05.622725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:07:24.811683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.616868+0000","last_change":"2024-04-26T17:53:03.621366+0000","last_active":"2024-04-26T17:53:05.616868+0000","last_peered":"2024-04-26T17:53:05.616868+0000","last_clean":"2024-04-26T17:53:05.616868+0000","last_became_active":"2024-04-26T17:53:03.621048+0000","last_became_peered":"2024-04-26T17:53:03.621048+0000","last_unstale":"2024-04-26T17:53:05.616868+0000","last_undegraded":"2024-04-26T17:53:05.616868+0000","last_fullsized":"2024-04-26T17:53:05.616868+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:31:06.484880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623708+0000","last_change":"2024-04-26T17:53:03.619383+0000","last_active":"2024-04-26T17:53:05.623708+0000","last_peered":"2024-04-26T17:53:05.623708+0000","last_clean":"2024-04-26T17:53:05.623708+0000","last_became_active":"2024-04-26T17:53:03.619017+0000","last_became_peered":"2024-04-26T17:53:03.619017+0000","last_unstale":"2024-04-26T17:53:05.623708+0000","last_undegraded":"2024-04-26T17:53:05.623708+0000","last_fullsized":"2024-04-26T17:53:05.623708+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:20.091959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.627451+0000","last_change":"2024-04-26T17:53:03.613458+0000","last_active":"2024-04-26T17:53:05.627451+0000","last_peered":"2024-04-26T17:53:05.627451+0000","last_clean":"2024-04-26T17:53:05.627451+0000","last_became_active":"2024-04-26T17:53:03.613123+0000","last_became_peered":"2024-04-26T17:53:03.613123+0000","last_unstale":"2024-04-26T17:53:05.627451+0000","last_undegraded":"2024-04-26T17:53:05.627451+0000","last_fullsized":"2024-04-26T17:53:05.627451+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:17:37.926486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.628299+0000","last_change":"2024-04-26T17:53:03.605521+0000","last_active":"2024-04-26T17:53:05.628299+0000","last_peered":"2024-04-26T17:53:05.628299+0000","last_clean":"2024-04-26T17:53:05.628299+0000","last_became_active":"2024-04-26T17:53:03.605145+0000","last_became_peered":"2024-04-26T17:53:03.605145+0000","last_unstale":"2024-04-26T17:53:05.628299+0000","last_undegraded":"2024-04-26T17:53:05.628299+0000","last_fullsized":"2024-04-26T17:53:05.628299+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:58:09.856021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.626284+0000","last_change":"2024-04-26T17:53:03.610866+0000","last_active":"2024-04-26T17:53:05.626284+0000","last_peered":"2024-04-26T17:53:05.626284+0000","last_clean":"2024-04-26T17:53:05.626284+0000","last_became_active":"2024-04-26T17:53:03.610326+0000","last_became_peered":"2024-04-26T17:53:03.610326+0000","last_unstale":"2024-04-26T17:53:05.626284+0000","last_undegraded":"2024-04-26T17:53:05.626284+0000","last_fullsized":"2024-04-26T17:53:05.626284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:20.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.624239+0000","last_change":"2024-04-26T17:53:03.615336+0000","last_active":"2024-04-26T17:53:05.624239+0000","last_peered":"2024-04-26T17:53:05.624239+0000","last_clean":"2024-04-26T17:53:05.624239+0000","last_became_active":"2024-04-26T17:53:03.614447+0000","last_became_peered":"2024-04-26T17:53:03.614447+0000","last_unstale":"2024-04-26T17:53:05.624239+0000","last_undegraded":"2024-04-26T17:53:05.624239+0000","last_fullsized":"2024-04-26T17:53:05.624239+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:36:10.091929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623142+0000","last_change":"2024-04-26T17:53:03.614863+0000","last_active":"2024-04-26T17:53:05.623142+0000","last_peered":"2024-04-26T17:53:05.623142+0000","last_clean":"2024-04-26T17:53:05.623142+0000","last_became_active":"2024-04-26T17:53:03.614520+0000","last_became_peered":"2024-04-26T17:53:03.614520+0000","last_unstale":"2024-04-26T17:53:05.623142+0000","last_undegraded":"2024-04-26T17:53:05.623142+0000","last_fullsized":"2024-04-26T17:53:05.623142+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:51.367785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.616615+0000","last_change":"2024-04-26T17:53:03.618521+0000","last_active":"2024-04-26T17:53:05.616615+0000","last_peered":"2024-04-26T17:53:05.616615+0000","last_clean":"2024-04-26T17:53:05.616615+0000","last_became_active":"2024-04-26T17:53:03.617658+0000","last_became_peered":"2024-04-26T17:53:03.617658+0000","last_unstale":"2024-04-26T17:53:05.616615+0000","last_undegraded":"2024-04-26T17:53:05.616615+0000","last_fullsized":"2024-04-26T17:53:05.616615+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:39.813098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280045,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62079,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":171,"num_read_kb":147,"num_write":164,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":186237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383704,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":462787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645341,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":1106951,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":463023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070502,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":462943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332140,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1054873,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495666,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":462412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":520283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":472268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":516629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":462723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920827,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":462849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":1054557,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":58078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":59819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":59656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T17:53:51.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T17:53:51.041 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:50 smithi031 ceph-mon[29164]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T17:53:51.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:50 smithi097 ceph-mon[32419]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T17:53:51.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:50 smithi186 ceph-mon[32797]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T17:53:51.268 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:52.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:51 smithi031 ceph-mon[29164]: pgmap v321: 97 pgs: 3 peering, 94 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-04-26T17:53:52.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:51 smithi031 ceph-mon[29164]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:53:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:51 smithi097 ceph-mon[32419]: pgmap v321: 97 pgs: 3 peering, 94 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-04-26T17:53:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:51 smithi097 ceph-mon[32419]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:53:52.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:51 smithi186 ceph-mon[32797]: pgmap v321: 97 pgs: 3 peering, 94 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-04-26T17:53:52.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:51 smithi186 ceph-mon[32797]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:53:52.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.948+0000 7f5a34efd700 1 -- 172.21.15.31:0/1847289941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a30103470 msgr2=0x7f5a301038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:52.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.948+0000 7f5a34efd700 1 --2- 172.21.15.31:0/1847289941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a30103470 0x7f5a301038f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009990 tx=0x7f5a18009ca0 comp rx=0 tx=0).stop 2024-04-26T17:53:52.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.948+0000 7f5a34efd700 1 -- 172.21.15.31:0/1847289941 shutdown_connections 2024-04-26T17:53:52.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.948+0000 7f5a34efd700 1 --2- 172.21.15.31:0/1847289941 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a301094c0 0x7f5a301098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:52.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.948+0000 7f5a34efd700 1 --2- 172.21.15.31:0/1847289941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a30103e30 0x7f5a30110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:52.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.948+0000 7f5a34efd700 1 --2- 172.21.15.31:0/1847289941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a30103470 0x7f5a301038f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:52.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.948+0000 7f5a34efd700 1 -- 172.21.15.31:0/1847289941 >> 172.21.15.31:0/1847289941 conn(0x7f5a3007a580 msgr2=0x7f5a3007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:52.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.948+0000 7f5a34efd700 1 -- 172.21.15.31:0/1847289941 shutdown_connections 2024-04-26T17:53:52.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.948+0000 7f5a34efd700 1 -- 172.21.15.31:0/1847289941 wait complete. 2024-04-26T17:53:52.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.949+0000 7f5a34efd700 1 Processor -- start 2024-04-26T17:53:52.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.949+0000 7f5a34efd700 1 -- start start 2024-04-26T17:53:52.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.949+0000 7f5a34efd700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a30103470 0x7f5a3010f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:52.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.949+0000 7f5a34efd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a30103e30 0x7f5a3010fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:52.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.949+0000 7f5a34efd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a301094c0 0x7f5a301a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:52.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.949+0000 7f5a34efd700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5a301136c0 con 0x7f5a30103e30 2024-04-26T17:53:52.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.949+0000 7f5a34efd700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f5a30113540 con 0x7f5a30103470 2024-04-26T17:53:52.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.949+0000 7f5a34efd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5a30113840 con 0x7f5a301094c0 2024-04-26T17:53:52.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.949+0000 7f5a27fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a30103e30 0x7f5a3010fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:52.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.949+0000 7f5a27fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a30103e30 0x7f5a3010fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57512/0 (socket says 172.21.15.31:57512) 2024-04-26T17:53:52.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.949+0000 7f5a27fff700 1 -- 172.21.15.31:0/230033618 learned_addr learned my addr 172.21.15.31:0/230033618 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:52.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.949+0000 7f5a2e59c700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a30103470 0x7f5a3010f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:52.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.949+0000 7f5a2ed9d700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a301094c0 0x7f5a301a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:52.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.950+0000 7f5a27fff700 1 -- 172.21.15.31:0/230033618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a301094c0 msgr2=0x7f5a301a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:52.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.950+0000 7f5a27fff700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a301094c0 0x7f5a301a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:52.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.950+0000 7f5a27fff700 1 -- 172.21.15.31:0/230033618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a30103470 msgr2=0x7f5a3010f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:52.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.950+0000 7f5a27fff700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a30103470 0x7f5a3010f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:52.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.950+0000 7f5a27fff700 1 -- 172.21.15.31:0/230033618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a301a52d0 con 0x7f5a30103e30 2024-04-26T17:53:52.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.950+0000 7f5a2ed9d700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a301094c0 0x7f5a301a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:53:52.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.950+0000 7f5a27fff700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a30103e30 0x7f5a3010fc00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5a18000c00 tx=0x7f5a180047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:52.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.950+0000 7f5a277fe700 1 -- 172.21.15.31:0/230033618 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a18027070 con 0x7f5a30103e30 2024-04-26T17:53:52.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.950+0000 7f5a34efd700 1 -- 172.21.15.31:0/230033618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a301a5560 con 0x7f5a30103e30 2024-04-26T17:53:52.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.950+0000 7f5a277fe700 1 -- 172.21.15.31:0/230033618 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a180040e0 con 0x7f5a30103e30 2024-04-26T17:53:52.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.950+0000 7f5a277fe700 1 -- 172.21.15.31:0/230033618 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a1800f460 con 0x7f5a30103e30 2024-04-26T17:53:52.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.950+0000 7f5a34efd700 1 -- 172.21.15.31:0/230033618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a301ace40 con 0x7f5a30103e30 2024-04-26T17:53:52.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.951+0000 7f5a34efd700 1 -- 172.21.15.31:0/230033618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a3006a110 con 0x7f5a30103e30 2024-04-26T17:53:52.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.952+0000 7f5a277fe700 1 -- 172.21.15.31:0/230033618 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5a1802a070 con 0x7f5a30103e30 2024-04-26T17:53:52.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.953+0000 7f5a277fe700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f5a1c0a3cd0 0x7f5a1c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:52.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.953+0000 7f5a2e59c700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f5a1c0a3cd0 0x7f5a1c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:52.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.953+0000 7f5a277fe700 1 -- 172.21.15.31:0/230033618 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f5a180d1970 con 0x7f5a30103e30 2024-04-26T17:53:52.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.954+0000 7f5a2e59c700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f5a1c0a3cd0 0x7f5a1c0a6190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5a20009a10 tx=0x7f5a20009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:52.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:52.956+0000 7f5a277fe700 1 -- 172.21.15.31:0/230033618 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5a18031070 con 0x7f5a30103e30 2024-04-26T17:53:53.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:53.160+0000 7f5a34efd700 1 -- 172.21.15.31:0/230033618 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5a30063860 con 0x7f5a1c0a3cd0 2024-04-26T17:53:53.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:53.167+0000 7f5a277fe700 1 -- 172.21.15.31:0/230033618 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258723 (secure 0 0 0) 0x7f5a30063860 con 0x7f5a1c0a3cd0 2024-04-26T17:53:53.171 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:53:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:53.170+0000 7f5a34efd700 1 -- 172.21.15.31:0/230033618 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f5a1c0a3cd0 msgr2=0x7f5a1c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:53.170+0000 7f5a34efd700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f5a1c0a3cd0 0x7f5a1c0a6190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5a20009a10 tx=0x7f5a20009190 comp rx=0 tx=0).stop 2024-04-26T17:53:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:53.170+0000 7f5a34efd700 1 -- 172.21.15.31:0/230033618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a30103e30 msgr2=0x7f5a3010fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:53.170+0000 7f5a34efd700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a30103e30 0x7f5a3010fc00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5a18000c00 tx=0x7f5a180047b0 comp rx=0 tx=0).stop 2024-04-26T17:53:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:53.170+0000 7f5a34efd700 1 -- 172.21.15.31:0/230033618 shutdown_connections 2024-04-26T17:53:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:53.170+0000 7f5a34efd700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f5a1c0a3cd0 0x7f5a1c0a6190 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:53.170+0000 7f5a34efd700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5a30103470 0x7f5a3010f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:53.170+0000 7f5a34efd700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a301094c0 0x7f5a301a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:53.170+0000 7f5a34efd700 1 --2- 172.21.15.31:0/230033618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a30103e30 0x7f5a3010fc00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:53.170+0000 7f5a34efd700 1 -- 172.21.15.31:0/230033618 >> 172.21.15.31:0/230033618 conn(0x7f5a3007a580 msgr2=0x7f5a30102140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:53.170+0000 7f5a34efd700 1 -- 172.21.15.31:0/230033618 shutdown_connections 2024-04-26T17:53:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:53.170+0000 7f5a34efd700 1 -- 172.21.15.31:0/230033618 wait complete. 2024-04-26T17:53:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-26T17:53:53.907 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-04-26T17:53:52.317938+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656938,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":333,"num_read_kb":282,"num_write":348,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":288,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361088,"kb_used_data":11896,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580736,"statfs":{"total":1151940427776,"available":1151570673664,"internally_reserved":0,"allocated":12181504,"data_stored":7636605,"data_compressed":106863,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9652,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":57,"num_read_kb":42,"num_write":12,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001973"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765218+0000","last_change":"2024-04-26T17:53:03.616592+0000","last_active":"2024-04-26T17:53:49.765218+0000","last_peered":"2024-04-26T17:53:49.765218+0000","last_clean":"2024-04-26T17:53:49.765218+0000","last_became_active":"2024-04-26T17:53:03.616255+0000","last_became_peered":"2024-04-26T17:53:03.616255+0000","last_unstale":"2024-04-26T17:53:49.765218+0000","last_undegraded":"2024-04-26T17:53:49.765218+0000","last_fullsized":"2024-04-26T17:53:49.765218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:43:31.183679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.759387+0000","last_change":"2024-04-26T17:53:03.615402+0000","last_active":"2024-04-26T17:53:48.759387+0000","last_peered":"2024-04-26T17:53:48.759387+0000","last_clean":"2024-04-26T17:53:48.759387+0000","last_became_active":"2024-04-26T17:53:03.615048+0000","last_became_peered":"2024-04-26T17:53:03.615048+0000","last_unstale":"2024-04-26T17:53:48.759387+0000","last_undegraded":"2024-04-26T17:53:48.759387+0000","last_fullsized":"2024-04-26T17:53:48.759387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:05:12.454770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.759086+0000","last_change":"2024-04-26T17:53:03.622625+0000","last_active":"2024-04-26T17:53:48.759086+0000","last_peered":"2024-04-26T17:53:48.759086+0000","last_clean":"2024-04-26T17:53:48.759086+0000","last_became_active":"2024-04-26T17:53:03.622305+0000","last_became_peered":"2024-04-26T17:53:03.622305+0000","last_unstale":"2024-04-26T17:53:48.759086+0000","last_undegraded":"2024-04-26T17:53:48.759086+0000","last_fullsized":"2024-04-26T17:53:48.759086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:06.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891229+0000","last_change":"2024-04-26T17:53:03.617650+0000","last_active":"2024-04-26T17:53:49.891229+0000","last_peered":"2024-04-26T17:53:49.891229+0000","last_clean":"2024-04-26T17:53:49.891229+0000","last_became_active":"2024-04-26T17:53:03.617220+0000","last_became_peered":"2024-04-26T17:53:03.617220+0000","last_unstale":"2024-04-26T17:53:49.891229+0000","last_undegraded":"2024-04-26T17:53:49.891229+0000","last_fullsized":"2024-04-26T17:53:49.891229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:31.765577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767329+0000","last_change":"2024-04-26T17:53:03.615429+0000","last_active":"2024-04-26T17:53:49.767329+0000","last_peered":"2024-04-26T17:53:49.767329+0000","last_clean":"2024-04-26T17:53:49.767329+0000","last_became_active":"2024-04-26T17:53:03.615079+0000","last_became_peered":"2024-04-26T17:53:03.615079+0000","last_unstale":"2024-04-26T17:53:49.767329+0000","last_undegraded":"2024-04-26T17:53:49.767329+0000","last_fullsized":"2024-04-26T17:53:49.767329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:20.994016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764669+0000","last_change":"2024-04-26T17:53:03.624619+0000","last_active":"2024-04-26T17:53:49.764669+0000","last_peered":"2024-04-26T17:53:49.764669+0000","last_clean":"2024-04-26T17:53:49.764669+0000","last_became_active":"2024-04-26T17:53:03.624251+0000","last_became_peered":"2024-04-26T17:53:03.624251+0000","last_unstale":"2024-04-26T17:53:49.764669+0000","last_undegraded":"2024-04-26T17:53:49.764669+0000","last_fullsized":"2024-04-26T17:53:49.764669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:50:17.535235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767186+0000","last_change":"2024-04-26T17:53:03.619325+0000","last_active":"2024-04-26T17:53:49.767186+0000","last_peered":"2024-04-26T17:53:49.767186+0000","last_clean":"2024-04-26T17:53:49.767186+0000","last_became_active":"2024-04-26T17:53:03.618989+0000","last_became_peered":"2024-04-26T17:53:03.618989+0000","last_unstale":"2024-04-26T17:53:49.767186+0000","last_undegraded":"2024-04-26T17:53:49.767186+0000","last_fullsized":"2024-04-26T17:53:49.767186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:10.195742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764403+0000","last_change":"2024-04-26T17:53:03.609346+0000","last_active":"2024-04-26T17:53:49.764403+0000","last_peered":"2024-04-26T17:53:49.764403+0000","last_clean":"2024-04-26T17:53:49.764403+0000","last_became_active":"2024-04-26T17:53:03.609021+0000","last_became_peered":"2024-04-26T17:53:03.609021+0000","last_unstale":"2024-04-26T17:53:49.764403+0000","last_undegraded":"2024-04-26T17:53:49.764403+0000","last_fullsized":"2024-04-26T17:53:49.764403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:44.328599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.624266+0000","last_change":"2024-04-26T17:53:03.618573+0000","last_active":"2024-04-26T17:53:05.624266+0000","last_peered":"2024-04-26T17:53:05.624266+0000","last_clean":"2024-04-26T17:53:05.624266+0000","last_became_active":"2024-04-26T17:53:03.618223+0000","last_became_peered":"2024-04-26T17:53:03.618223+0000","last_unstale":"2024-04-26T17:53:05.624266+0000","last_undegraded":"2024-04-26T17:53:05.624266+0000","last_fullsized":"2024-04-26T17:53:05.624266+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:16.749205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765013+0000","last_change":"2024-04-26T17:53:03.621530+0000","last_active":"2024-04-26T17:53:49.765013+0000","last_peered":"2024-04-26T17:53:49.765013+0000","last_clean":"2024-04-26T17:53:49.765013+0000","last_became_active":"2024-04-26T17:53:03.621174+0000","last_became_peered":"2024-04-26T17:53:03.621174+0000","last_unstale":"2024-04-26T17:53:49.765013+0000","last_undegraded":"2024-04-26T17:53:49.765013+0000","last_fullsized":"2024-04-26T17:53:49.765013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:31.388232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767569+0000","last_change":"2024-04-26T17:53:03.617682+0000","last_active":"2024-04-26T17:53:49.767569+0000","last_peered":"2024-04-26T17:53:49.767569+0000","last_clean":"2024-04-26T17:53:49.767569+0000","last_became_active":"2024-04-26T17:53:03.617348+0000","last_became_peered":"2024-04-26T17:53:03.617348+0000","last_unstale":"2024-04-26T17:53:49.767569+0000","last_undegraded":"2024-04-26T17:53:49.767569+0000","last_fullsized":"2024-04-26T17:53:49.767569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:21.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891137+0000","last_change":"2024-04-26T17:53:03.617955+0000","last_active":"2024-04-26T17:53:49.891137+0000","last_peered":"2024-04-26T17:53:49.891137+0000","last_clean":"2024-04-26T17:53:49.891137+0000","last_became_active":"2024-04-26T17:53:03.616765+0000","last_became_peered":"2024-04-26T17:53:03.616765+0000","last_unstale":"2024-04-26T17:53:49.891137+0000","last_undegraded":"2024-04-26T17:53:49.891137+0000","last_fullsized":"2024-04-26T17:53:49.891137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:59.292277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.622873+0000","last_change":"2024-04-26T17:53:03.613070+0000","last_active":"2024-04-26T17:53:05.622873+0000","last_peered":"2024-04-26T17:53:05.622873+0000","last_clean":"2024-04-26T17:53:05.622873+0000","last_became_active":"2024-04-26T17:53:03.612263+0000","last_became_peered":"2024-04-26T17:53:03.612263+0000","last_unstale":"2024-04-26T17:53:05.622873+0000","last_undegraded":"2024-04-26T17:53:05.622873+0000","last_fullsized":"2024-04-26T17:53:05.622873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:55.329158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.759305+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:53:48.759305+0000","last_peered":"2024-04-26T17:53:48.759305+0000","last_clean":"2024-04-26T17:53:48.759305+0000","last_became_active":"2024-04-26T17:53:03.616623+0000","last_became_peered":"2024-04-26T17:53:03.616623+0000","last_unstale":"2024-04-26T17:53:48.759305+0000","last_undegraded":"2024-04-26T17:53:48.759305+0000","last_fullsized":"2024-04-26T17:53:48.759305+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:04.231549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.758706+0000","last_change":"2024-04-26T17:53:03.622147+0000","last_active":"2024-04-26T17:53:48.758706+0000","last_peered":"2024-04-26T17:53:48.758706+0000","last_clean":"2024-04-26T17:53:48.758706+0000","last_became_active":"2024-04-26T17:53:03.621542+0000","last_became_peered":"2024-04-26T17:53:03.621542+0000","last_unstale":"2024-04-26T17:53:48.758706+0000","last_undegraded":"2024-04-26T17:53:48.758706+0000","last_fullsized":"2024-04-26T17:53:48.758706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:01.746954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623200+0000","last_change":"2024-04-26T17:53:03.624433+0000","last_active":"2024-04-26T17:53:05.623200+0000","last_peered":"2024-04-26T17:53:05.623200+0000","last_clean":"2024-04-26T17:53:05.623200+0000","last_became_active":"2024-04-26T17:53:03.624135+0000","last_became_peered":"2024-04-26T17:53:03.624135+0000","last_unstale":"2024-04-26T17:53:05.623200+0000","last_undegraded":"2024-04-26T17:53:05.623200+0000","last_fullsized":"2024-04-26T17:53:05.623200+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:32:20.521917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765321+0000","last_change":"2024-04-26T17:53:03.623607+0000","last_active":"2024-04-26T17:53:49.765321+0000","last_peered":"2024-04-26T17:53:49.765321+0000","last_clean":"2024-04-26T17:53:49.765321+0000","last_became_active":"2024-04-26T17:53:03.621783+0000","last_became_peered":"2024-04-26T17:53:03.621783+0000","last_unstale":"2024-04-26T17:53:49.765321+0000","last_undegraded":"2024-04-26T17:53:49.765321+0000","last_fullsized":"2024-04-26T17:53:49.765321+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:38:59.872868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.756636+0000","last_change":"2024-04-26T17:53:03.614974+0000","last_active":"2024-04-26T17:53:48.756636+0000","last_peered":"2024-04-26T17:53:48.756636+0000","last_clean":"2024-04-26T17:53:48.756636+0000","last_became_active":"2024-04-26T17:53:03.614619+0000","last_became_peered":"2024-04-26T17:53:03.614619+0000","last_unstale":"2024-04-26T17:53:48.756636+0000","last_undegraded":"2024-04-26T17:53:48.756636+0000","last_fullsized":"2024-04-26T17:53:48.756636+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:50.729531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769611+0000","last_change":"2024-04-26T17:53:03.611140+0000","last_active":"2024-04-26T17:53:49.769611+0000","last_peered":"2024-04-26T17:53:49.769611+0000","last_clean":"2024-04-26T17:53:49.769611+0000","last_became_active":"2024-04-26T17:53:03.610549+0000","last_became_peered":"2024-04-26T17:53:03.610549+0000","last_unstale":"2024-04-26T17:53:49.769611+0000","last_undegraded":"2024-04-26T17:53:49.769611+0000","last_fullsized":"2024-04-26T17:53:49.769611+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:46:17.178598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.756562+0000","last_change":"2024-04-26T17:53:00.585971+0000","last_active":"2024-04-26T17:53:48.756562+0000","last_peered":"2024-04-26T17:53:48.756562+0000","last_clean":"2024-04-26T17:53:48.756562+0000","last_became_active":"2024-04-26T17:53:00.585648+0000","last_became_peered":"2024-04-26T17:53:00.585648+0000","last_unstale":"2024-04-26T17:53:48.756562+0000","last_undegraded":"2024-04-26T17:53:48.756562+0000","last_fullsized":"2024-04-26T17:53:48.756562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:00:24.082773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":2,"reported_epoch":89,"state":"peering","last_fresh":"2024-04-26T17:53:03.598208+0000","last_change":"2024-04-26T17:53:49.721525+0000","last_active":"2024-04-26T17:53:02.586248+0000","last_peered":"2024-04-26T17:53:02.586248+0000","last_clean":"2024-04-26T17:53:02.586248+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T17:53:03.598208+0000","last_undegraded":"2024-04-26T17:53:03.598208+0000","last_fullsized":"2024-04-26T17:53:03.598208+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.766024+0000","last_change":"2024-04-26T17:53:01.145438+0000","last_active":"2024-04-26T17:53:49.766024+0000","last_peered":"2024-04-26T17:53:49.766024+0000","last_clean":"2024-04-26T17:53:49.766024+0000","last_became_active":"2024-04-26T17:53:01.145030+0000","last_became_peered":"2024-04-26T17:53:01.145030+0000","last_unstale":"2024-04-26T17:53:49.766024+0000","last_undegraded":"2024-04-26T17:53:49.766024+0000","last_fullsized":"2024-04-26T17:53:49.766024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:15.650247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.759766+0000","last_change":"2024-04-26T17:53:03.622647+0000","last_active":"2024-04-26T17:53:48.759766+0000","last_peered":"2024-04-26T17:53:48.759766+0000","last_clean":"2024-04-26T17:53:48.759766+0000","last_became_active":"2024-04-26T17:53:03.622298+0000","last_became_peered":"2024-04-26T17:53:03.622298+0000","last_unstale":"2024-04-26T17:53:48.759766+0000","last_undegraded":"2024-04-26T17:53:48.759766+0000","last_fullsized":"2024-04-26T17:53:48.759766+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:37.230539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.775803+0000","last_change":"2024-04-26T17:53:49.775405+0000","last_active":"2024-04-26T17:53:49.775803+0000","last_peered":"2024-04-26T17:53:49.775803+0000","last_clean":"2024-04-26T17:53:49.775803+0000","last_became_active":"2024-04-26T17:53:49.770536+0000","last_became_peered":"2024-04-26T17:53:49.770536+0000","last_unstale":"2024-04-26T17:53:49.775803+0000","last_undegraded":"2024-04-26T17:53:49.775803+0000","last_fullsized":"2024-04-26T17:53:49.775803+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:34.643452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.624551+0000","last_change":"2024-04-26T17:53:03.617462+0000","last_active":"2024-04-26T17:53:05.624551+0000","last_peered":"2024-04-26T17:53:05.624551+0000","last_clean":"2024-04-26T17:53:05.624551+0000","last_became_active":"2024-04-26T17:53:03.617115+0000","last_became_peered":"2024-04-26T17:53:03.617115+0000","last_unstale":"2024-04-26T17:53:05.624551+0000","last_undegraded":"2024-04-26T17:53:05.624551+0000","last_fullsized":"2024-04-26T17:53:05.624551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:11.406146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767810+0000","last_change":"2024-04-26T17:53:00.583378+0000","last_active":"2024-04-26T17:53:49.767810+0000","last_peered":"2024-04-26T17:53:49.767810+0000","last_clean":"2024-04-26T17:53:49.767810+0000","last_became_active":"2024-04-26T17:53:00.583020+0000","last_became_peered":"2024-04-26T17:53:00.583020+0000","last_unstale":"2024-04-26T17:53:49.767810+0000","last_undegraded":"2024-04-26T17:53:49.767810+0000","last_fullsized":"2024-04-26T17:53:49.767810+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:24.089996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759512+0000","last_change":"2024-04-26T17:53:03.609557+0000","last_active":"2024-04-26T17:53:49.759512+0000","last_peered":"2024-04-26T17:53:49.759512+0000","last_clean":"2024-04-26T17:53:49.759512+0000","last_became_active":"2024-04-26T17:53:03.609028+0000","last_became_peered":"2024-04-26T17:53:03.609028+0000","last_unstale":"2024-04-26T17:53:49.759512+0000","last_undegraded":"2024-04-26T17:53:49.759512+0000","last_fullsized":"2024-04-26T17:53:49.759512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:39:22.860877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759190+0000","last_change":"2024-04-26T17:53:00.586683+0000","last_active":"2024-04-26T17:53:49.759190+0000","last_peered":"2024-04-26T17:53:49.759190+0000","last_clean":"2024-04-26T17:53:49.759190+0000","last_became_active":"2024-04-26T17:53:00.586272+0000","last_became_peered":"2024-04-26T17:53:00.586272+0000","last_unstale":"2024-04-26T17:53:49.759190+0000","last_undegraded":"2024-04-26T17:53:49.759190+0000","last_fullsized":"2024-04-26T17:53:49.759190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:20:29.068607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767148+0000","last_change":"2024-04-26T17:53:03.611073+0000","last_active":"2024-04-26T17:53:49.767148+0000","last_peered":"2024-04-26T17:53:49.767148+0000","last_clean":"2024-04-26T17:53:49.767148+0000","last_became_active":"2024-04-26T17:53:03.610701+0000","last_became_peered":"2024-04-26T17:53:03.610701+0000","last_unstale":"2024-04-26T17:53:49.767148+0000","last_undegraded":"2024-04-26T17:53:49.767148+0000","last_fullsized":"2024-04-26T17:53:49.767148+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:01:00.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764638+0000","last_change":"2024-04-26T17:53:00.586082+0000","last_active":"2024-04-26T17:53:49.764638+0000","last_peered":"2024-04-26T17:53:49.764638+0000","last_clean":"2024-04-26T17:53:49.764638+0000","last_became_active":"2024-04-26T17:53:00.585678+0000","last_became_peered":"2024-04-26T17:53:00.585678+0000","last_unstale":"2024-04-26T17:53:49.764638+0000","last_undegraded":"2024-04-26T17:53:49.764638+0000","last_fullsized":"2024-04-26T17:53:49.764638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:55:25.523846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759383+0000","last_change":"2024-04-26T17:53:03.600230+0000","last_active":"2024-04-26T17:53:49.759383+0000","last_peered":"2024-04-26T17:53:49.759383+0000","last_clean":"2024-04-26T17:53:49.759383+0000","last_became_active":"2024-04-26T17:53:03.599713+0000","last_became_peered":"2024-04-26T17:53:03.599713+0000","last_unstale":"2024-04-26T17:53:49.759383+0000","last_undegraded":"2024-04-26T17:53:49.759383+0000","last_fullsized":"2024-04-26T17:53:49.759383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:45.822561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:19.687458+0000","last_change":"2024-04-26T17:53:00.584588+0000","last_active":"2024-04-26T17:53:19.687458+0000","last_peered":"2024-04-26T17:53:19.687458+0000","last_clean":"2024-04-26T17:53:19.687458+0000","last_became_active":"2024-04-26T17:53:00.584284+0000","last_became_peered":"2024-04-26T17:53:00.584284+0000","last_unstale":"2024-04-26T17:53:19.687458+0000","last_undegraded":"2024-04-26T17:53:19.687458+0000","last_fullsized":"2024-04-26T17:53:19.687458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:08.428904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764864+0000","last_change":"2024-04-26T17:53:03.615237+0000","last_active":"2024-04-26T17:53:49.764864+0000","last_peered":"2024-04-26T17:53:49.764864+0000","last_clean":"2024-04-26T17:53:49.764864+0000","last_became_active":"2024-04-26T17:53:03.614704+0000","last_became_peered":"2024-04-26T17:53:03.614704+0000","last_unstale":"2024-04-26T17:53:49.764864+0000","last_undegraded":"2024-04-26T17:53:49.764864+0000","last_fullsized":"2024-04-26T17:53:49.764864+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:27.251839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758907+0000","last_change":"2024-04-26T17:53:00.589287+0000","last_active":"2024-04-26T17:53:49.758907+0000","last_peered":"2024-04-26T17:53:49.758907+0000","last_clean":"2024-04-26T17:53:49.758907+0000","last_became_active":"2024-04-26T17:53:00.588119+0000","last_became_peered":"2024-04-26T17:53:00.588119+0000","last_unstale":"2024-04-26T17:53:49.758907+0000","last_undegraded":"2024-04-26T17:53:49.758907+0000","last_fullsized":"2024-04-26T17:53:49.758907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:25.389777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.756432+0000","last_change":"2024-04-26T17:53:03.608061+0000","last_active":"2024-04-26T17:53:48.756432+0000","last_peered":"2024-04-26T17:53:48.756432+0000","last_clean":"2024-04-26T17:53:48.756432+0000","last_became_active":"2024-04-26T17:53:03.607624+0000","last_became_peered":"2024-04-26T17:53:03.607624+0000","last_unstale":"2024-04-26T17:53:48.756432+0000","last_undegraded":"2024-04-26T17:53:48.756432+0000","last_fullsized":"2024-04-26T17:53:48.756432+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:15.111011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764914+0000","last_change":"2024-04-26T17:53:00.588621+0000","last_active":"2024-04-26T17:53:49.764914+0000","last_peered":"2024-04-26T17:53:49.764914+0000","last_clean":"2024-04-26T17:53:49.764914+0000","last_became_active":"2024-04-26T17:53:00.587330+0000","last_became_peered":"2024-04-26T17:53:00.587330+0000","last_unstale":"2024-04-26T17:53:49.764914+0000","last_undegraded":"2024-04-26T17:53:49.764914+0000","last_fullsized":"2024-04-26T17:53:49.764914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:01:12.178610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.624899+0000","last_change":"2024-04-26T17:53:03.615723+0000","last_active":"2024-04-26T17:53:05.624899+0000","last_peered":"2024-04-26T17:53:05.624899+0000","last_clean":"2024-04-26T17:53:05.624899+0000","last_became_active":"2024-04-26T17:53:03.614931+0000","last_became_peered":"2024-04-26T17:53:03.614931+0000","last_unstale":"2024-04-26T17:53:05.624899+0000","last_undegraded":"2024-04-26T17:53:05.624899+0000","last_fullsized":"2024-04-26T17:53:05.624899+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:09.663876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767693+0000","last_change":"2024-04-26T17:53:00.589232+0000","last_active":"2024-04-26T17:53:49.767693+0000","last_peered":"2024-04-26T17:53:49.767693+0000","last_clean":"2024-04-26T17:53:49.767693+0000","last_became_active":"2024-04-26T17:53:00.588863+0000","last_became_peered":"2024-04-26T17:53:00.588863+0000","last_unstale":"2024-04-26T17:53:49.767693+0000","last_undegraded":"2024-04-26T17:53:49.767693+0000","last_fullsized":"2024-04-26T17:53:49.767693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:14.360782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.759664+0000","last_change":"2024-04-26T17:53:03.624923+0000","last_active":"2024-04-26T17:53:48.759664+0000","last_peered":"2024-04-26T17:53:48.759664+0000","last_clean":"2024-04-26T17:53:48.759664+0000","last_became_active":"2024-04-26T17:53:03.624620+0000","last_became_peered":"2024-04-26T17:53:03.624620+0000","last_unstale":"2024-04-26T17:53:48.759664+0000","last_undegraded":"2024-04-26T17:53:48.759664+0000","last_fullsized":"2024-04-26T17:53:48.759664+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:50:51.259563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759526+0000","last_change":"2024-04-26T17:53:01.146950+0000","last_active":"2024-04-26T17:53:49.759526+0000","last_peered":"2024-04-26T17:53:49.759526+0000","last_clean":"2024-04-26T17:53:49.759526+0000","last_became_active":"2024-04-26T17:53:01.146591+0000","last_became_peered":"2024-04-26T17:53:01.146591+0000","last_unstale":"2024-04-26T17:53:49.759526+0000","last_undegraded":"2024-04-26T17:53:49.759526+0000","last_fullsized":"2024-04-26T17:53:49.759526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:17.756660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.625037+0000","last_change":"2024-04-26T17:53:03.626944+0000","last_active":"2024-04-26T17:53:05.625037+0000","last_peered":"2024-04-26T17:53:05.625037+0000","last_clean":"2024-04-26T17:53:05.625037+0000","last_became_active":"2024-04-26T17:53:03.626594+0000","last_became_peered":"2024-04-26T17:53:03.626594+0000","last_unstale":"2024-04-26T17:53:05.625037+0000","last_undegraded":"2024-04-26T17:53:05.625037+0000","last_fullsized":"2024-04-26T17:53:05.625037+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:52:42.997251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'30","reported_seq":122,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:50.739715+0000","last_change":"2024-04-26T17:53:00.588353+0000","last_active":"2024-04-26T17:53:50.739715+0000","last_peered":"2024-04-26T17:53:50.739715+0000","last_clean":"2024-04-26T17:53:50.739715+0000","last_became_active":"2024-04-26T17:53:00.586351+0000","last_became_peered":"2024-04-26T17:53:00.586351+0000","last_unstale":"2024-04-26T17:53:50.739715+0000","last_undegraded":"2024-04-26T17:53:50.739715+0000","last_fullsized":"2024-04-26T17:53:50.739715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:41.660525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62335,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":66,"num_read_kb":89,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768625+0000","last_change":"2024-04-26T17:53:03.612052+0000","last_active":"2024-04-26T17:53:49.768625+0000","last_peered":"2024-04-26T17:53:49.768625+0000","last_clean":"2024-04-26T17:53:49.768625+0000","last_became_active":"2024-04-26T17:53:03.611733+0000","last_became_peered":"2024-04-26T17:53:03.611733+0000","last_unstale":"2024-04-26T17:53:49.768625+0000","last_undegraded":"2024-04-26T17:53:49.768625+0000","last_fullsized":"2024-04-26T17:53:49.768625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:05:23.958317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769171+0000","last_change":"2024-04-26T17:53:00.588284+0000","last_active":"2024-04-26T17:53:49.769171+0000","last_peered":"2024-04-26T17:53:49.769171+0000","last_clean":"2024-04-26T17:53:49.769171+0000","last_became_active":"2024-04-26T17:53:00.587942+0000","last_became_peered":"2024-04-26T17:53:00.587942+0000","last_unstale":"2024-04-26T17:53:49.769171+0000","last_undegraded":"2024-04-26T17:53:49.769171+0000","last_fullsized":"2024-04-26T17:53:49.769171+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:43.782971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.622973+0000","last_change":"2024-04-26T17:53:03.613092+0000","last_active":"2024-04-26T17:53:05.622973+0000","last_peered":"2024-04-26T17:53:05.622973+0000","last_clean":"2024-04-26T17:53:05.622973+0000","last_became_active":"2024-04-26T17:53:03.612773+0000","last_became_peered":"2024-04-26T17:53:03.612773+0000","last_unstale":"2024-04-26T17:53:05.622973+0000","last_undegraded":"2024-04-26T17:53:05.622973+0000","last_fullsized":"2024-04-26T17:53:05.622973+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:41:21.340375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759054+0000","last_change":"2024-04-26T17:53:00.588777+0000","last_active":"2024-04-26T17:53:49.759054+0000","last_peered":"2024-04-26T17:53:49.759054+0000","last_clean":"2024-04-26T17:53:49.759054+0000","last_became_active":"2024-04-26T17:53:00.588380+0000","last_became_peered":"2024-04-26T17:53:00.588380+0000","last_unstale":"2024-04-26T17:53:49.759054+0000","last_undegraded":"2024-04-26T17:53:49.759054+0000","last_fullsized":"2024-04-26T17:53:49.759054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:44.587381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767750+0000","last_change":"2024-04-26T17:53:03.620344+0000","last_active":"2024-04-26T17:53:49.767750+0000","last_peered":"2024-04-26T17:53:49.767750+0000","last_clean":"2024-04-26T17:53:49.767750+0000","last_became_active":"2024-04-26T17:53:03.619944+0000","last_became_peered":"2024-04-26T17:53:03.619944+0000","last_unstale":"2024-04-26T17:53:49.767750+0000","last_undegraded":"2024-04-26T17:53:49.767750+0000","last_fullsized":"2024-04-26T17:53:49.767750+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:08:31.801342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891069+0000","last_change":"2024-04-26T17:53:00.587733+0000","last_active":"2024-04-26T17:53:49.891069+0000","last_peered":"2024-04-26T17:53:49.891069+0000","last_clean":"2024-04-26T17:53:49.891069+0000","last_became_active":"2024-04-26T17:53:00.587373+0000","last_became_peered":"2024-04-26T17:53:00.587373+0000","last_unstale":"2024-04-26T17:53:49.891069+0000","last_undegraded":"2024-04-26T17:53:49.891069+0000","last_fullsized":"2024-04-26T17:53:49.891069+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:10.978944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767121+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:53:49.767121+0000","last_peered":"2024-04-26T17:53:49.767121+0000","last_clean":"2024-04-26T17:53:49.767121+0000","last_became_active":"2024-04-26T17:53:03.616588+0000","last_became_peered":"2024-04-26T17:53:03.616588+0000","last_unstale":"2024-04-26T17:53:49.767121+0000","last_undegraded":"2024-04-26T17:53:49.767121+0000","last_fullsized":"2024-04-26T17:53:49.767121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:46:38.419613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767233+0000","last_change":"2024-04-26T17:53:01.146922+0000","last_active":"2024-04-26T17:53:49.767233+0000","last_peered":"2024-04-26T17:53:49.767233+0000","last_clean":"2024-04-26T17:53:49.767233+0000","last_became_active":"2024-04-26T17:53:01.146520+0000","last_became_peered":"2024-04-26T17:53:01.146520+0000","last_unstale":"2024-04-26T17:53:49.767233+0000","last_undegraded":"2024-04-26T17:53:49.767233+0000","last_fullsized":"2024-04-26T17:53:49.767233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:12:49.238012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758964+0000","last_change":"2024-04-26T17:53:03.608513+0000","last_active":"2024-04-26T17:53:49.758964+0000","last_peered":"2024-04-26T17:53:49.758964+0000","last_clean":"2024-04-26T17:53:49.758964+0000","last_became_active":"2024-04-26T17:53:03.608167+0000","last_became_peered":"2024-04-26T17:53:03.608167+0000","last_unstale":"2024-04-26T17:53:49.758964+0000","last_undegraded":"2024-04-26T17:53:49.758964+0000","last_fullsized":"2024-04-26T17:53:49.758964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:03.002756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769191+0000","last_change":"2024-04-26T17:53:00.583878+0000","last_active":"2024-04-26T17:53:49.769191+0000","last_peered":"2024-04-26T17:53:49.769191+0000","last_clean":"2024-04-26T17:53:49.769191+0000","last_became_active":"2024-04-26T17:53:00.583159+0000","last_became_peered":"2024-04-26T17:53:00.583159+0000","last_unstale":"2024-04-26T17:53:49.769191+0000","last_undegraded":"2024-04-26T17:53:49.769191+0000","last_fullsized":"2024-04-26T17:53:49.769191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:49.850942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765106+0000","last_change":"2024-04-26T17:53:03.614467+0000","last_active":"2024-04-26T17:53:49.765106+0000","last_peered":"2024-04-26T17:53:49.765106+0000","last_clean":"2024-04-26T17:53:49.765106+0000","last_became_active":"2024-04-26T17:53:03.613972+0000","last_became_peered":"2024-04-26T17:53:03.613972+0000","last_unstale":"2024-04-26T17:53:49.765106+0000","last_undegraded":"2024-04-26T17:53:49.765106+0000","last_fullsized":"2024-04-26T17:53:49.765106+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:27.259777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"peering","last_fresh":"2024-04-26T17:53:05.613913+0000","last_change":"2024-04-26T17:53:48.756965+0000","last_active":"2024-04-26T17:53:48.756965+0000","last_peered":"2024-04-26T17:53:05.613913+0000","last_clean":"2024-04-26T17:53:05.613913+0000","last_became_active":"2024-04-26T17:53:01.145146+0000","last_became_peered":"2024-04-26T17:53:01.145146+0000","last_unstale":"2024-04-26T17:53:05.613913+0000","last_undegraded":"2024-04-26T17:53:05.613913+0000","last_fullsized":"2024-04-26T17:53:05.613913+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,11],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764767+0000","last_change":"2024-04-26T17:53:03.613888+0000","last_active":"2024-04-26T17:53:49.764767+0000","last_peered":"2024-04-26T17:53:49.764767+0000","last_clean":"2024-04-26T17:53:49.764767+0000","last_became_active":"2024-04-26T17:53:03.613499+0000","last_became_peered":"2024-04-26T17:53:03.613499+0000","last_unstale":"2024-04-26T17:53:49.764767+0000","last_undegraded":"2024-04-26T17:53:49.764767+0000","last_fullsized":"2024-04-26T17:53:49.764767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:15.549419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767375+0000","last_change":"2024-04-26T17:53:00.587803+0000","last_active":"2024-04-26T17:53:49.767375+0000","last_peered":"2024-04-26T17:53:49.767375+0000","last_clean":"2024-04-26T17:53:49.767375+0000","last_became_active":"2024-04-26T17:53:00.587382+0000","last_became_peered":"2024-04-26T17:53:00.587382+0000","last_unstale":"2024-04-26T17:53:49.767375+0000","last_undegraded":"2024-04-26T17:53:49.767375+0000","last_fullsized":"2024-04-26T17:53:49.767375+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:38.249478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769008+0000","last_change":"2024-04-26T17:53:49.769008+0000","last_active":"2024-04-26T17:53:49.769008+0000","last_peered":"2024-04-26T17:53:49.769008+0000","last_clean":"2024-04-26T17:53:49.769008+0000","last_became_active":"2024-04-26T17:53:49.768625+0000","last_became_peered":"2024-04-26T17:53:49.768625+0000","last_unstale":"2024-04-26T17:53:49.769008+0000","last_undegraded":"2024-04-26T17:53:49.769008+0000","last_fullsized":"2024-04-26T17:53:49.769008+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:27:46.412625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759215+0000","last_change":"2024-04-26T17:53:03.612899+0000","last_active":"2024-04-26T17:53:49.759215+0000","last_peered":"2024-04-26T17:53:49.759215+0000","last_clean":"2024-04-26T17:53:49.759215+0000","last_became_active":"2024-04-26T17:53:03.612564+0000","last_became_peered":"2024-04-26T17:53:03.612564+0000","last_unstale":"2024-04-26T17:53:49.759215+0000","last_undegraded":"2024-04-26T17:53:49.759215+0000","last_fullsized":"2024-04-26T17:53:49.759215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:07.368256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":52,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:46.732353+0000","last_change":"2024-04-26T17:53:01.145471+0000","last_active":"2024-04-26T17:53:46.732353+0000","last_peered":"2024-04-26T17:53:46.732353+0000","last_clean":"2024-04-26T17:53:46.732353+0000","last_became_active":"2024-04-26T17:53:01.145040+0000","last_became_peered":"2024-04-26T17:53:01.145040+0000","last_unstale":"2024-04-26T17:53:46.732353+0000","last_undegraded":"2024-04-26T17:53:46.732353+0000","last_fullsized":"2024-04-26T17:53:46.732353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:46:27.437329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765218+0000","last_change":"2024-04-26T17:53:03.618600+0000","last_active":"2024-04-26T17:53:49.765218+0000","last_peered":"2024-04-26T17:53:49.765218+0000","last_clean":"2024-04-26T17:53:49.765218+0000","last_became_active":"2024-04-26T17:53:03.618139+0000","last_became_peered":"2024-04-26T17:53:03.618139+0000","last_unstale":"2024-04-26T17:53:49.765218+0000","last_undegraded":"2024-04-26T17:53:49.765218+0000","last_fullsized":"2024-04-26T17:53:49.765218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:56.007659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891071+0000","last_change":"2024-04-26T17:53:00.586386+0000","last_active":"2024-04-26T17:53:49.891071+0000","last_peered":"2024-04-26T17:53:49.891071+0000","last_clean":"2024-04-26T17:53:49.891071+0000","last_became_active":"2024-04-26T17:53:00.585953+0000","last_became_peered":"2024-04-26T17:53:00.585953+0000","last_unstale":"2024-04-26T17:53:49.891071+0000","last_undegraded":"2024-04-26T17:53:49.891071+0000","last_fullsized":"2024-04-26T17:53:49.891071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:39:57.160983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":357,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767252+0000","last_change":"2024-04-26T17:51:38.839853+0000","last_active":"2024-04-26T17:53:49.767252+0000","last_peered":"2024-04-26T17:53:49.767252+0000","last_clean":"2024-04-26T17:53:49.767252+0000","last_became_active":"2024-04-26T17:51:38.839318+0000","last_became_peered":"2024-04-26T17:51:38.839318+0000","last_unstale":"2024-04-26T17:53:49.767252+0000","last_undegraded":"2024-04-26T17:53:49.767252+0000","last_fullsized":"2024-04-26T17:53:49.767252+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_clean_scrub_stamp":"2024-04-26T17:48:12.577543+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:08:50.683663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.622208+0000","last_change":"2024-04-26T17:53:03.614170+0000","last_active":"2024-04-26T17:53:05.622208+0000","last_peered":"2024-04-26T17:53:05.622208+0000","last_clean":"2024-04-26T17:53:05.622208+0000","last_became_active":"2024-04-26T17:53:03.613836+0000","last_became_peered":"2024-04-26T17:53:03.613836+0000","last_unstale":"2024-04-26T17:53:05.622208+0000","last_undegraded":"2024-04-26T17:53:05.622208+0000","last_fullsized":"2024-04-26T17:53:05.622208+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:48.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767374+0000","last_change":"2024-04-26T17:53:00.587319+0000","last_active":"2024-04-26T17:53:49.767374+0000","last_peered":"2024-04-26T17:53:49.767374+0000","last_clean":"2024-04-26T17:53:49.767374+0000","last_became_active":"2024-04-26T17:53:00.586983+0000","last_became_peered":"2024-04-26T17:53:00.586983+0000","last_unstale":"2024-04-26T17:53:49.767374+0000","last_undegraded":"2024-04-26T17:53:49.767374+0000","last_fullsized":"2024-04-26T17:53:49.767374+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:03:43.470013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768739+0000","last_change":"2024-04-26T17:53:01.145672+0000","last_active":"2024-04-26T17:53:49.768739+0000","last_peered":"2024-04-26T17:53:49.768739+0000","last_clean":"2024-04-26T17:53:49.768739+0000","last_became_active":"2024-04-26T17:53:01.145274+0000","last_became_peered":"2024-04-26T17:53:01.145274+0000","last_unstale":"2024-04-26T17:53:49.768739+0000","last_undegraded":"2024-04-26T17:53:49.768739+0000","last_fullsized":"2024-04-26T17:53:49.768739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:21.122352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623379+0000","last_change":"2024-04-26T17:53:03.611232+0000","last_active":"2024-04-26T17:53:05.623379+0000","last_peered":"2024-04-26T17:53:05.623379+0000","last_clean":"2024-04-26T17:53:05.623379+0000","last_became_active":"2024-04-26T17:53:03.610905+0000","last_became_peered":"2024-04-26T17:53:03.610905+0000","last_unstale":"2024-04-26T17:53:05.623379+0000","last_undegraded":"2024-04-26T17:53:05.623379+0000","last_fullsized":"2024-04-26T17:53:05.623379+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:31.426237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'11","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.756696+0000","last_change":"2024-04-26T17:53:00.580220+0000","last_active":"2024-04-26T17:53:48.756696+0000","last_peered":"2024-04-26T17:53:48.756696+0000","last_clean":"2024-04-26T17:53:48.756696+0000","last_became_active":"2024-04-26T17:53:00.579818+0000","last_became_peered":"2024-04-26T17:53:00.579818+0000","last_unstale":"2024-04-26T17:53:48.756696+0000","last_undegraded":"2024-04-26T17:53:48.756696+0000","last_fullsized":"2024-04-26T17:53:48.756696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:15.260489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765821+0000","last_change":"2024-04-26T17:53:03.610882+0000","last_active":"2024-04-26T17:53:49.765821+0000","last_peered":"2024-04-26T17:53:49.765821+0000","last_clean":"2024-04-26T17:53:49.765821+0000","last_became_active":"2024-04-26T17:53:03.609979+0000","last_became_peered":"2024-04-26T17:53:03.609979+0000","last_unstale":"2024-04-26T17:53:49.765821+0000","last_undegraded":"2024-04-26T17:53:49.765821+0000","last_fullsized":"2024-04-26T17:53:49.765821+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:33.167579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891536+0000","last_change":"2024-04-26T17:53:01.358555+0000","last_active":"2024-04-26T17:53:49.891536+0000","last_peered":"2024-04-26T17:53:49.891536+0000","last_clean":"2024-04-26T17:53:49.891536+0000","last_became_active":"2024-04-26T17:53:01.358138+0000","last_became_peered":"2024-04-26T17:53:01.358138+0000","last_unstale":"2024-04-26T17:53:49.891536+0000","last_undegraded":"2024-04-26T17:53:49.891536+0000","last_fullsized":"2024-04-26T17:53:49.891536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:14.886609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.759505+0000","last_change":"2024-04-26T17:53:03.618816+0000","last_active":"2024-04-26T17:53:48.759505+0000","last_peered":"2024-04-26T17:53:48.759505+0000","last_clean":"2024-04-26T17:53:48.759505+0000","last_became_active":"2024-04-26T17:53:03.618184+0000","last_became_peered":"2024-04-26T17:53:03.618184+0000","last_unstale":"2024-04-26T17:53:48.759505+0000","last_undegraded":"2024-04-26T17:53:48.759505+0000","last_fullsized":"2024-04-26T17:53:48.759505+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:07.514516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:18.679325+0000","last_change":"2024-04-26T17:53:01.147081+0000","last_active":"2024-04-26T17:53:18.679325+0000","last_peered":"2024-04-26T17:53:18.679325+0000","last_clean":"2024-04-26T17:53:18.679325+0000","last_became_active":"2024-04-26T17:53:01.146698+0000","last_became_peered":"2024-04-26T17:53:01.146698+0000","last_unstale":"2024-04-26T17:53:18.679325+0000","last_undegraded":"2024-04-26T17:53:18.679325+0000","last_fullsized":"2024-04-26T17:53:18.679325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:44:19.927316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767053+0000","last_change":"2024-04-26T17:53:03.612592+0000","last_active":"2024-04-26T17:53:49.767053+0000","last_peered":"2024-04-26T17:53:49.767053+0000","last_clean":"2024-04-26T17:53:49.767053+0000","last_became_active":"2024-04-26T17:53:03.612274+0000","last_became_peered":"2024-04-26T17:53:03.612274+0000","last_unstale":"2024-04-26T17:53:49.767053+0000","last_undegraded":"2024-04-26T17:53:49.767053+0000","last_fullsized":"2024-04-26T17:53:49.767053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:30:48.461947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764834+0000","last_change":"2024-04-26T17:53:00.586430+0000","last_active":"2024-04-26T17:53:49.764834+0000","last_peered":"2024-04-26T17:53:49.764834+0000","last_clean":"2024-04-26T17:53:49.764834+0000","last_became_active":"2024-04-26T17:53:00.586004+0000","last_became_peered":"2024-04-26T17:53:00.586004+0000","last_unstale":"2024-04-26T17:53:49.764834+0000","last_undegraded":"2024-04-26T17:53:49.764834+0000","last_fullsized":"2024-04-26T17:53:49.764834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:47:18.224890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764726+0000","last_change":"2024-04-26T17:53:03.623081+0000","last_active":"2024-04-26T17:53:49.764726+0000","last_peered":"2024-04-26T17:53:49.764726+0000","last_clean":"2024-04-26T17:53:49.764726+0000","last_became_active":"2024-04-26T17:53:03.622761+0000","last_became_peered":"2024-04-26T17:53:03.622761+0000","last_unstale":"2024-04-26T17:53:49.764726+0000","last_undegraded":"2024-04-26T17:53:49.764726+0000","last_fullsized":"2024-04-26T17:53:49.764726+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:39.520075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768849+0000","last_change":"2024-04-26T17:53:00.583013+0000","last_active":"2024-04-26T17:53:49.768849+0000","last_peered":"2024-04-26T17:53:49.768849+0000","last_clean":"2024-04-26T17:53:49.768849+0000","last_became_active":"2024-04-26T17:53:00.582653+0000","last_became_peered":"2024-04-26T17:53:00.582653+0000","last_unstale":"2024-04-26T17:53:49.768849+0000","last_undegraded":"2024-04-26T17:53:49.768849+0000","last_fullsized":"2024-04-26T17:53:49.768849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:09.448621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765042+0000","last_change":"2024-04-26T17:53:03.618232+0000","last_active":"2024-04-26T17:53:49.765042+0000","last_peered":"2024-04-26T17:53:49.765042+0000","last_clean":"2024-04-26T17:53:49.765042+0000","last_became_active":"2024-04-26T17:53:03.617906+0000","last_became_peered":"2024-04-26T17:53:03.617906+0000","last_unstale":"2024-04-26T17:53:49.765042+0000","last_undegraded":"2024-04-26T17:53:49.765042+0000","last_fullsized":"2024-04-26T17:53:49.765042+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:17.357398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768301+0000","last_change":"2024-04-26T17:53:00.586565+0000","last_active":"2024-04-26T17:53:49.768301+0000","last_peered":"2024-04-26T17:53:49.768301+0000","last_clean":"2024-04-26T17:53:49.768301+0000","last_became_active":"2024-04-26T17:53:00.586161+0000","last_became_peered":"2024-04-26T17:53:00.586161+0000","last_unstale":"2024-04-26T17:53:49.768301+0000","last_undegraded":"2024-04-26T17:53:49.768301+0000","last_fullsized":"2024-04-26T17:53:49.768301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:56:25.294055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767400+0000","last_change":"2024-04-26T17:53:03.620340+0000","last_active":"2024-04-26T17:53:49.767400+0000","last_peered":"2024-04-26T17:53:49.767400+0000","last_clean":"2024-04-26T17:53:49.767400+0000","last_became_active":"2024-04-26T17:53:03.619897+0000","last_became_peered":"2024-04-26T17:53:03.619897+0000","last_unstale":"2024-04-26T17:53:49.767400+0000","last_undegraded":"2024-04-26T17:53:49.767400+0000","last_fullsized":"2024-04-26T17:53:49.767400+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:12:40.662070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769496+0000","last_change":"2024-04-26T17:53:01.358131+0000","last_active":"2024-04-26T17:53:49.769496+0000","last_peered":"2024-04-26T17:53:49.769496+0000","last_clean":"2024-04-26T17:53:49.769496+0000","last_became_active":"2024-04-26T17:53:01.357796+0000","last_became_peered":"2024-04-26T17:53:01.357796+0000","last_unstale":"2024-04-26T17:53:49.769496+0000","last_undegraded":"2024-04-26T17:53:49.769496+0000","last_fullsized":"2024-04-26T17:53:49.769496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:47.849873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767708+0000","last_change":"2024-04-26T17:53:03.616133+0000","last_active":"2024-04-26T17:53:49.767708+0000","last_peered":"2024-04-26T17:53:49.767708+0000","last_clean":"2024-04-26T17:53:49.767708+0000","last_became_active":"2024-04-26T17:53:03.615797+0000","last_became_peered":"2024-04-26T17:53:03.615797+0000","last_unstale":"2024-04-26T17:53:49.767708+0000","last_undegraded":"2024-04-26T17:53:49.767708+0000","last_fullsized":"2024-04-26T17:53:49.767708+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:05.698489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759241+0000","last_change":"2024-04-26T17:53:00.587696+0000","last_active":"2024-04-26T17:53:49.759241+0000","last_peered":"2024-04-26T17:53:49.759241+0000","last_clean":"2024-04-26T17:53:49.759241+0000","last_became_active":"2024-04-26T17:53:00.587305+0000","last_became_peered":"2024-04-26T17:53:00.587305+0000","last_unstale":"2024-04-26T17:53:49.759241+0000","last_undegraded":"2024-04-26T17:53:49.759241+0000","last_fullsized":"2024-04-26T17:53:49.759241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:39.185753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767528+0000","last_change":"2024-04-26T17:53:03.613076+0000","last_active":"2024-04-26T17:53:49.767528+0000","last_peered":"2024-04-26T17:53:49.767528+0000","last_clean":"2024-04-26T17:53:49.767528+0000","last_became_active":"2024-04-26T17:53:03.612746+0000","last_became_peered":"2024-04-26T17:53:03.612746+0000","last_unstale":"2024-04-26T17:53:49.767528+0000","last_undegraded":"2024-04-26T17:53:49.767528+0000","last_fullsized":"2024-04-26T17:53:49.767528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:20.920154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.629072+0000","last_change":"2024-04-26T17:53:00.582575+0000","last_active":"2024-04-26T17:53:05.629072+0000","last_peered":"2024-04-26T17:53:05.629072+0000","last_clean":"2024-04-26T17:53:05.629072+0000","last_became_active":"2024-04-26T17:53:00.582215+0000","last_became_peered":"2024-04-26T17:53:00.582215+0000","last_unstale":"2024-04-26T17:53:05.629072+0000","last_undegraded":"2024-04-26T17:53:05.629072+0000","last_fullsized":"2024-04-26T17:53:05.629072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:35:12.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767871+0000","last_change":"2024-04-26T17:53:03.616269+0000","last_active":"2024-04-26T17:53:49.767871+0000","last_peered":"2024-04-26T17:53:49.767871+0000","last_clean":"2024-04-26T17:53:49.767871+0000","last_became_active":"2024-04-26T17:53:03.615896+0000","last_became_peered":"2024-04-26T17:53:03.615896+0000","last_unstale":"2024-04-26T17:53:49.767871+0000","last_undegraded":"2024-04-26T17:53:49.767871+0000","last_fullsized":"2024-04-26T17:53:49.767871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:12.111392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764802+0000","last_change":"2024-04-26T17:53:03.621589+0000","last_active":"2024-04-26T17:53:49.764802+0000","last_peered":"2024-04-26T17:53:49.764802+0000","last_clean":"2024-04-26T17:53:49.764802+0000","last_became_active":"2024-04-26T17:53:03.621211+0000","last_became_peered":"2024-04-26T17:53:03.621211+0000","last_unstale":"2024-04-26T17:53:49.764802+0000","last_undegraded":"2024-04-26T17:53:49.764802+0000","last_fullsized":"2024-04-26T17:53:49.764802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.487476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768626+0000","last_change":"2024-04-26T17:53:03.624536+0000","last_active":"2024-04-26T17:53:49.768626+0000","last_peered":"2024-04-26T17:53:49.768626+0000","last_clean":"2024-04-26T17:53:49.768626+0000","last_became_active":"2024-04-26T17:53:03.624234+0000","last_became_peered":"2024-04-26T17:53:03.624234+0000","last_unstale":"2024-04-26T17:53:49.768626+0000","last_undegraded":"2024-04-26T17:53:49.768626+0000","last_fullsized":"2024-04-26T17:53:49.768626+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:31:36.322688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768748+0000","last_change":"2024-04-26T17:53:03.620096+0000","last_active":"2024-04-26T17:53:49.768748+0000","last_peered":"2024-04-26T17:53:49.768748+0000","last_clean":"2024-04-26T17:53:49.768748+0000","last_became_active":"2024-04-26T17:53:03.619729+0000","last_became_peered":"2024-04-26T17:53:03.619729+0000","last_unstale":"2024-04-26T17:53:49.768748+0000","last_undegraded":"2024-04-26T17:53:49.768748+0000","last_fullsized":"2024-04-26T17:53:49.768748+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:17.987728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"peering","last_fresh":"2024-04-26T17:53:05.626877+0000","last_change":"2024-04-26T17:53:48.760704+0000","last_active":"2024-04-26T17:53:48.760704+0000","last_peered":"2024-04-26T17:53:05.626877+0000","last_clean":"2024-04-26T17:53:05.626877+0000","last_became_active":"2024-04-26T17:53:03.618062+0000","last_became_peered":"2024-04-26T17:53:03.618062+0000","last_unstale":"2024-04-26T17:53:05.626877+0000","last_undegraded":"2024-04-26T17:53:05.626877+0000","last_fullsized":"2024-04-26T17:53:05.626877+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.622725+0000","last_change":"2024-04-26T17:53:03.619193+0000","last_active":"2024-04-26T17:53:05.622725+0000","last_peered":"2024-04-26T17:53:05.622725+0000","last_clean":"2024-04-26T17:53:05.622725+0000","last_became_active":"2024-04-26T17:53:03.618776+0000","last_became_peered":"2024-04-26T17:53:03.618776+0000","last_unstale":"2024-04-26T17:53:05.622725+0000","last_undegraded":"2024-04-26T17:53:05.622725+0000","last_fullsized":"2024-04-26T17:53:05.622725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:07:24.811683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765044+0000","last_change":"2024-04-26T17:53:03.621366+0000","last_active":"2024-04-26T17:53:49.765044+0000","last_peered":"2024-04-26T17:53:49.765044+0000","last_clean":"2024-04-26T17:53:49.765044+0000","last_became_active":"2024-04-26T17:53:03.621048+0000","last_became_peered":"2024-04-26T17:53:03.621048+0000","last_unstale":"2024-04-26T17:53:49.765044+0000","last_undegraded":"2024-04-26T17:53:49.765044+0000","last_fullsized":"2024-04-26T17:53:49.765044+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:31:06.484880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769806+0000","last_change":"2024-04-26T17:53:03.619383+0000","last_active":"2024-04-26T17:53:49.769806+0000","last_peered":"2024-04-26T17:53:49.769806+0000","last_clean":"2024-04-26T17:53:49.769806+0000","last_became_active":"2024-04-26T17:53:03.619017+0000","last_became_peered":"2024-04-26T17:53:03.619017+0000","last_unstale":"2024-04-26T17:53:49.769806+0000","last_undegraded":"2024-04-26T17:53:49.769806+0000","last_fullsized":"2024-04-26T17:53:49.769806+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:20.091959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.770695+0000","last_change":"2024-04-26T17:53:49.770695+0000","last_active":"2024-04-26T17:53:49.770695+0000","last_peered":"2024-04-26T17:53:49.770695+0000","last_clean":"2024-04-26T17:53:49.770695+0000","last_became_active":"2024-04-26T17:53:49.770303+0000","last_became_peered":"2024-04-26T17:53:49.770303+0000","last_unstale":"2024-04-26T17:53:49.770695+0000","last_undegraded":"2024-04-26T17:53:49.770695+0000","last_fullsized":"2024-04-26T17:53:49.770695+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:22.830129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758969+0000","last_change":"2024-04-26T17:53:03.605521+0000","last_active":"2024-04-26T17:53:49.758969+0000","last_peered":"2024-04-26T17:53:49.758969+0000","last_clean":"2024-04-26T17:53:49.758969+0000","last_became_active":"2024-04-26T17:53:03.605145+0000","last_became_peered":"2024-04-26T17:53:03.605145+0000","last_unstale":"2024-04-26T17:53:49.758969+0000","last_undegraded":"2024-04-26T17:53:49.758969+0000","last_fullsized":"2024-04-26T17:53:49.758969+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:58:09.856021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764723+0000","last_change":"2024-04-26T17:53:03.610866+0000","last_active":"2024-04-26T17:53:49.764723+0000","last_peered":"2024-04-26T17:53:49.764723+0000","last_clean":"2024-04-26T17:53:49.764723+0000","last_became_active":"2024-04-26T17:53:03.610326+0000","last_became_peered":"2024-04-26T17:53:03.610326+0000","last_unstale":"2024-04-26T17:53:49.764723+0000","last_undegraded":"2024-04-26T17:53:49.764723+0000","last_fullsized":"2024-04-26T17:53:49.764723+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:20.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.624239+0000","last_change":"2024-04-26T17:53:03.615336+0000","last_active":"2024-04-26T17:53:05.624239+0000","last_peered":"2024-04-26T17:53:05.624239+0000","last_clean":"2024-04-26T17:53:05.624239+0000","last_became_active":"2024-04-26T17:53:03.614447+0000","last_became_peered":"2024-04-26T17:53:03.614447+0000","last_unstale":"2024-04-26T17:53:05.624239+0000","last_undegraded":"2024-04-26T17:53:05.624239+0000","last_fullsized":"2024-04-26T17:53:05.624239+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:36:10.091929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T17:53:05.623142+0000","last_change":"2024-04-26T17:53:03.614863+0000","last_active":"2024-04-26T17:53:05.623142+0000","last_peered":"2024-04-26T17:53:05.623142+0000","last_clean":"2024-04-26T17:53:05.623142+0000","last_became_active":"2024-04-26T17:53:03.614520+0000","last_became_peered":"2024-04-26T17:53:03.614520+0000","last_unstale":"2024-04-26T17:53:05.623142+0000","last_undegraded":"2024-04-26T17:53:05.623142+0000","last_fullsized":"2024-04-26T17:53:05.623142+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:51.367785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764960+0000","last_change":"2024-04-26T17:53:03.618521+0000","last_active":"2024-04-26T17:53:49.764960+0000","last_peered":"2024-04-26T17:53:49.764960+0000","last_clean":"2024-04-26T17:53:49.764960+0000","last_became_active":"2024-04-26T17:53:03.617658+0000","last_became_peered":"2024-04-26T17:53:03.617658+0000","last_unstale":"2024-04-26T17:53:49.764960+0000","last_undegraded":"2024-04-26T17:53:49.764960+0000","last_fullsized":"2024-04-26T17:53:49.764960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:39.813098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280045,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66336,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":197,"num_read_kb":163,"num_write":165,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":194751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383705,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":479802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645342,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1130496,"data_stored":1129361,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":463023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070503,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":479958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332141,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":1071888,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495666,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":462412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":520283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":472268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":543296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182464,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":471308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":479864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":1063142,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":58078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":63913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T17:53:53.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T17:53:54.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:53 smithi031 ceph-mon[29164]: pgmap v322: 97 pgs: 3 peering, 94 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T17:53:54.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:53 smithi097 ceph-mon[32419]: pgmap v322: 97 pgs: 3 peering, 94 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T17:53:54.163 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:54.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:53 smithi186 ceph-mon[32797]: pgmap v322: 97 pgs: 3 peering, 94 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T17:53:54.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:54 smithi031 ceph-mon[29164]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:53:55.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:54 smithi097 ceph-mon[32419]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:53:55.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:54 smithi186 ceph-mon[32797]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:53:55.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.801+0000 7fac61383700 1 -- 172.21.15.31:0/4276335331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c06c4f0 msgr2=0x7fac5c10f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:55.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.801+0000 7fac61383700 1 --2- 172.21.15.31:0/4276335331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c06c4f0 0x7fac5c10f930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fac50009990 tx=0x7fac50009ca0 comp rx=0 tx=0).stop 2024-04-26T17:53:55.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.801+0000 7fac61383700 1 -- 172.21.15.31:0/4276335331 shutdown_connections 2024-04-26T17:53:55.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.801+0000 7fac61383700 1 --2- 172.21.15.31:0/4276335331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c06bbd0 0x7fac5c06bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:55.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.801+0000 7fac61383700 1 --2- 172.21.15.31:0/4276335331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5c10fe70 0x7fac5c112260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:55.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.801+0000 7fac61383700 1 --2- 172.21.15.31:0/4276335331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c06c4f0 0x7fac5c10f930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:55.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.801+0000 7fac61383700 1 -- 172.21.15.31:0/4276335331 >> 172.21.15.31:0/4276335331 conn(0x7fac5c0793c0 msgr2=0x7fac5c0797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:55.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.801+0000 7fac61383700 1 -- 172.21.15.31:0/4276335331 shutdown_connections 2024-04-26T17:53:55.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.801+0000 7fac61383700 1 -- 172.21.15.31:0/4276335331 wait complete. 2024-04-26T17:53:55.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.802+0000 7fac61383700 1 Processor -- start 2024-04-26T17:53:55.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.802+0000 7fac61383700 1 -- start start 2024-04-26T17:53:55.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.802+0000 7fac61383700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c06bbd0 0x7fac5c19bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:55.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.802+0000 7fac61383700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c06c4f0 0x7fac5c19c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:55.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.802+0000 7fac61383700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5c10fe70 0x7fac5c1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:55.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac61383700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1158a0 con 0x7fac5c06bbd0 2024-04-26T17:53:55.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac61383700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c115720 con 0x7fac5c06c4f0 2024-04-26T17:53:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac61383700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c115a20 con 0x7fac5c10fe70 2024-04-26T17:53:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac5affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c06bbd0 0x7fac5c19bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac5affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c06bbd0 0x7fac5c19bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36572/0 (socket says 172.21.15.31:36572) 2024-04-26T17:53:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac5affd700 1 -- 172.21.15.31:0/3279197220 learned_addr learned my addr 172.21.15.31:0/3279197220 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac5a7fc700 1 --2- 172.21.15.31:0/3279197220 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c06c4f0 0x7fac5c19c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac5b7fe700 1 --2- 172.21.15.31:0/3279197220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5c10fe70 0x7fac5c1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac5affd700 1 -- 172.21.15.31:0/3279197220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5c10fe70 msgr2=0x7fac5c1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac5affd700 1 --2- 172.21.15.31:0/3279197220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5c10fe70 0x7fac5c1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac5affd700 1 -- 172.21.15.31:0/3279197220 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c06c4f0 msgr2=0x7fac5c19c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac5affd700 1 --2- 172.21.15.31:0/3279197220 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c06c4f0 0x7fac5c19c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac5affd700 1 -- 172.21.15.31:0/3279197220 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5c1a0d70 con 0x7fac5c06bbd0 2024-04-26T17:53:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac5affd700 1 --2- 172.21.15.31:0/3279197220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c06bbd0 0x7fac5c19bdb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00d950 tx=0x7fac4c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac43fff700 1 -- 172.21.15.31:0/3279197220 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c016070 con 0x7fac5c06bbd0 2024-04-26T17:53:55.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac61383700 1 -- 172.21.15.31:0/3279197220 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5c1a1060 con 0x7fac5c06bbd0 2024-04-26T17:53:55.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac43fff700 1 -- 172.21.15.31:0/3279197220 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac4c009df0 con 0x7fac5c06bbd0 2024-04-26T17:53:55.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac43fff700 1 -- 172.21.15.31:0/3279197220 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00f3c0 con 0x7fac5c06bbd0 2024-04-26T17:53:55.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.803+0000 7fac61383700 1 -- 172.21.15.31:0/3279197220 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5c1a88e0 con 0x7fac5c06bbd0 2024-04-26T17:53:55.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.804+0000 7fac61383700 1 -- 172.21.15.31:0/3279197220 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac5c06a110 con 0x7fac5c06bbd0 2024-04-26T17:53:55.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.806+0000 7fac43fff700 1 -- 172.21.15.31:0/3279197220 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fac4c019070 con 0x7fac5c06bbd0 2024-04-26T17:53:55.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.807+0000 7fac43fff700 1 --2- 172.21.15.31:0/3279197220 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac440a3cd0 0x7fac440a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:55.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.807+0000 7fac5a7fc700 1 --2- 172.21.15.31:0/3279197220 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac440a3cd0 0x7fac440a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:55.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.807+0000 7fac43fff700 1 -- 172.21.15.31:0/3279197220 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fac4c0d0130 con 0x7fac5c06bbd0 2024-04-26T17:53:55.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.807+0000 7fac5a7fc700 1 --2- 172.21.15.31:0/3279197220 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac440a3cd0 0x7fac440a6190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fac5000b5c0 tx=0x7fac50005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:55.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:55.811+0000 7fac43fff700 1 -- 172.21.15.31:0/3279197220 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac4c09c9e0 con 0x7fac5c06bbd0 2024-04-26T17:53:56.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:56.018+0000 7fac61383700 1 -- 172.21.15.31:0/3279197220 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fac5c063860 con 0x7fac440a3cd0 2024-04-26T17:53:56.018 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:55 smithi031 ceph-mon[29164]: pgmap v323: 97 pgs: 2 peering, 95 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 639 B/s wr, 5 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T17:53:56.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:56.025+0000 7fac43fff700 1 -- 172.21.15.31:0/3279197220 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258786 (secure 0 0 0) 0x7fac5c063860 con 0x7fac440a3cd0 2024-04-26T17:53:56.029 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:53:56.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:56.028+0000 7fac61383700 1 -- 172.21.15.31:0/3279197220 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac440a3cd0 msgr2=0x7fac440a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:56.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:56.028+0000 7fac61383700 1 --2- 172.21.15.31:0/3279197220 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac440a3cd0 0x7fac440a6190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fac5000b5c0 tx=0x7fac50005960 comp rx=0 tx=0).stop 2024-04-26T17:53:56.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:56.028+0000 7fac61383700 1 -- 172.21.15.31:0/3279197220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c06bbd0 msgr2=0x7fac5c19bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:56.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:56.028+0000 7fac61383700 1 --2- 172.21.15.31:0/3279197220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c06bbd0 0x7fac5c19bdb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00d950 tx=0x7fac4c00dc60 comp rx=0 tx=0).stop 2024-04-26T17:53:56.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:56.028+0000 7fac61383700 1 -- 172.21.15.31:0/3279197220 shutdown_connections 2024-04-26T17:53:56.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:56.028+0000 7fac61383700 1 --2- 172.21.15.31:0/3279197220 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac440a3cd0 0x7fac440a6190 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:56.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:56.028+0000 7fac61383700 1 --2- 172.21.15.31:0/3279197220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5c10fe70 0x7fac5c1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:56.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:56.028+0000 7fac61383700 1 --2- 172.21.15.31:0/3279197220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac5c06bbd0 0x7fac5c19bdb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:56.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:56.028+0000 7fac61383700 1 --2- 172.21.15.31:0/3279197220 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c06c4f0 0x7fac5c19c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:56.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:56.028+0000 7fac61383700 1 -- 172.21.15.31:0/3279197220 >> 172.21.15.31:0/3279197220 conn(0x7fac5c0793c0 msgr2=0x7fac5c1114e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:56.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:56.028+0000 7fac61383700 1 -- 172.21.15.31:0/3279197220 shutdown_connections 2024-04-26T17:53:56.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:56.028+0000 7fac61383700 1 -- 172.21.15.31:0/3279197220 wait complete. 2024-04-26T17:53:56.032 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-26T17:53:56.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:55 smithi097 ceph-mon[32419]: pgmap v323: 97 pgs: 2 peering, 95 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 639 B/s wr, 5 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T17:53:56.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:55 smithi186 ceph-mon[32797]: pgmap v323: 97 pgs: 2 peering, 95 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 639 B/s wr, 5 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T17:53:56.691 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-04-26T17:53:54.318439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656938,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":338,"num_read_kb":287,"num_write":348,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":288,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361156,"kb_used_data":11960,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580668,"statfs":{"total":1151940427776,"available":1151570604032,"internally_reserved":0,"allocated":12247040,"data_stored":7683322,"data_compressed":106863,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4257,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":26,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001971"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765218+0000","last_change":"2024-04-26T17:53:03.616592+0000","last_active":"2024-04-26T17:53:49.765218+0000","last_peered":"2024-04-26T17:53:49.765218+0000","last_clean":"2024-04-26T17:53:49.765218+0000","last_became_active":"2024-04-26T17:53:03.616255+0000","last_became_peered":"2024-04-26T17:53:03.616255+0000","last_unstale":"2024-04-26T17:53:49.765218+0000","last_undegraded":"2024-04-26T17:53:49.765218+0000","last_fullsized":"2024-04-26T17:53:49.765218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:43:31.183679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765248+0000","last_change":"2024-04-26T17:53:03.615402+0000","last_active":"2024-04-26T17:53:49.765248+0000","last_peered":"2024-04-26T17:53:49.765248+0000","last_clean":"2024-04-26T17:53:49.765248+0000","last_became_active":"2024-04-26T17:53:03.615048+0000","last_became_peered":"2024-04-26T17:53:03.615048+0000","last_unstale":"2024-04-26T17:53:49.765248+0000","last_undegraded":"2024-04-26T17:53:49.765248+0000","last_fullsized":"2024-04-26T17:53:49.765248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:05:12.454770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765118+0000","last_change":"2024-04-26T17:53:03.622625+0000","last_active":"2024-04-26T17:53:49.765118+0000","last_peered":"2024-04-26T17:53:49.765118+0000","last_clean":"2024-04-26T17:53:49.765118+0000","last_became_active":"2024-04-26T17:53:03.622305+0000","last_became_peered":"2024-04-26T17:53:03.622305+0000","last_unstale":"2024-04-26T17:53:49.765118+0000","last_undegraded":"2024-04-26T17:53:49.765118+0000","last_fullsized":"2024-04-26T17:53:49.765118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:06.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891229+0000","last_change":"2024-04-26T17:53:03.617650+0000","last_active":"2024-04-26T17:53:49.891229+0000","last_peered":"2024-04-26T17:53:49.891229+0000","last_clean":"2024-04-26T17:53:49.891229+0000","last_became_active":"2024-04-26T17:53:03.617220+0000","last_became_peered":"2024-04-26T17:53:03.617220+0000","last_unstale":"2024-04-26T17:53:49.891229+0000","last_undegraded":"2024-04-26T17:53:49.891229+0000","last_fullsized":"2024-04-26T17:53:49.891229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:31.765577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767329+0000","last_change":"2024-04-26T17:53:03.615429+0000","last_active":"2024-04-26T17:53:49.767329+0000","last_peered":"2024-04-26T17:53:49.767329+0000","last_clean":"2024-04-26T17:53:49.767329+0000","last_became_active":"2024-04-26T17:53:03.615079+0000","last_became_peered":"2024-04-26T17:53:03.615079+0000","last_unstale":"2024-04-26T17:53:49.767329+0000","last_undegraded":"2024-04-26T17:53:49.767329+0000","last_fullsized":"2024-04-26T17:53:49.767329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:20.994016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764669+0000","last_change":"2024-04-26T17:53:03.624619+0000","last_active":"2024-04-26T17:53:49.764669+0000","last_peered":"2024-04-26T17:53:49.764669+0000","last_clean":"2024-04-26T17:53:49.764669+0000","last_became_active":"2024-04-26T17:53:03.624251+0000","last_became_peered":"2024-04-26T17:53:03.624251+0000","last_unstale":"2024-04-26T17:53:49.764669+0000","last_undegraded":"2024-04-26T17:53:49.764669+0000","last_fullsized":"2024-04-26T17:53:49.764669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:50:17.535235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767186+0000","last_change":"2024-04-26T17:53:03.619325+0000","last_active":"2024-04-26T17:53:49.767186+0000","last_peered":"2024-04-26T17:53:49.767186+0000","last_clean":"2024-04-26T17:53:49.767186+0000","last_became_active":"2024-04-26T17:53:03.618989+0000","last_became_peered":"2024-04-26T17:53:03.618989+0000","last_unstale":"2024-04-26T17:53:49.767186+0000","last_undegraded":"2024-04-26T17:53:49.767186+0000","last_fullsized":"2024-04-26T17:53:49.767186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:10.195742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764403+0000","last_change":"2024-04-26T17:53:03.609346+0000","last_active":"2024-04-26T17:53:49.764403+0000","last_peered":"2024-04-26T17:53:49.764403+0000","last_clean":"2024-04-26T17:53:49.764403+0000","last_became_active":"2024-04-26T17:53:03.609021+0000","last_became_peered":"2024-04-26T17:53:03.609021+0000","last_unstale":"2024-04-26T17:53:49.764403+0000","last_undegraded":"2024-04-26T17:53:49.764403+0000","last_fullsized":"2024-04-26T17:53:49.764403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:44.328599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759099+0000","last_change":"2024-04-26T17:53:03.618573+0000","last_active":"2024-04-26T17:53:49.759099+0000","last_peered":"2024-04-26T17:53:49.759099+0000","last_clean":"2024-04-26T17:53:49.759099+0000","last_became_active":"2024-04-26T17:53:03.618223+0000","last_became_peered":"2024-04-26T17:53:03.618223+0000","last_unstale":"2024-04-26T17:53:49.759099+0000","last_undegraded":"2024-04-26T17:53:49.759099+0000","last_fullsized":"2024-04-26T17:53:49.759099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:16.749205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765013+0000","last_change":"2024-04-26T17:53:03.621530+0000","last_active":"2024-04-26T17:53:49.765013+0000","last_peered":"2024-04-26T17:53:49.765013+0000","last_clean":"2024-04-26T17:53:49.765013+0000","last_became_active":"2024-04-26T17:53:03.621174+0000","last_became_peered":"2024-04-26T17:53:03.621174+0000","last_unstale":"2024-04-26T17:53:49.765013+0000","last_undegraded":"2024-04-26T17:53:49.765013+0000","last_fullsized":"2024-04-26T17:53:49.765013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:31.388232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767569+0000","last_change":"2024-04-26T17:53:03.617682+0000","last_active":"2024-04-26T17:53:49.767569+0000","last_peered":"2024-04-26T17:53:49.767569+0000","last_clean":"2024-04-26T17:53:49.767569+0000","last_became_active":"2024-04-26T17:53:03.617348+0000","last_became_peered":"2024-04-26T17:53:03.617348+0000","last_unstale":"2024-04-26T17:53:49.767569+0000","last_undegraded":"2024-04-26T17:53:49.767569+0000","last_fullsized":"2024-04-26T17:53:49.767569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:21.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891137+0000","last_change":"2024-04-26T17:53:03.617955+0000","last_active":"2024-04-26T17:53:49.891137+0000","last_peered":"2024-04-26T17:53:49.891137+0000","last_clean":"2024-04-26T17:53:49.891137+0000","last_became_active":"2024-04-26T17:53:03.616765+0000","last_became_peered":"2024-04-26T17:53:03.616765+0000","last_unstale":"2024-04-26T17:53:49.891137+0000","last_undegraded":"2024-04-26T17:53:49.891137+0000","last_fullsized":"2024-04-26T17:53:49.891137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:59.292277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767807+0000","last_change":"2024-04-26T17:53:49.767807+0000","last_active":"2024-04-26T17:53:49.767807+0000","last_peered":"2024-04-26T17:53:49.767807+0000","last_clean":"2024-04-26T17:53:49.767807+0000","last_became_active":"2024-04-26T17:53:49.767404+0000","last_became_peered":"2024-04-26T17:53:49.767404+0000","last_unstale":"2024-04-26T17:53:49.767807+0000","last_undegraded":"2024-04-26T17:53:49.767807+0000","last_fullsized":"2024-04-26T17:53:49.767807+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:20:45.482580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764992+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:53:49.764992+0000","last_peered":"2024-04-26T17:53:49.764992+0000","last_clean":"2024-04-26T17:53:49.764992+0000","last_became_active":"2024-04-26T17:53:03.616623+0000","last_became_peered":"2024-04-26T17:53:03.616623+0000","last_unstale":"2024-04-26T17:53:49.764992+0000","last_undegraded":"2024-04-26T17:53:49.764992+0000","last_fullsized":"2024-04-26T17:53:49.764992+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:04.231549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764782+0000","last_change":"2024-04-26T17:53:03.622147+0000","last_active":"2024-04-26T17:53:49.764782+0000","last_peered":"2024-04-26T17:53:49.764782+0000","last_clean":"2024-04-26T17:53:49.764782+0000","last_became_active":"2024-04-26T17:53:03.621542+0000","last_became_peered":"2024-04-26T17:53:03.621542+0000","last_unstale":"2024-04-26T17:53:49.764782+0000","last_undegraded":"2024-04-26T17:53:49.764782+0000","last_fullsized":"2024-04-26T17:53:49.764782+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:01.746954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758591+0000","last_change":"2024-04-26T17:53:03.624433+0000","last_active":"2024-04-26T17:53:49.758591+0000","last_peered":"2024-04-26T17:53:49.758591+0000","last_clean":"2024-04-26T17:53:49.758591+0000","last_became_active":"2024-04-26T17:53:03.624135+0000","last_became_peered":"2024-04-26T17:53:03.624135+0000","last_unstale":"2024-04-26T17:53:49.758591+0000","last_undegraded":"2024-04-26T17:53:49.758591+0000","last_fullsized":"2024-04-26T17:53:49.758591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:32:20.521917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765321+0000","last_change":"2024-04-26T17:53:03.623607+0000","last_active":"2024-04-26T17:53:49.765321+0000","last_peered":"2024-04-26T17:53:49.765321+0000","last_clean":"2024-04-26T17:53:49.765321+0000","last_became_active":"2024-04-26T17:53:03.621783+0000","last_became_peered":"2024-04-26T17:53:03.621783+0000","last_unstale":"2024-04-26T17:53:49.765321+0000","last_undegraded":"2024-04-26T17:53:49.765321+0000","last_fullsized":"2024-04-26T17:53:49.765321+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:38:59.872868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.756636+0000","last_change":"2024-04-26T17:53:03.614974+0000","last_active":"2024-04-26T17:53:48.756636+0000","last_peered":"2024-04-26T17:53:48.756636+0000","last_clean":"2024-04-26T17:53:48.756636+0000","last_became_active":"2024-04-26T17:53:03.614619+0000","last_became_peered":"2024-04-26T17:53:03.614619+0000","last_unstale":"2024-04-26T17:53:48.756636+0000","last_undegraded":"2024-04-26T17:53:48.756636+0000","last_fullsized":"2024-04-26T17:53:48.756636+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:50.729531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769611+0000","last_change":"2024-04-26T17:53:03.611140+0000","last_active":"2024-04-26T17:53:49.769611+0000","last_peered":"2024-04-26T17:53:49.769611+0000","last_clean":"2024-04-26T17:53:49.769611+0000","last_became_active":"2024-04-26T17:53:03.610549+0000","last_became_peered":"2024-04-26T17:53:03.610549+0000","last_unstale":"2024-04-26T17:53:49.769611+0000","last_undegraded":"2024-04-26T17:53:49.769611+0000","last_fullsized":"2024-04-26T17:53:49.769611+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:46:17.178598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.756562+0000","last_change":"2024-04-26T17:53:00.585971+0000","last_active":"2024-04-26T17:53:48.756562+0000","last_peered":"2024-04-26T17:53:48.756562+0000","last_clean":"2024-04-26T17:53:48.756562+0000","last_became_active":"2024-04-26T17:53:00.585648+0000","last_became_peered":"2024-04-26T17:53:00.585648+0000","last_unstale":"2024-04-26T17:53:48.756562+0000","last_undegraded":"2024-04-26T17:53:48.756562+0000","last_fullsized":"2024-04-26T17:53:48.756562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:00:24.082773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":2,"reported_epoch":89,"state":"peering","last_fresh":"2024-04-26T17:53:03.598208+0000","last_change":"2024-04-26T17:53:49.721525+0000","last_active":"2024-04-26T17:53:02.586248+0000","last_peered":"2024-04-26T17:53:02.586248+0000","last_clean":"2024-04-26T17:53:02.586248+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T17:53:03.598208+0000","last_undegraded":"2024-04-26T17:53:03.598208+0000","last_fullsized":"2024-04-26T17:53:03.598208+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.766024+0000","last_change":"2024-04-26T17:53:01.145438+0000","last_active":"2024-04-26T17:53:49.766024+0000","last_peered":"2024-04-26T17:53:49.766024+0000","last_clean":"2024-04-26T17:53:49.766024+0000","last_became_active":"2024-04-26T17:53:01.145030+0000","last_became_peered":"2024-04-26T17:53:01.145030+0000","last_unstale":"2024-04-26T17:53:49.766024+0000","last_undegraded":"2024-04-26T17:53:49.766024+0000","last_fullsized":"2024-04-26T17:53:49.766024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:15.650247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765536+0000","last_change":"2024-04-26T17:53:03.622647+0000","last_active":"2024-04-26T17:53:49.765536+0000","last_peered":"2024-04-26T17:53:49.765536+0000","last_clean":"2024-04-26T17:53:49.765536+0000","last_became_active":"2024-04-26T17:53:03.622298+0000","last_became_peered":"2024-04-26T17:53:03.622298+0000","last_unstale":"2024-04-26T17:53:49.765536+0000","last_undegraded":"2024-04-26T17:53:49.765536+0000","last_fullsized":"2024-04-26T17:53:49.765536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:37.230539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.775803+0000","last_change":"2024-04-26T17:53:49.775405+0000","last_active":"2024-04-26T17:53:49.775803+0000","last_peered":"2024-04-26T17:53:49.775803+0000","last_clean":"2024-04-26T17:53:49.775803+0000","last_became_active":"2024-04-26T17:53:49.770536+0000","last_became_peered":"2024-04-26T17:53:49.770536+0000","last_unstale":"2024-04-26T17:53:49.775803+0000","last_undegraded":"2024-04-26T17:53:49.775803+0000","last_fullsized":"2024-04-26T17:53:49.775803+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:34.643452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758674+0000","last_change":"2024-04-26T17:53:03.617462+0000","last_active":"2024-04-26T17:53:49.758674+0000","last_peered":"2024-04-26T17:53:49.758674+0000","last_clean":"2024-04-26T17:53:49.758674+0000","last_became_active":"2024-04-26T17:53:03.617115+0000","last_became_peered":"2024-04-26T17:53:03.617115+0000","last_unstale":"2024-04-26T17:53:49.758674+0000","last_undegraded":"2024-04-26T17:53:49.758674+0000","last_fullsized":"2024-04-26T17:53:49.758674+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:11.406146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767810+0000","last_change":"2024-04-26T17:53:00.583378+0000","last_active":"2024-04-26T17:53:49.767810+0000","last_peered":"2024-04-26T17:53:49.767810+0000","last_clean":"2024-04-26T17:53:49.767810+0000","last_became_active":"2024-04-26T17:53:00.583020+0000","last_became_peered":"2024-04-26T17:53:00.583020+0000","last_unstale":"2024-04-26T17:53:49.767810+0000","last_undegraded":"2024-04-26T17:53:49.767810+0000","last_fullsized":"2024-04-26T17:53:49.767810+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:24.089996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759512+0000","last_change":"2024-04-26T17:53:03.609557+0000","last_active":"2024-04-26T17:53:49.759512+0000","last_peered":"2024-04-26T17:53:49.759512+0000","last_clean":"2024-04-26T17:53:49.759512+0000","last_became_active":"2024-04-26T17:53:03.609028+0000","last_became_peered":"2024-04-26T17:53:03.609028+0000","last_unstale":"2024-04-26T17:53:49.759512+0000","last_undegraded":"2024-04-26T17:53:49.759512+0000","last_fullsized":"2024-04-26T17:53:49.759512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:39:22.860877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759190+0000","last_change":"2024-04-26T17:53:00.586683+0000","last_active":"2024-04-26T17:53:49.759190+0000","last_peered":"2024-04-26T17:53:49.759190+0000","last_clean":"2024-04-26T17:53:49.759190+0000","last_became_active":"2024-04-26T17:53:00.586272+0000","last_became_peered":"2024-04-26T17:53:00.586272+0000","last_unstale":"2024-04-26T17:53:49.759190+0000","last_undegraded":"2024-04-26T17:53:49.759190+0000","last_fullsized":"2024-04-26T17:53:49.759190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:20:29.068607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767148+0000","last_change":"2024-04-26T17:53:03.611073+0000","last_active":"2024-04-26T17:53:49.767148+0000","last_peered":"2024-04-26T17:53:49.767148+0000","last_clean":"2024-04-26T17:53:49.767148+0000","last_became_active":"2024-04-26T17:53:03.610701+0000","last_became_peered":"2024-04-26T17:53:03.610701+0000","last_unstale":"2024-04-26T17:53:49.767148+0000","last_undegraded":"2024-04-26T17:53:49.767148+0000","last_fullsized":"2024-04-26T17:53:49.767148+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:01:00.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764638+0000","last_change":"2024-04-26T17:53:00.586082+0000","last_active":"2024-04-26T17:53:49.764638+0000","last_peered":"2024-04-26T17:53:49.764638+0000","last_clean":"2024-04-26T17:53:49.764638+0000","last_became_active":"2024-04-26T17:53:00.585678+0000","last_became_peered":"2024-04-26T17:53:00.585678+0000","last_unstale":"2024-04-26T17:53:49.764638+0000","last_undegraded":"2024-04-26T17:53:49.764638+0000","last_fullsized":"2024-04-26T17:53:49.764638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:55:25.523846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759383+0000","last_change":"2024-04-26T17:53:03.600230+0000","last_active":"2024-04-26T17:53:49.759383+0000","last_peered":"2024-04-26T17:53:49.759383+0000","last_clean":"2024-04-26T17:53:49.759383+0000","last_became_active":"2024-04-26T17:53:03.599713+0000","last_became_peered":"2024-04-26T17:53:03.599713+0000","last_unstale":"2024-04-26T17:53:49.759383+0000","last_undegraded":"2024-04-26T17:53:49.759383+0000","last_fullsized":"2024-04-26T17:53:49.759383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:45.822561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760523+0000","last_change":"2024-04-26T17:53:00.584588+0000","last_active":"2024-04-26T17:53:49.760523+0000","last_peered":"2024-04-26T17:53:49.760523+0000","last_clean":"2024-04-26T17:53:49.760523+0000","last_became_active":"2024-04-26T17:53:00.584284+0000","last_became_peered":"2024-04-26T17:53:00.584284+0000","last_unstale":"2024-04-26T17:53:49.760523+0000","last_undegraded":"2024-04-26T17:53:49.760523+0000","last_fullsized":"2024-04-26T17:53:49.760523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:08.428904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764864+0000","last_change":"2024-04-26T17:53:03.615237+0000","last_active":"2024-04-26T17:53:49.764864+0000","last_peered":"2024-04-26T17:53:49.764864+0000","last_clean":"2024-04-26T17:53:49.764864+0000","last_became_active":"2024-04-26T17:53:03.614704+0000","last_became_peered":"2024-04-26T17:53:03.614704+0000","last_unstale":"2024-04-26T17:53:49.764864+0000","last_undegraded":"2024-04-26T17:53:49.764864+0000","last_fullsized":"2024-04-26T17:53:49.764864+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:27.251839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758907+0000","last_change":"2024-04-26T17:53:00.589287+0000","last_active":"2024-04-26T17:53:49.758907+0000","last_peered":"2024-04-26T17:53:49.758907+0000","last_clean":"2024-04-26T17:53:49.758907+0000","last_became_active":"2024-04-26T17:53:00.588119+0000","last_became_peered":"2024-04-26T17:53:00.588119+0000","last_unstale":"2024-04-26T17:53:49.758907+0000","last_undegraded":"2024-04-26T17:53:49.758907+0000","last_fullsized":"2024-04-26T17:53:49.758907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:25.389777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.756432+0000","last_change":"2024-04-26T17:53:03.608061+0000","last_active":"2024-04-26T17:53:48.756432+0000","last_peered":"2024-04-26T17:53:48.756432+0000","last_clean":"2024-04-26T17:53:48.756432+0000","last_became_active":"2024-04-26T17:53:03.607624+0000","last_became_peered":"2024-04-26T17:53:03.607624+0000","last_unstale":"2024-04-26T17:53:48.756432+0000","last_undegraded":"2024-04-26T17:53:48.756432+0000","last_fullsized":"2024-04-26T17:53:48.756432+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:15.111011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764914+0000","last_change":"2024-04-26T17:53:00.588621+0000","last_active":"2024-04-26T17:53:49.764914+0000","last_peered":"2024-04-26T17:53:49.764914+0000","last_clean":"2024-04-26T17:53:49.764914+0000","last_became_active":"2024-04-26T17:53:00.587330+0000","last_became_peered":"2024-04-26T17:53:00.587330+0000","last_unstale":"2024-04-26T17:53:49.764914+0000","last_undegraded":"2024-04-26T17:53:49.764914+0000","last_fullsized":"2024-04-26T17:53:49.764914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:01:12.178610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759017+0000","last_change":"2024-04-26T17:53:03.615723+0000","last_active":"2024-04-26T17:53:49.759017+0000","last_peered":"2024-04-26T17:53:49.759017+0000","last_clean":"2024-04-26T17:53:49.759017+0000","last_became_active":"2024-04-26T17:53:03.614931+0000","last_became_peered":"2024-04-26T17:53:03.614931+0000","last_unstale":"2024-04-26T17:53:49.759017+0000","last_undegraded":"2024-04-26T17:53:49.759017+0000","last_fullsized":"2024-04-26T17:53:49.759017+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:09.663876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767693+0000","last_change":"2024-04-26T17:53:00.589232+0000","last_active":"2024-04-26T17:53:49.767693+0000","last_peered":"2024-04-26T17:53:49.767693+0000","last_clean":"2024-04-26T17:53:49.767693+0000","last_became_active":"2024-04-26T17:53:00.588863+0000","last_became_peered":"2024-04-26T17:53:00.588863+0000","last_unstale":"2024-04-26T17:53:49.767693+0000","last_undegraded":"2024-04-26T17:53:49.767693+0000","last_fullsized":"2024-04-26T17:53:49.767693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:14.360782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765501+0000","last_change":"2024-04-26T17:53:03.624923+0000","last_active":"2024-04-26T17:53:49.765501+0000","last_peered":"2024-04-26T17:53:49.765501+0000","last_clean":"2024-04-26T17:53:49.765501+0000","last_became_active":"2024-04-26T17:53:03.624620+0000","last_became_peered":"2024-04-26T17:53:03.624620+0000","last_unstale":"2024-04-26T17:53:49.765501+0000","last_undegraded":"2024-04-26T17:53:49.765501+0000","last_fullsized":"2024-04-26T17:53:49.765501+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:50:51.259563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759526+0000","last_change":"2024-04-26T17:53:01.146950+0000","last_active":"2024-04-26T17:53:49.759526+0000","last_peered":"2024-04-26T17:53:49.759526+0000","last_clean":"2024-04-26T17:53:49.759526+0000","last_became_active":"2024-04-26T17:53:01.146591+0000","last_became_peered":"2024-04-26T17:53:01.146591+0000","last_unstale":"2024-04-26T17:53:49.759526+0000","last_undegraded":"2024-04-26T17:53:49.759526+0000","last_fullsized":"2024-04-26T17:53:49.759526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:17.756660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758787+0000","last_change":"2024-04-26T17:53:03.626944+0000","last_active":"2024-04-26T17:53:49.758787+0000","last_peered":"2024-04-26T17:53:49.758787+0000","last_clean":"2024-04-26T17:53:49.758787+0000","last_became_active":"2024-04-26T17:53:03.626594+0000","last_became_peered":"2024-04-26T17:53:03.626594+0000","last_unstale":"2024-04-26T17:53:49.758787+0000","last_undegraded":"2024-04-26T17:53:49.758787+0000","last_fullsized":"2024-04-26T17:53:49.758787+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:52:42.997251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'30","reported_seq":122,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:50.739715+0000","last_change":"2024-04-26T17:53:00.588353+0000","last_active":"2024-04-26T17:53:50.739715+0000","last_peered":"2024-04-26T17:53:50.739715+0000","last_clean":"2024-04-26T17:53:50.739715+0000","last_became_active":"2024-04-26T17:53:00.586351+0000","last_became_peered":"2024-04-26T17:53:00.586351+0000","last_unstale":"2024-04-26T17:53:50.739715+0000","last_undegraded":"2024-04-26T17:53:50.739715+0000","last_fullsized":"2024-04-26T17:53:50.739715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:41.660525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62335,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":66,"num_read_kb":89,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768625+0000","last_change":"2024-04-26T17:53:03.612052+0000","last_active":"2024-04-26T17:53:49.768625+0000","last_peered":"2024-04-26T17:53:49.768625+0000","last_clean":"2024-04-26T17:53:49.768625+0000","last_became_active":"2024-04-26T17:53:03.611733+0000","last_became_peered":"2024-04-26T17:53:03.611733+0000","last_unstale":"2024-04-26T17:53:49.768625+0000","last_undegraded":"2024-04-26T17:53:49.768625+0000","last_fullsized":"2024-04-26T17:53:49.768625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:05:23.958317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769171+0000","last_change":"2024-04-26T17:53:00.588284+0000","last_active":"2024-04-26T17:53:49.769171+0000","last_peered":"2024-04-26T17:53:49.769171+0000","last_clean":"2024-04-26T17:53:49.769171+0000","last_became_active":"2024-04-26T17:53:00.587942+0000","last_became_peered":"2024-04-26T17:53:00.587942+0000","last_unstale":"2024-04-26T17:53:49.769171+0000","last_undegraded":"2024-04-26T17:53:49.769171+0000","last_fullsized":"2024-04-26T17:53:49.769171+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:43.782971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760366+0000","last_change":"2024-04-26T17:53:03.613092+0000","last_active":"2024-04-26T17:53:49.760366+0000","last_peered":"2024-04-26T17:53:49.760366+0000","last_clean":"2024-04-26T17:53:49.760366+0000","last_became_active":"2024-04-26T17:53:03.612773+0000","last_became_peered":"2024-04-26T17:53:03.612773+0000","last_unstale":"2024-04-26T17:53:49.760366+0000","last_undegraded":"2024-04-26T17:53:49.760366+0000","last_fullsized":"2024-04-26T17:53:49.760366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:41:21.340375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759054+0000","last_change":"2024-04-26T17:53:00.588777+0000","last_active":"2024-04-26T17:53:49.759054+0000","last_peered":"2024-04-26T17:53:49.759054+0000","last_clean":"2024-04-26T17:53:49.759054+0000","last_became_active":"2024-04-26T17:53:00.588380+0000","last_became_peered":"2024-04-26T17:53:00.588380+0000","last_unstale":"2024-04-26T17:53:49.759054+0000","last_undegraded":"2024-04-26T17:53:49.759054+0000","last_fullsized":"2024-04-26T17:53:49.759054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:44.587381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767750+0000","last_change":"2024-04-26T17:53:03.620344+0000","last_active":"2024-04-26T17:53:49.767750+0000","last_peered":"2024-04-26T17:53:49.767750+0000","last_clean":"2024-04-26T17:53:49.767750+0000","last_became_active":"2024-04-26T17:53:03.619944+0000","last_became_peered":"2024-04-26T17:53:03.619944+0000","last_unstale":"2024-04-26T17:53:49.767750+0000","last_undegraded":"2024-04-26T17:53:49.767750+0000","last_fullsized":"2024-04-26T17:53:49.767750+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:08:31.801342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891069+0000","last_change":"2024-04-26T17:53:00.587733+0000","last_active":"2024-04-26T17:53:49.891069+0000","last_peered":"2024-04-26T17:53:49.891069+0000","last_clean":"2024-04-26T17:53:49.891069+0000","last_became_active":"2024-04-26T17:53:00.587373+0000","last_became_peered":"2024-04-26T17:53:00.587373+0000","last_unstale":"2024-04-26T17:53:49.891069+0000","last_undegraded":"2024-04-26T17:53:49.891069+0000","last_fullsized":"2024-04-26T17:53:49.891069+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:10.978944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767121+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:53:49.767121+0000","last_peered":"2024-04-26T17:53:49.767121+0000","last_clean":"2024-04-26T17:53:49.767121+0000","last_became_active":"2024-04-26T17:53:03.616588+0000","last_became_peered":"2024-04-26T17:53:03.616588+0000","last_unstale":"2024-04-26T17:53:49.767121+0000","last_undegraded":"2024-04-26T17:53:49.767121+0000","last_fullsized":"2024-04-26T17:53:49.767121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:46:38.419613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767233+0000","last_change":"2024-04-26T17:53:01.146922+0000","last_active":"2024-04-26T17:53:49.767233+0000","last_peered":"2024-04-26T17:53:49.767233+0000","last_clean":"2024-04-26T17:53:49.767233+0000","last_became_active":"2024-04-26T17:53:01.146520+0000","last_became_peered":"2024-04-26T17:53:01.146520+0000","last_unstale":"2024-04-26T17:53:49.767233+0000","last_undegraded":"2024-04-26T17:53:49.767233+0000","last_fullsized":"2024-04-26T17:53:49.767233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:12:49.238012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758964+0000","last_change":"2024-04-26T17:53:03.608513+0000","last_active":"2024-04-26T17:53:49.758964+0000","last_peered":"2024-04-26T17:53:49.758964+0000","last_clean":"2024-04-26T17:53:49.758964+0000","last_became_active":"2024-04-26T17:53:03.608167+0000","last_became_peered":"2024-04-26T17:53:03.608167+0000","last_unstale":"2024-04-26T17:53:49.758964+0000","last_undegraded":"2024-04-26T17:53:49.758964+0000","last_fullsized":"2024-04-26T17:53:49.758964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:03.002756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769191+0000","last_change":"2024-04-26T17:53:00.583878+0000","last_active":"2024-04-26T17:53:49.769191+0000","last_peered":"2024-04-26T17:53:49.769191+0000","last_clean":"2024-04-26T17:53:49.769191+0000","last_became_active":"2024-04-26T17:53:00.583159+0000","last_became_peered":"2024-04-26T17:53:00.583159+0000","last_unstale":"2024-04-26T17:53:49.769191+0000","last_undegraded":"2024-04-26T17:53:49.769191+0000","last_fullsized":"2024-04-26T17:53:49.769191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:49.850942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765106+0000","last_change":"2024-04-26T17:53:03.614467+0000","last_active":"2024-04-26T17:53:49.765106+0000","last_peered":"2024-04-26T17:53:49.765106+0000","last_clean":"2024-04-26T17:53:49.765106+0000","last_became_active":"2024-04-26T17:53:03.613972+0000","last_became_peered":"2024-04-26T17:53:03.613972+0000","last_unstale":"2024-04-26T17:53:49.765106+0000","last_undegraded":"2024-04-26T17:53:49.765106+0000","last_fullsized":"2024-04-26T17:53:49.765106+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:27.259777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"peering","last_fresh":"2024-04-26T17:53:05.613913+0000","last_change":"2024-04-26T17:53:48.756965+0000","last_active":"2024-04-26T17:53:48.756965+0000","last_peered":"2024-04-26T17:53:05.613913+0000","last_clean":"2024-04-26T17:53:05.613913+0000","last_became_active":"2024-04-26T17:53:01.145146+0000","last_became_peered":"2024-04-26T17:53:01.145146+0000","last_unstale":"2024-04-26T17:53:05.613913+0000","last_undegraded":"2024-04-26T17:53:05.613913+0000","last_fullsized":"2024-04-26T17:53:05.613913+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,11],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764767+0000","last_change":"2024-04-26T17:53:03.613888+0000","last_active":"2024-04-26T17:53:49.764767+0000","last_peered":"2024-04-26T17:53:49.764767+0000","last_clean":"2024-04-26T17:53:49.764767+0000","last_became_active":"2024-04-26T17:53:03.613499+0000","last_became_peered":"2024-04-26T17:53:03.613499+0000","last_unstale":"2024-04-26T17:53:49.764767+0000","last_undegraded":"2024-04-26T17:53:49.764767+0000","last_fullsized":"2024-04-26T17:53:49.764767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:15.549419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767375+0000","last_change":"2024-04-26T17:53:00.587803+0000","last_active":"2024-04-26T17:53:49.767375+0000","last_peered":"2024-04-26T17:53:49.767375+0000","last_clean":"2024-04-26T17:53:49.767375+0000","last_became_active":"2024-04-26T17:53:00.587382+0000","last_became_peered":"2024-04-26T17:53:00.587382+0000","last_unstale":"2024-04-26T17:53:49.767375+0000","last_undegraded":"2024-04-26T17:53:49.767375+0000","last_fullsized":"2024-04-26T17:53:49.767375+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:38.249478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769008+0000","last_change":"2024-04-26T17:53:49.769008+0000","last_active":"2024-04-26T17:53:49.769008+0000","last_peered":"2024-04-26T17:53:49.769008+0000","last_clean":"2024-04-26T17:53:49.769008+0000","last_became_active":"2024-04-26T17:53:49.768625+0000","last_became_peered":"2024-04-26T17:53:49.768625+0000","last_unstale":"2024-04-26T17:53:49.769008+0000","last_undegraded":"2024-04-26T17:53:49.769008+0000","last_fullsized":"2024-04-26T17:53:49.769008+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:27:46.412625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759215+0000","last_change":"2024-04-26T17:53:03.612899+0000","last_active":"2024-04-26T17:53:49.759215+0000","last_peered":"2024-04-26T17:53:49.759215+0000","last_clean":"2024-04-26T17:53:49.759215+0000","last_became_active":"2024-04-26T17:53:03.612564+0000","last_became_peered":"2024-04-26T17:53:03.612564+0000","last_unstale":"2024-04-26T17:53:49.759215+0000","last_undegraded":"2024-04-26T17:53:49.759215+0000","last_fullsized":"2024-04-26T17:53:49.759215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:07.368256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:51.739582+0000","last_change":"2024-04-26T17:53:01.145471+0000","last_active":"2024-04-26T17:53:51.739582+0000","last_peered":"2024-04-26T17:53:51.739582+0000","last_clean":"2024-04-26T17:53:51.739582+0000","last_became_active":"2024-04-26T17:53:01.145040+0000","last_became_peered":"2024-04-26T17:53:01.145040+0000","last_unstale":"2024-04-26T17:53:51.739582+0000","last_undegraded":"2024-04-26T17:53:51.739582+0000","last_fullsized":"2024-04-26T17:53:51.739582+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:46:27.437329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":31,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765218+0000","last_change":"2024-04-26T17:53:03.618600+0000","last_active":"2024-04-26T17:53:49.765218+0000","last_peered":"2024-04-26T17:53:49.765218+0000","last_clean":"2024-04-26T17:53:49.765218+0000","last_became_active":"2024-04-26T17:53:03.618139+0000","last_became_peered":"2024-04-26T17:53:03.618139+0000","last_unstale":"2024-04-26T17:53:49.765218+0000","last_undegraded":"2024-04-26T17:53:49.765218+0000","last_fullsized":"2024-04-26T17:53:49.765218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:56.007659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891071+0000","last_change":"2024-04-26T17:53:00.586386+0000","last_active":"2024-04-26T17:53:49.891071+0000","last_peered":"2024-04-26T17:53:49.891071+0000","last_clean":"2024-04-26T17:53:49.891071+0000","last_became_active":"2024-04-26T17:53:00.585953+0000","last_became_peered":"2024-04-26T17:53:00.585953+0000","last_unstale":"2024-04-26T17:53:49.891071+0000","last_undegraded":"2024-04-26T17:53:49.891071+0000","last_fullsized":"2024-04-26T17:53:49.891071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:39:57.160983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":357,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767252+0000","last_change":"2024-04-26T17:51:38.839853+0000","last_active":"2024-04-26T17:53:49.767252+0000","last_peered":"2024-04-26T17:53:49.767252+0000","last_clean":"2024-04-26T17:53:49.767252+0000","last_became_active":"2024-04-26T17:51:38.839318+0000","last_became_peered":"2024-04-26T17:51:38.839318+0000","last_unstale":"2024-04-26T17:53:49.767252+0000","last_undegraded":"2024-04-26T17:53:49.767252+0000","last_fullsized":"2024-04-26T17:53:49.767252+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_clean_scrub_stamp":"2024-04-26T17:48:12.577543+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:08:50.683663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760286+0000","last_change":"2024-04-26T17:53:03.614170+0000","last_active":"2024-04-26T17:53:49.760286+0000","last_peered":"2024-04-26T17:53:49.760286+0000","last_clean":"2024-04-26T17:53:49.760286+0000","last_became_active":"2024-04-26T17:53:03.613836+0000","last_became_peered":"2024-04-26T17:53:03.613836+0000","last_unstale":"2024-04-26T17:53:49.760286+0000","last_undegraded":"2024-04-26T17:53:49.760286+0000","last_fullsized":"2024-04-26T17:53:49.760286+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:48.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767374+0000","last_change":"2024-04-26T17:53:00.587319+0000","last_active":"2024-04-26T17:53:49.767374+0000","last_peered":"2024-04-26T17:53:49.767374+0000","last_clean":"2024-04-26T17:53:49.767374+0000","last_became_active":"2024-04-26T17:53:00.586983+0000","last_became_peered":"2024-04-26T17:53:00.586983+0000","last_unstale":"2024-04-26T17:53:49.767374+0000","last_undegraded":"2024-04-26T17:53:49.767374+0000","last_fullsized":"2024-04-26T17:53:49.767374+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:03:43.470013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768739+0000","last_change":"2024-04-26T17:53:01.145672+0000","last_active":"2024-04-26T17:53:49.768739+0000","last_peered":"2024-04-26T17:53:49.768739+0000","last_clean":"2024-04-26T17:53:49.768739+0000","last_became_active":"2024-04-26T17:53:01.145274+0000","last_became_peered":"2024-04-26T17:53:01.145274+0000","last_unstale":"2024-04-26T17:53:49.768739+0000","last_undegraded":"2024-04-26T17:53:49.768739+0000","last_fullsized":"2024-04-26T17:53:49.768739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:21.122352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760603+0000","last_change":"2024-04-26T17:53:03.611232+0000","last_active":"2024-04-26T17:53:49.760603+0000","last_peered":"2024-04-26T17:53:49.760603+0000","last_clean":"2024-04-26T17:53:49.760603+0000","last_became_active":"2024-04-26T17:53:03.610905+0000","last_became_peered":"2024-04-26T17:53:03.610905+0000","last_unstale":"2024-04-26T17:53:49.760603+0000","last_undegraded":"2024-04-26T17:53:49.760603+0000","last_fullsized":"2024-04-26T17:53:49.760603+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:31.426237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'11","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T17:53:48.756696+0000","last_change":"2024-04-26T17:53:00.580220+0000","last_active":"2024-04-26T17:53:48.756696+0000","last_peered":"2024-04-26T17:53:48.756696+0000","last_clean":"2024-04-26T17:53:48.756696+0000","last_became_active":"2024-04-26T17:53:00.579818+0000","last_became_peered":"2024-04-26T17:53:00.579818+0000","last_unstale":"2024-04-26T17:53:48.756696+0000","last_undegraded":"2024-04-26T17:53:48.756696+0000","last_fullsized":"2024-04-26T17:53:48.756696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:15.260489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765821+0000","last_change":"2024-04-26T17:53:03.610882+0000","last_active":"2024-04-26T17:53:49.765821+0000","last_peered":"2024-04-26T17:53:49.765821+0000","last_clean":"2024-04-26T17:53:49.765821+0000","last_became_active":"2024-04-26T17:53:03.609979+0000","last_became_peered":"2024-04-26T17:53:03.609979+0000","last_unstale":"2024-04-26T17:53:49.765821+0000","last_undegraded":"2024-04-26T17:53:49.765821+0000","last_fullsized":"2024-04-26T17:53:49.765821+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:33.167579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891536+0000","last_change":"2024-04-26T17:53:01.358555+0000","last_active":"2024-04-26T17:53:49.891536+0000","last_peered":"2024-04-26T17:53:49.891536+0000","last_clean":"2024-04-26T17:53:49.891536+0000","last_became_active":"2024-04-26T17:53:01.358138+0000","last_became_peered":"2024-04-26T17:53:01.358138+0000","last_unstale":"2024-04-26T17:53:49.891536+0000","last_undegraded":"2024-04-26T17:53:49.891536+0000","last_fullsized":"2024-04-26T17:53:49.891536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:14.886609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.766115+0000","last_change":"2024-04-26T17:53:03.618816+0000","last_active":"2024-04-26T17:53:49.766115+0000","last_peered":"2024-04-26T17:53:49.766115+0000","last_clean":"2024-04-26T17:53:49.766115+0000","last_became_active":"2024-04-26T17:53:03.618184+0000","last_became_peered":"2024-04-26T17:53:03.618184+0000","last_unstale":"2024-04-26T17:53:49.766115+0000","last_undegraded":"2024-04-26T17:53:49.766115+0000","last_fullsized":"2024-04-26T17:53:49.766115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:07.514516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758599+0000","last_change":"2024-04-26T17:53:01.147081+0000","last_active":"2024-04-26T17:53:49.758599+0000","last_peered":"2024-04-26T17:53:49.758599+0000","last_clean":"2024-04-26T17:53:49.758599+0000","last_became_active":"2024-04-26T17:53:01.146698+0000","last_became_peered":"2024-04-26T17:53:01.146698+0000","last_unstale":"2024-04-26T17:53:49.758599+0000","last_undegraded":"2024-04-26T17:53:49.758599+0000","last_fullsized":"2024-04-26T17:53:49.758599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:44:19.927316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767053+0000","last_change":"2024-04-26T17:53:03.612592+0000","last_active":"2024-04-26T17:53:49.767053+0000","last_peered":"2024-04-26T17:53:49.767053+0000","last_clean":"2024-04-26T17:53:49.767053+0000","last_became_active":"2024-04-26T17:53:03.612274+0000","last_became_peered":"2024-04-26T17:53:03.612274+0000","last_unstale":"2024-04-26T17:53:49.767053+0000","last_undegraded":"2024-04-26T17:53:49.767053+0000","last_fullsized":"2024-04-26T17:53:49.767053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:30:48.461947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764834+0000","last_change":"2024-04-26T17:53:00.586430+0000","last_active":"2024-04-26T17:53:49.764834+0000","last_peered":"2024-04-26T17:53:49.764834+0000","last_clean":"2024-04-26T17:53:49.764834+0000","last_became_active":"2024-04-26T17:53:00.586004+0000","last_became_peered":"2024-04-26T17:53:00.586004+0000","last_unstale":"2024-04-26T17:53:49.764834+0000","last_undegraded":"2024-04-26T17:53:49.764834+0000","last_fullsized":"2024-04-26T17:53:49.764834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:47:18.224890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764726+0000","last_change":"2024-04-26T17:53:03.623081+0000","last_active":"2024-04-26T17:53:49.764726+0000","last_peered":"2024-04-26T17:53:49.764726+0000","last_clean":"2024-04-26T17:53:49.764726+0000","last_became_active":"2024-04-26T17:53:03.622761+0000","last_became_peered":"2024-04-26T17:53:03.622761+0000","last_unstale":"2024-04-26T17:53:49.764726+0000","last_undegraded":"2024-04-26T17:53:49.764726+0000","last_fullsized":"2024-04-26T17:53:49.764726+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:39.520075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768849+0000","last_change":"2024-04-26T17:53:00.583013+0000","last_active":"2024-04-26T17:53:49.768849+0000","last_peered":"2024-04-26T17:53:49.768849+0000","last_clean":"2024-04-26T17:53:49.768849+0000","last_became_active":"2024-04-26T17:53:00.582653+0000","last_became_peered":"2024-04-26T17:53:00.582653+0000","last_unstale":"2024-04-26T17:53:49.768849+0000","last_undegraded":"2024-04-26T17:53:49.768849+0000","last_fullsized":"2024-04-26T17:53:49.768849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:09.448621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765042+0000","last_change":"2024-04-26T17:53:03.618232+0000","last_active":"2024-04-26T17:53:49.765042+0000","last_peered":"2024-04-26T17:53:49.765042+0000","last_clean":"2024-04-26T17:53:49.765042+0000","last_became_active":"2024-04-26T17:53:03.617906+0000","last_became_peered":"2024-04-26T17:53:03.617906+0000","last_unstale":"2024-04-26T17:53:49.765042+0000","last_undegraded":"2024-04-26T17:53:49.765042+0000","last_fullsized":"2024-04-26T17:53:49.765042+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:17.357398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768301+0000","last_change":"2024-04-26T17:53:00.586565+0000","last_active":"2024-04-26T17:53:49.768301+0000","last_peered":"2024-04-26T17:53:49.768301+0000","last_clean":"2024-04-26T17:53:49.768301+0000","last_became_active":"2024-04-26T17:53:00.586161+0000","last_became_peered":"2024-04-26T17:53:00.586161+0000","last_unstale":"2024-04-26T17:53:49.768301+0000","last_undegraded":"2024-04-26T17:53:49.768301+0000","last_fullsized":"2024-04-26T17:53:49.768301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:56:25.294055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767400+0000","last_change":"2024-04-26T17:53:03.620340+0000","last_active":"2024-04-26T17:53:49.767400+0000","last_peered":"2024-04-26T17:53:49.767400+0000","last_clean":"2024-04-26T17:53:49.767400+0000","last_became_active":"2024-04-26T17:53:03.619897+0000","last_became_peered":"2024-04-26T17:53:03.619897+0000","last_unstale":"2024-04-26T17:53:49.767400+0000","last_undegraded":"2024-04-26T17:53:49.767400+0000","last_fullsized":"2024-04-26T17:53:49.767400+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:12:40.662070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769496+0000","last_change":"2024-04-26T17:53:01.358131+0000","last_active":"2024-04-26T17:53:49.769496+0000","last_peered":"2024-04-26T17:53:49.769496+0000","last_clean":"2024-04-26T17:53:49.769496+0000","last_became_active":"2024-04-26T17:53:01.357796+0000","last_became_peered":"2024-04-26T17:53:01.357796+0000","last_unstale":"2024-04-26T17:53:49.769496+0000","last_undegraded":"2024-04-26T17:53:49.769496+0000","last_fullsized":"2024-04-26T17:53:49.769496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:47.849873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767708+0000","last_change":"2024-04-26T17:53:03.616133+0000","last_active":"2024-04-26T17:53:49.767708+0000","last_peered":"2024-04-26T17:53:49.767708+0000","last_clean":"2024-04-26T17:53:49.767708+0000","last_became_active":"2024-04-26T17:53:03.615797+0000","last_became_peered":"2024-04-26T17:53:03.615797+0000","last_unstale":"2024-04-26T17:53:49.767708+0000","last_undegraded":"2024-04-26T17:53:49.767708+0000","last_fullsized":"2024-04-26T17:53:49.767708+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:05.698489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759241+0000","last_change":"2024-04-26T17:53:00.587696+0000","last_active":"2024-04-26T17:53:49.759241+0000","last_peered":"2024-04-26T17:53:49.759241+0000","last_clean":"2024-04-26T17:53:49.759241+0000","last_became_active":"2024-04-26T17:53:00.587305+0000","last_became_peered":"2024-04-26T17:53:00.587305+0000","last_unstale":"2024-04-26T17:53:49.759241+0000","last_undegraded":"2024-04-26T17:53:49.759241+0000","last_fullsized":"2024-04-26T17:53:49.759241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:39.185753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767528+0000","last_change":"2024-04-26T17:53:03.613076+0000","last_active":"2024-04-26T17:53:49.767528+0000","last_peered":"2024-04-26T17:53:49.767528+0000","last_clean":"2024-04-26T17:53:49.767528+0000","last_became_active":"2024-04-26T17:53:03.612746+0000","last_became_peered":"2024-04-26T17:53:03.612746+0000","last_unstale":"2024-04-26T17:53:49.767528+0000","last_undegraded":"2024-04-26T17:53:49.767528+0000","last_fullsized":"2024-04-26T17:53:49.767528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:20.920154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760620+0000","last_change":"2024-04-26T17:53:00.582575+0000","last_active":"2024-04-26T17:53:49.760620+0000","last_peered":"2024-04-26T17:53:49.760620+0000","last_clean":"2024-04-26T17:53:49.760620+0000","last_became_active":"2024-04-26T17:53:00.582215+0000","last_became_peered":"2024-04-26T17:53:00.582215+0000","last_unstale":"2024-04-26T17:53:49.760620+0000","last_undegraded":"2024-04-26T17:53:49.760620+0000","last_fullsized":"2024-04-26T17:53:49.760620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:35:12.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767871+0000","last_change":"2024-04-26T17:53:03.616269+0000","last_active":"2024-04-26T17:53:49.767871+0000","last_peered":"2024-04-26T17:53:49.767871+0000","last_clean":"2024-04-26T17:53:49.767871+0000","last_became_active":"2024-04-26T17:53:03.615896+0000","last_became_peered":"2024-04-26T17:53:03.615896+0000","last_unstale":"2024-04-26T17:53:49.767871+0000","last_undegraded":"2024-04-26T17:53:49.767871+0000","last_fullsized":"2024-04-26T17:53:49.767871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:12.111392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764802+0000","last_change":"2024-04-26T17:53:03.621589+0000","last_active":"2024-04-26T17:53:49.764802+0000","last_peered":"2024-04-26T17:53:49.764802+0000","last_clean":"2024-04-26T17:53:49.764802+0000","last_became_active":"2024-04-26T17:53:03.621211+0000","last_became_peered":"2024-04-26T17:53:03.621211+0000","last_unstale":"2024-04-26T17:53:49.764802+0000","last_undegraded":"2024-04-26T17:53:49.764802+0000","last_fullsized":"2024-04-26T17:53:49.764802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.487476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768626+0000","last_change":"2024-04-26T17:53:03.624536+0000","last_active":"2024-04-26T17:53:49.768626+0000","last_peered":"2024-04-26T17:53:49.768626+0000","last_clean":"2024-04-26T17:53:49.768626+0000","last_became_active":"2024-04-26T17:53:03.624234+0000","last_became_peered":"2024-04-26T17:53:03.624234+0000","last_unstale":"2024-04-26T17:53:49.768626+0000","last_undegraded":"2024-04-26T17:53:49.768626+0000","last_fullsized":"2024-04-26T17:53:49.768626+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:31:36.322688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768748+0000","last_change":"2024-04-26T17:53:03.620096+0000","last_active":"2024-04-26T17:53:49.768748+0000","last_peered":"2024-04-26T17:53:49.768748+0000","last_clean":"2024-04-26T17:53:49.768748+0000","last_became_active":"2024-04-26T17:53:03.619729+0000","last_became_peered":"2024-04-26T17:53:03.619729+0000","last_unstale":"2024-04-26T17:53:49.768748+0000","last_undegraded":"2024-04-26T17:53:49.768748+0000","last_fullsized":"2024-04-26T17:53:49.768748+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:17.987728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768829+0000","last_change":"2024-04-26T17:53:49.768829+0000","last_active":"2024-04-26T17:53:49.768829+0000","last_peered":"2024-04-26T17:53:49.768829+0000","last_clean":"2024-04-26T17:53:49.768829+0000","last_became_active":"2024-04-26T17:53:49.768415+0000","last_became_peered":"2024-04-26T17:53:49.768415+0000","last_unstale":"2024-04-26T17:53:49.768829+0000","last_undegraded":"2024-04-26T17:53:49.768829+0000","last_fullsized":"2024-04-26T17:53:49.768829+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:57.364355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760497+0000","last_change":"2024-04-26T17:53:03.619193+0000","last_active":"2024-04-26T17:53:49.760497+0000","last_peered":"2024-04-26T17:53:49.760497+0000","last_clean":"2024-04-26T17:53:49.760497+0000","last_became_active":"2024-04-26T17:53:03.618776+0000","last_became_peered":"2024-04-26T17:53:03.618776+0000","last_unstale":"2024-04-26T17:53:49.760497+0000","last_undegraded":"2024-04-26T17:53:49.760497+0000","last_fullsized":"2024-04-26T17:53:49.760497+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:07:24.811683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765044+0000","last_change":"2024-04-26T17:53:03.621366+0000","last_active":"2024-04-26T17:53:49.765044+0000","last_peered":"2024-04-26T17:53:49.765044+0000","last_clean":"2024-04-26T17:53:49.765044+0000","last_became_active":"2024-04-26T17:53:03.621048+0000","last_became_peered":"2024-04-26T17:53:03.621048+0000","last_unstale":"2024-04-26T17:53:49.765044+0000","last_undegraded":"2024-04-26T17:53:49.765044+0000","last_fullsized":"2024-04-26T17:53:49.765044+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:31:06.484880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769806+0000","last_change":"2024-04-26T17:53:03.619383+0000","last_active":"2024-04-26T17:53:49.769806+0000","last_peered":"2024-04-26T17:53:49.769806+0000","last_clean":"2024-04-26T17:53:49.769806+0000","last_became_active":"2024-04-26T17:53:03.619017+0000","last_became_peered":"2024-04-26T17:53:03.619017+0000","last_unstale":"2024-04-26T17:53:49.769806+0000","last_undegraded":"2024-04-26T17:53:49.769806+0000","last_fullsized":"2024-04-26T17:53:49.769806+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:20.091959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.770695+0000","last_change":"2024-04-26T17:53:49.770695+0000","last_active":"2024-04-26T17:53:49.770695+0000","last_peered":"2024-04-26T17:53:49.770695+0000","last_clean":"2024-04-26T17:53:49.770695+0000","last_became_active":"2024-04-26T17:53:49.770303+0000","last_became_peered":"2024-04-26T17:53:49.770303+0000","last_unstale":"2024-04-26T17:53:49.770695+0000","last_undegraded":"2024-04-26T17:53:49.770695+0000","last_fullsized":"2024-04-26T17:53:49.770695+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:22.830129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758969+0000","last_change":"2024-04-26T17:53:03.605521+0000","last_active":"2024-04-26T17:53:49.758969+0000","last_peered":"2024-04-26T17:53:49.758969+0000","last_clean":"2024-04-26T17:53:49.758969+0000","last_became_active":"2024-04-26T17:53:03.605145+0000","last_became_peered":"2024-04-26T17:53:03.605145+0000","last_unstale":"2024-04-26T17:53:49.758969+0000","last_undegraded":"2024-04-26T17:53:49.758969+0000","last_fullsized":"2024-04-26T17:53:49.758969+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:58:09.856021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764723+0000","last_change":"2024-04-26T17:53:03.610866+0000","last_active":"2024-04-26T17:53:49.764723+0000","last_peered":"2024-04-26T17:53:49.764723+0000","last_clean":"2024-04-26T17:53:49.764723+0000","last_became_active":"2024-04-26T17:53:03.610326+0000","last_became_peered":"2024-04-26T17:53:03.610326+0000","last_unstale":"2024-04-26T17:53:49.764723+0000","last_undegraded":"2024-04-26T17:53:49.764723+0000","last_fullsized":"2024-04-26T17:53:49.764723+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:20.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758767+0000","last_change":"2024-04-26T17:53:03.615336+0000","last_active":"2024-04-26T17:53:49.758767+0000","last_peered":"2024-04-26T17:53:49.758767+0000","last_clean":"2024-04-26T17:53:49.758767+0000","last_became_active":"2024-04-26T17:53:03.614447+0000","last_became_peered":"2024-04-26T17:53:03.614447+0000","last_unstale":"2024-04-26T17:53:49.758767+0000","last_undegraded":"2024-04-26T17:53:49.758767+0000","last_fullsized":"2024-04-26T17:53:49.758767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:36:10.091929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760835+0000","last_change":"2024-04-26T17:53:03.614863+0000","last_active":"2024-04-26T17:53:49.760835+0000","last_peered":"2024-04-26T17:53:49.760835+0000","last_clean":"2024-04-26T17:53:49.760835+0000","last_became_active":"2024-04-26T17:53:03.614520+0000","last_became_peered":"2024-04-26T17:53:03.614520+0000","last_unstale":"2024-04-26T17:53:49.760835+0000","last_undegraded":"2024-04-26T17:53:49.760835+0000","last_fullsized":"2024-04-26T17:53:49.760835+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:51.367785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764960+0000","last_change":"2024-04-26T17:53:03.618521+0000","last_active":"2024-04-26T17:53:49.764960+0000","last_peered":"2024-04-26T17:53:49.764960+0000","last_clean":"2024-04-26T17:53:49.764960+0000","last_became_active":"2024-04-26T17:53:03.617658+0000","last_became_peered":"2024-04-26T17:53:03.617658+0000","last_unstale":"2024-04-26T17:53:49.764960+0000","last_undegraded":"2024-04-26T17:53:49.764960+0000","last_fullsized":"2024-04-26T17:53:49.764960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:39.813098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280045,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66336,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":202,"num_read_kb":168,"num_write":165,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":199008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383705,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":479802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645342,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1130496,"data_stored":1129361,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":463023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070503,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":479958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332141,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":1071888,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495667,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":983040,"data_stored":479427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":541555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":480698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":543296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182464,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":471308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":479864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":1063142,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":63913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T17:53:56.696 INFO:tasks.cephadm.ceph_manager.ceph:PG 3.1e is not active+clean 2024-04-26T17:53:56.697 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '3.1e', 'version': "0'0", 'reported_seq': 2, 'reported_epoch': 89, 'state': 'peering', 'last_fresh': '2024-04-26T17:53:03.598208+0000', 'last_change': '2024-04-26T17:53:49.721525+0000', 'last_active': '2024-04-26T17:53:02.586248+0000', 'last_peered': '2024-04-26T17:53:02.586248+0000', 'last_clean': '2024-04-26T17:53:02.586248+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T17:53:03.598208+0000', 'last_undegraded': '2024-04-26T17:53:03.598208+0000', 'last_fullsized': '2024-04-26T17:53:03.598208+0000', 'mapping_epoch': 89, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 85, 'last_epoch_clean': 86, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T17:53:02.586248+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T17:53:02.586248+0000', 'last_clean_scrub_stamp': '2024-04-26T17:53:02.586248+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9, 2, 5], 'acting': [9, 2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-04-26T17:53:56.697 INFO:tasks.cephadm.ceph_manager.ceph:PG 2.4 is not active+clean 2024-04-26T17:53:56.697 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 88, 'state': 'peering', 'last_fresh': '2024-04-26T17:53:05.613913+0000', 'last_change': '2024-04-26T17:53:48.756965+0000', 'last_active': '2024-04-26T17:53:48.756965+0000', 'last_peered': '2024-04-26T17:53:05.613913+0000', 'last_clean': '2024-04-26T17:53:05.613913+0000', 'last_became_active': '2024-04-26T17:53:01.145146+0000', 'last_became_peered': '2024-04-26T17:53:01.145146+0000', 'last_unstale': '2024-04-26T17:53:05.613913+0000', 'last_undegraded': '2024-04-26T17:53:05.613913+0000', 'last_fullsized': '2024-04-26T17:53:05.613913+0000', 'mapping_epoch': 89, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 83, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T17:52:59.560312+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T17:52:59.560312+0000', 'last_clean_scrub_stamp': '2024-04-26T17:52:59.560312+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9, 11, 0], 'acting': [9, 11, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 2, 11], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-04-26T17:53:56.697 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-26T17:53:56.949 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:53:57.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:56 smithi031 ceph-mon[29164]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:53:57.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:56 smithi097 ceph-mon[32419]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:53:57.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:56 smithi186 ceph-mon[32797]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:53:58.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:57 smithi031 ceph-mon[29164]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 639 B/s wr, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T17:53:58.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:57 smithi097 ceph-mon[32419]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 639 B/s wr, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T17:53:58.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:57 smithi186 ceph-mon[32797]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 639 B/s wr, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T17:53:58.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.626+0000 7f4001d9a700 1 -- 172.21.15.31:0/2993581031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ffc1094a0 msgr2=0x7f3ffc109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:58.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.626+0000 7f4001d9a700 1 --2- 172.21.15.31:0/2993581031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ffc1094a0 0x7f3ffc109880 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0009990 tx=0x7f3ff0009ca0 comp rx=0 tx=0).stop 2024-04-26T17:53:58.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.626+0000 7f4001d9a700 1 -- 172.21.15.31:0/2993581031 shutdown_connections 2024-04-26T17:53:58.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.626+0000 7f4001d9a700 1 --2- 172.21.15.31:0/2993581031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ffc06bad0 0x7f3ffc1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:58.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.626+0000 7f4001d9a700 1 --2- 172.21.15.31:0/2993581031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ffc1094a0 0x7f3ffc109880 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:58.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.626+0000 7f4001d9a700 1 --2- 172.21.15.31:0/2993581031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ffc06b130 0x7f3ffc06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:58.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.626+0000 7f4001d9a700 1 -- 172.21.15.31:0/2993581031 >> 172.21.15.31:0/2993581031 conn(0x7f3ffc0792d0 msgr2=0x7f3ffc0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:58.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.626+0000 7f4001d9a700 1 -- 172.21.15.31:0/2993581031 shutdown_connections 2024-04-26T17:53:58.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.626+0000 7f4001d9a700 1 -- 172.21.15.31:0/2993581031 wait complete. 2024-04-26T17:53:58.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f4001d9a700 1 Processor -- start 2024-04-26T17:53:58.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f4001d9a700 1 -- start start 2024-04-26T17:53:58.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f4001d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ffc06b130 0x7f3ffc1023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:58.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f4001d9a700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ffc06bad0 0x7f3ffc102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:58.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f4001d9a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ffc1094a0 0x7f3ffc102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:58.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f4001d9a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc113590 con 0x7f3ffc06b130 2024-04-26T17:53:58.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f4001d9a700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc113410 con 0x7f3ffc06bad0 2024-04-26T17:53:58.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f4001d9a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc113710 con 0x7f3ffc1094a0 2024-04-26T17:53:58.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f3ffb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ffc06b130 0x7f3ffc1023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:58.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f3ffb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ffc06b130 0x7f3ffc1023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36584/0 (socket says 172.21.15.31:36584) 2024-04-26T17:53:58.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f3ffb7fe700 1 -- 172.21.15.31:0/764524795 learned_addr learned my addr 172.21.15.31:0/764524795 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:53:58.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f3ffaffd700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ffc06bad0 0x7f3ffc102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:58.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f3ffbfff700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ffc1094a0 0x7f3ffc102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:58.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f3ffb7fe700 1 -- 172.21.15.31:0/764524795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ffc1094a0 msgr2=0x7f3ffc102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:58.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f3ffb7fe700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ffc1094a0 0x7f3ffc102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:58.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f3ffb7fe700 1 -- 172.21.15.31:0/764524795 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ffc06bad0 msgr2=0x7f3ffc102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:58.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f3ffb7fe700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ffc06bad0 0x7f3ffc102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:58.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f3ffb7fe700 1 -- 172.21.15.31:0/764524795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ffc1a8010 con 0x7f3ffc06b130 2024-04-26T17:53:58.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.627+0000 7f3ffaffd700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ffc06bad0 0x7f3ffc102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:53:58.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.628+0000 7f3ffb7fe700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ffc06b130 0x7f3ffc1023f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0000c00 tx=0x7f3ff00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:58.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.628+0000 7f3ff8ff9700 1 -- 172.21.15.31:0/764524795 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0027070 con 0x7f3ffc06b130 2024-04-26T17:53:58.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.628+0000 7f3ff8ff9700 1 -- 172.21.15.31:0/764524795 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff0003fa0 con 0x7f3ffc06b130 2024-04-26T17:53:58.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.628+0000 7f4001d9a700 1 -- 172.21.15.31:0/764524795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ffc1a82a0 con 0x7f3ffc06b130 2024-04-26T17:53:58.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.628+0000 7f3ff8ff9700 1 -- 172.21.15.31:0/764524795 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0022db0 con 0x7f3ffc06b130 2024-04-26T17:53:58.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.628+0000 7f4001d9a700 1 -- 172.21.15.31:0/764524795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ffc1a86f0 con 0x7f3ffc06b130 2024-04-26T17:53:58.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.629+0000 7f4001d9a700 1 -- 172.21.15.31:0/764524795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ffc06a110 con 0x7f3ffc06b130 2024-04-26T17:53:58.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.630+0000 7f3ff8ff9700 1 -- 172.21.15.31:0/764524795 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3ff002a070 con 0x7f3ffc06b130 2024-04-26T17:53:58.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.631+0000 7f3ff8ff9700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3fe40a3cd0 0x7f3fe40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:53:58.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.631+0000 7f3ffaffd700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3fe40a3cd0 0x7f3fe40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:53:58.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.631+0000 7f3ff8ff9700 1 -- 172.21.15.31:0/764524795 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f3ff00d16b0 con 0x7f3ffc06b130 2024-04-26T17:53:58.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.631+0000 7f3ffaffd700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3fe40a3cd0 0x7f3fe40a6190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc079a10 tx=0x7f3fec005f50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:53:58.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.634+0000 7f3ff8ff9700 1 -- 172.21.15.31:0/764524795 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3ff0013070 con 0x7f3ffc06b130 2024-04-26T17:53:58.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.940+0000 7f4001d9a700 1 -- 172.21.15.31:0/764524795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f3ffc02e650 con 0x7f3ffc06b130 2024-04-26T17:53:58.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.940+0000 7f3ff8ff9700 1 -- 172.21.15.31:0/764524795 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+2461 (secure 0 0 0) 0x7f3ff009df90 con 0x7f3ffc06b130 2024-04-26T17:53:58.943 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:53:58.943 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":430,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":90,"num_osds":12,"num_up_osds":12,"osd_up_since":1714153920,"num_in_osds":12,"osd_in_since":1714153900,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":64,"data_bytes":656938,"bytes_used":369881088,"bytes_avail":1151570546688,"bytes_total":1151940427776,"recovering_objects_per_sec":0,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":1,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"read_bytes_sec":4389,"write_bytes_sec":535,"read_op_per_sec":6,"write_op_per_sec":0},"fsmap":{"epoch":25,"id":1,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":1,"rank":2,"name":"e","status":"up:standby-replay","gid":24428},{"filesystem_id":1,"rank":1,"name":"f","status":"up:standby-replay","gid":24439},{"filesystem_id":1,"rank":0,"name":"c","status":"up:standby-replay","gid":24445},{"filesystem_id":1,"rank":2,"name":"h","status":"up:active","gid":24446},{"filesystem_id":1,"rank":1,"name":"i","status":"up:active","gid":24457},{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":24473}],"up:standby":6},"mgrmap":{"available":true,"num_standbys":2,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.31:8443/"}},"servicemap":{"epoch":24,"modified":"2024-04-26T17:53:48.317268+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"e":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"f":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"h":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"i":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T17:53:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.942+0000 7f4001d9a700 1 -- 172.21.15.31:0/764524795 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3fe40a3cd0 msgr2=0x7f3fe40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.942+0000 7f4001d9a700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3fe40a3cd0 0x7f3fe40a6190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc079a10 tx=0x7f3fec005f50 comp rx=0 tx=0).stop 2024-04-26T17:53:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.942+0000 7f4001d9a700 1 -- 172.21.15.31:0/764524795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ffc06b130 msgr2=0x7f3ffc1023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:53:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.942+0000 7f4001d9a700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ffc06b130 0x7f3ffc1023f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0000c00 tx=0x7f3ff00047b0 comp rx=0 tx=0).stop 2024-04-26T17:53:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.942+0000 7f4001d9a700 1 -- 172.21.15.31:0/764524795 shutdown_connections 2024-04-26T17:53:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.942+0000 7f4001d9a700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f3fe40a3cd0 0x7f3fe40a6190 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.942+0000 7f4001d9a700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ffc1094a0 0x7f3ffc102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:58.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.942+0000 7f4001d9a700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ffc06b130 0x7f3ffc1023f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:58.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.942+0000 7f4001d9a700 1 --2- 172.21.15.31:0/764524795 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3ffc06bad0 0x7f3ffc102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:53:58.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.942+0000 7f4001d9a700 1 -- 172.21.15.31:0/764524795 >> 172.21.15.31:0/764524795 conn(0x7f3ffc0792d0 msgr2=0x7f3ffc1004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:53:58.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.943+0000 7f4001d9a700 1 -- 172.21.15.31:0/764524795 shutdown_connections 2024-04-26T17:53:58.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:53:58.943+0000 7f4001d9a700 1 -- 172.21.15.31:0/764524795 wait complete. 2024-04-26T17:53:59.619 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-26T17:53:59.619 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T17:53:59.882 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:00.005 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:59 smithi031 ceph-mon[29164]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 535 B/s wr, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T17:54:00.005 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:59 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/764524795' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T17:54:00.005 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:53:59 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:54:00.069 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:59 smithi186 ceph-mon[32797]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 535 B/s wr, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T17:54:00.069 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:59 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/764524795' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T17:54:00.069 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:53:59 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:54:00.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:59 smithi097 ceph-mon[32419]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 535 B/s wr, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T17:54:00.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:59 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/764524795' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T17:54:00.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:53:59 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:54:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.526+0000 7fe2b0172700 1 -- 172.21.15.31:0/1039072859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2a8109480 msgr2=0x7fe2a8109860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.527+0000 7fe2b0172700 1 --2- 172.21.15.31:0/1039072859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2a8109480 0x7fe2a8109860 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009990 tx=0x7fe2a4009ca0 comp rx=0 tx=0).stop 2024-04-26T17:54:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.527+0000 7fe2b0172700 1 -- 172.21.15.31:0/1039072859 shutdown_connections 2024-04-26T17:54:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.527+0000 7fe2b0172700 1 --2- 172.21.15.31:0/1039072859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2a8103ea0 0x7fe2a81102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.527+0000 7fe2b0172700 1 --2- 172.21.15.31:0/1039072859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2a8109480 0x7fe2a8109860 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.527+0000 7fe2b0172700 1 --2- 172.21.15.31:0/1039072859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2a81034e0 0x7fe2a8103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.527+0000 7fe2b0172700 1 -- 172.21.15.31:0/1039072859 >> 172.21.15.31:0/1039072859 conn(0x7fe2a807a5e0 msgr2=0x7fe2a807a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.527+0000 7fe2b0172700 1 -- 172.21.15.31:0/1039072859 shutdown_connections 2024-04-26T17:54:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.527+0000 7fe2b0172700 1 -- 172.21.15.31:0/1039072859 wait complete. 2024-04-26T17:54:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.527+0000 7fe2b0172700 1 Processor -- start 2024-04-26T17:54:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2b0172700 1 -- start start 2024-04-26T17:54:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2b0172700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2a81034e0 0x7fe2a819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2b0172700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2a8103ea0 0x7fe2a819fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2b0172700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2a8109480 0x7fe2a8199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2b0172700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a8113590 con 0x7fe2a8109480 2024-04-26T17:54:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2b0172700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a8113410 con 0x7fe2a8103ea0 2024-04-26T17:54:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2b0172700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a8113710 con 0x7fe2a81034e0 2024-04-26T17:54:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2ae70f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2a8109480 0x7fe2a8199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2ae70f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2a8109480 0x7fe2a8199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36612/0 (socket says 172.21.15.31:36612) 2024-04-26T17:54:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2ae70f700 1 -- 172.21.15.31:0/1354595577 learned_addr learned my addr 172.21.15.31:0/1354595577 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2ad70d700 1 --2- 172.21.15.31:0/1354595577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2a8103ea0 0x7fe2a819fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2adf0e700 1 --2- 172.21.15.31:0/1354595577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2a81034e0 0x7fe2a819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2ae70f700 1 -- 172.21.15.31:0/1354595577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2a81034e0 msgr2=0x7fe2a819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2ae70f700 1 --2- 172.21.15.31:0/1354595577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2a81034e0 0x7fe2a819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2ae70f700 1 -- 172.21.15.31:0/1354595577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2a8103ea0 msgr2=0x7fe2a819fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2ae70f700 1 --2- 172.21.15.31:0/1354595577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2a8103ea0 0x7fe2a819fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2ae70f700 1 -- 172.21.15.31:0/1354595577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a819a090 con 0x7fe2a8109480 2024-04-26T17:54:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.528+0000 7fe2ae70f700 1 --2- 172.21.15.31:0/1354595577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2a8109480 0x7fe2a8199930 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0007ee0 tx=0x7fe2a000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:01.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.529+0000 7fe29effd700 1 -- 172.21.15.31:0/1354595577 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0004bb0 con 0x7fe2a8109480 2024-04-26T17:54:01.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.529+0000 7fe29effd700 1 -- 172.21.15.31:0/1354595577 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2a0004d50 con 0x7fe2a8109480 2024-04-26T17:54:01.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.529+0000 7fe2b0172700 1 -- 172.21.15.31:0/1354595577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2a819a380 con 0x7fe2a8109480 2024-04-26T17:54:01.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.529+0000 7fe29effd700 1 -- 172.21.15.31:0/1354595577 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0005920 con 0x7fe2a8109480 2024-04-26T17:54:01.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.529+0000 7fe2b0172700 1 -- 172.21.15.31:0/1354595577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2a81a6640 con 0x7fe2a8109480 2024-04-26T17:54:01.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.530+0000 7fe2b0172700 1 -- 172.21.15.31:0/1354595577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2a8104b30 con 0x7fe2a8109480 2024-04-26T17:54:01.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.531+0000 7fe29effd700 1 -- 172.21.15.31:0/1354595577 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe2a001a020 con 0x7fe2a8109480 2024-04-26T17:54:01.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.532+0000 7fe29effd700 1 --2- 172.21.15.31:0/1354595577 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe2940a3cd0 0x7fe2940a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:01.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.532+0000 7fe2adf0e700 1 --2- 172.21.15.31:0/1354595577 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe2940a3cd0 0x7fe2940a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:01.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.532+0000 7fe29effd700 1 -- 172.21.15.31:0/1354595577 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fe2a00cfbd0 con 0x7fe2a8109480 2024-04-26T17:54:01.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.532+0000 7fe2adf0e700 1 --2- 172.21.15.31:0/1354595577 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe2940a3cd0 0x7fe2940a6190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe2a819aa70 tx=0x7fe2a4005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:01.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.536+0000 7fe29effd700 1 -- 172.21.15.31:0/1354595577 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe2a0010070 con 0x7fe2a8109480 2024-04-26T17:54:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.749+0000 7fe2b0172700 1 -- 172.21.15.31:0/1354595577 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe2a8063860 con 0x7fe2940a3cd0 2024-04-26T17:54:01.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.756+0000 7fe29effd700 1 -- 172.21.15.31:0/1354595577 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258889 (secure 0 0 0) 0x7fe2a8063860 con 0x7fe2940a3cd0 2024-04-26T17:54:01.760 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:54:01.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.759+0000 7fe2b0172700 1 -- 172.21.15.31:0/1354595577 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe2940a3cd0 msgr2=0x7fe2940a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.759+0000 7fe2b0172700 1 --2- 172.21.15.31:0/1354595577 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe2940a3cd0 0x7fe2940a6190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe2a819aa70 tx=0x7fe2a4005960 comp rx=0 tx=0).stop 2024-04-26T17:54:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.759+0000 7fe2b0172700 1 -- 172.21.15.31:0/1354595577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2a8109480 msgr2=0x7fe2a8199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.759+0000 7fe2b0172700 1 --2- 172.21.15.31:0/1354595577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2a8109480 0x7fe2a8199930 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0007ee0 tx=0x7fe2a000d3b0 comp rx=0 tx=0).stop 2024-04-26T17:54:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.759+0000 7fe2b0172700 1 -- 172.21.15.31:0/1354595577 shutdown_connections 2024-04-26T17:54:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.759+0000 7fe2b0172700 1 --2- 172.21.15.31:0/1354595577 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe2940a3cd0 0x7fe2940a6190 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.759+0000 7fe2b0172700 1 --2- 172.21.15.31:0/1354595577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2a8109480 0x7fe2a8199930 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.759+0000 7fe2b0172700 1 --2- 172.21.15.31:0/1354595577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2a81034e0 0x7fe2a819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.759+0000 7fe2b0172700 1 --2- 172.21.15.31:0/1354595577 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2a8103ea0 0x7fe2a819fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.759+0000 7fe2b0172700 1 -- 172.21.15.31:0/1354595577 >> 172.21.15.31:0/1354595577 conn(0x7fe2a807a5e0 msgr2=0x7fe2a8101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.760+0000 7fe2b0172700 1 -- 172.21.15.31:0/1354595577 shutdown_connections 2024-04-26T17:54:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:01.760+0000 7fe2b0172700 1 -- 172.21.15.31:0/1354595577 wait complete. 2024-04-26T17:54:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-26T17:54:02.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:01 smithi031 ceph-mon[29164]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 484 B/s wr, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T17:54:02.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:02.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:02.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:02.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:01 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:01 smithi097 ceph-mon[32419]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 484 B/s wr, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T17:54:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:01 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:02.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:01 smithi186 ceph-mon[32797]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 484 B/s wr, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T17:54:02.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:02.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:02.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:02.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:01 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:02.678 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-04-26T17:54:00.319867+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656938,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":369,"num_read_kb":307,"num_write":348,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361212,"kb_used_data":12012,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580612,"statfs":{"total":1151940427776,"available":1151570546688,"internally_reserved":0,"allocated":12300288,"data_stored":7721454,"data_compressed":106863,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4257,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":41,"num_write":1,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.560673"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765218+0000","last_change":"2024-04-26T17:53:03.616592+0000","last_active":"2024-04-26T17:53:49.765218+0000","last_peered":"2024-04-26T17:53:49.765218+0000","last_clean":"2024-04-26T17:53:49.765218+0000","last_became_active":"2024-04-26T17:53:03.616255+0000","last_became_peered":"2024-04-26T17:53:03.616255+0000","last_unstale":"2024-04-26T17:53:49.765218+0000","last_undegraded":"2024-04-26T17:53:49.765218+0000","last_fullsized":"2024-04-26T17:53:49.765218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:43:31.183679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765248+0000","last_change":"2024-04-26T17:53:03.615402+0000","last_active":"2024-04-26T17:53:49.765248+0000","last_peered":"2024-04-26T17:53:49.765248+0000","last_clean":"2024-04-26T17:53:49.765248+0000","last_became_active":"2024-04-26T17:53:03.615048+0000","last_became_peered":"2024-04-26T17:53:03.615048+0000","last_unstale":"2024-04-26T17:53:49.765248+0000","last_undegraded":"2024-04-26T17:53:49.765248+0000","last_fullsized":"2024-04-26T17:53:49.765248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:05:12.454770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765118+0000","last_change":"2024-04-26T17:53:03.622625+0000","last_active":"2024-04-26T17:53:49.765118+0000","last_peered":"2024-04-26T17:53:49.765118+0000","last_clean":"2024-04-26T17:53:49.765118+0000","last_became_active":"2024-04-26T17:53:03.622305+0000","last_became_peered":"2024-04-26T17:53:03.622305+0000","last_unstale":"2024-04-26T17:53:49.765118+0000","last_undegraded":"2024-04-26T17:53:49.765118+0000","last_fullsized":"2024-04-26T17:53:49.765118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:06.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891229+0000","last_change":"2024-04-26T17:53:03.617650+0000","last_active":"2024-04-26T17:53:49.891229+0000","last_peered":"2024-04-26T17:53:49.891229+0000","last_clean":"2024-04-26T17:53:49.891229+0000","last_became_active":"2024-04-26T17:53:03.617220+0000","last_became_peered":"2024-04-26T17:53:03.617220+0000","last_unstale":"2024-04-26T17:53:49.891229+0000","last_undegraded":"2024-04-26T17:53:49.891229+0000","last_fullsized":"2024-04-26T17:53:49.891229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:31.765577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767329+0000","last_change":"2024-04-26T17:53:03.615429+0000","last_active":"2024-04-26T17:53:49.767329+0000","last_peered":"2024-04-26T17:53:49.767329+0000","last_clean":"2024-04-26T17:53:49.767329+0000","last_became_active":"2024-04-26T17:53:03.615079+0000","last_became_peered":"2024-04-26T17:53:03.615079+0000","last_unstale":"2024-04-26T17:53:49.767329+0000","last_undegraded":"2024-04-26T17:53:49.767329+0000","last_fullsized":"2024-04-26T17:53:49.767329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:20.994016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764669+0000","last_change":"2024-04-26T17:53:03.624619+0000","last_active":"2024-04-26T17:53:49.764669+0000","last_peered":"2024-04-26T17:53:49.764669+0000","last_clean":"2024-04-26T17:53:49.764669+0000","last_became_active":"2024-04-26T17:53:03.624251+0000","last_became_peered":"2024-04-26T17:53:03.624251+0000","last_unstale":"2024-04-26T17:53:49.764669+0000","last_undegraded":"2024-04-26T17:53:49.764669+0000","last_fullsized":"2024-04-26T17:53:49.764669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:50:17.535235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767186+0000","last_change":"2024-04-26T17:53:03.619325+0000","last_active":"2024-04-26T17:53:49.767186+0000","last_peered":"2024-04-26T17:53:49.767186+0000","last_clean":"2024-04-26T17:53:49.767186+0000","last_became_active":"2024-04-26T17:53:03.618989+0000","last_became_peered":"2024-04-26T17:53:03.618989+0000","last_unstale":"2024-04-26T17:53:49.767186+0000","last_undegraded":"2024-04-26T17:53:49.767186+0000","last_fullsized":"2024-04-26T17:53:49.767186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:10.195742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764403+0000","last_change":"2024-04-26T17:53:03.609346+0000","last_active":"2024-04-26T17:53:49.764403+0000","last_peered":"2024-04-26T17:53:49.764403+0000","last_clean":"2024-04-26T17:53:49.764403+0000","last_became_active":"2024-04-26T17:53:03.609021+0000","last_became_peered":"2024-04-26T17:53:03.609021+0000","last_unstale":"2024-04-26T17:53:49.764403+0000","last_undegraded":"2024-04-26T17:53:49.764403+0000","last_fullsized":"2024-04-26T17:53:49.764403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:44.328599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759099+0000","last_change":"2024-04-26T17:53:03.618573+0000","last_active":"2024-04-26T17:53:49.759099+0000","last_peered":"2024-04-26T17:53:49.759099+0000","last_clean":"2024-04-26T17:53:49.759099+0000","last_became_active":"2024-04-26T17:53:03.618223+0000","last_became_peered":"2024-04-26T17:53:03.618223+0000","last_unstale":"2024-04-26T17:53:49.759099+0000","last_undegraded":"2024-04-26T17:53:49.759099+0000","last_fullsized":"2024-04-26T17:53:49.759099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:16.749205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765013+0000","last_change":"2024-04-26T17:53:03.621530+0000","last_active":"2024-04-26T17:53:49.765013+0000","last_peered":"2024-04-26T17:53:49.765013+0000","last_clean":"2024-04-26T17:53:49.765013+0000","last_became_active":"2024-04-26T17:53:03.621174+0000","last_became_peered":"2024-04-26T17:53:03.621174+0000","last_unstale":"2024-04-26T17:53:49.765013+0000","last_undegraded":"2024-04-26T17:53:49.765013+0000","last_fullsized":"2024-04-26T17:53:49.765013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:31.388232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767569+0000","last_change":"2024-04-26T17:53:03.617682+0000","last_active":"2024-04-26T17:53:49.767569+0000","last_peered":"2024-04-26T17:53:49.767569+0000","last_clean":"2024-04-26T17:53:49.767569+0000","last_became_active":"2024-04-26T17:53:03.617348+0000","last_became_peered":"2024-04-26T17:53:03.617348+0000","last_unstale":"2024-04-26T17:53:49.767569+0000","last_undegraded":"2024-04-26T17:53:49.767569+0000","last_fullsized":"2024-04-26T17:53:49.767569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:21.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891137+0000","last_change":"2024-04-26T17:53:03.617955+0000","last_active":"2024-04-26T17:53:49.891137+0000","last_peered":"2024-04-26T17:53:49.891137+0000","last_clean":"2024-04-26T17:53:49.891137+0000","last_became_active":"2024-04-26T17:53:03.616765+0000","last_became_peered":"2024-04-26T17:53:03.616765+0000","last_unstale":"2024-04-26T17:53:49.891137+0000","last_undegraded":"2024-04-26T17:53:49.891137+0000","last_fullsized":"2024-04-26T17:53:49.891137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:59.292277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767807+0000","last_change":"2024-04-26T17:53:49.767807+0000","last_active":"2024-04-26T17:53:49.767807+0000","last_peered":"2024-04-26T17:53:49.767807+0000","last_clean":"2024-04-26T17:53:49.767807+0000","last_became_active":"2024-04-26T17:53:49.767404+0000","last_became_peered":"2024-04-26T17:53:49.767404+0000","last_unstale":"2024-04-26T17:53:49.767807+0000","last_undegraded":"2024-04-26T17:53:49.767807+0000","last_fullsized":"2024-04-26T17:53:49.767807+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:20:45.482580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764992+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:53:49.764992+0000","last_peered":"2024-04-26T17:53:49.764992+0000","last_clean":"2024-04-26T17:53:49.764992+0000","last_became_active":"2024-04-26T17:53:03.616623+0000","last_became_peered":"2024-04-26T17:53:03.616623+0000","last_unstale":"2024-04-26T17:53:49.764992+0000","last_undegraded":"2024-04-26T17:53:49.764992+0000","last_fullsized":"2024-04-26T17:53:49.764992+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:04.231549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764782+0000","last_change":"2024-04-26T17:53:03.622147+0000","last_active":"2024-04-26T17:53:49.764782+0000","last_peered":"2024-04-26T17:53:49.764782+0000","last_clean":"2024-04-26T17:53:49.764782+0000","last_became_active":"2024-04-26T17:53:03.621542+0000","last_became_peered":"2024-04-26T17:53:03.621542+0000","last_unstale":"2024-04-26T17:53:49.764782+0000","last_undegraded":"2024-04-26T17:53:49.764782+0000","last_fullsized":"2024-04-26T17:53:49.764782+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:01.746954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758591+0000","last_change":"2024-04-26T17:53:03.624433+0000","last_active":"2024-04-26T17:53:49.758591+0000","last_peered":"2024-04-26T17:53:49.758591+0000","last_clean":"2024-04-26T17:53:49.758591+0000","last_became_active":"2024-04-26T17:53:03.624135+0000","last_became_peered":"2024-04-26T17:53:03.624135+0000","last_unstale":"2024-04-26T17:53:49.758591+0000","last_undegraded":"2024-04-26T17:53:49.758591+0000","last_fullsized":"2024-04-26T17:53:49.758591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:32:20.521917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765321+0000","last_change":"2024-04-26T17:53:03.623607+0000","last_active":"2024-04-26T17:53:49.765321+0000","last_peered":"2024-04-26T17:53:49.765321+0000","last_clean":"2024-04-26T17:53:49.765321+0000","last_became_active":"2024-04-26T17:53:03.621783+0000","last_became_peered":"2024-04-26T17:53:03.621783+0000","last_unstale":"2024-04-26T17:53:49.765321+0000","last_undegraded":"2024-04-26T17:53:49.765321+0000","last_fullsized":"2024-04-26T17:53:49.765321+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:38:59.872868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.762359+0000","last_change":"2024-04-26T17:53:03.614974+0000","last_active":"2024-04-26T17:53:49.762359+0000","last_peered":"2024-04-26T17:53:49.762359+0000","last_clean":"2024-04-26T17:53:49.762359+0000","last_became_active":"2024-04-26T17:53:03.614619+0000","last_became_peered":"2024-04-26T17:53:03.614619+0000","last_unstale":"2024-04-26T17:53:49.762359+0000","last_undegraded":"2024-04-26T17:53:49.762359+0000","last_fullsized":"2024-04-26T17:53:49.762359+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:50.729531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769611+0000","last_change":"2024-04-26T17:53:03.611140+0000","last_active":"2024-04-26T17:53:49.769611+0000","last_peered":"2024-04-26T17:53:49.769611+0000","last_clean":"2024-04-26T17:53:49.769611+0000","last_became_active":"2024-04-26T17:53:03.610549+0000","last_became_peered":"2024-04-26T17:53:03.610549+0000","last_unstale":"2024-04-26T17:53:49.769611+0000","last_undegraded":"2024-04-26T17:53:49.769611+0000","last_fullsized":"2024-04-26T17:53:49.769611+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:46:17.178598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.761817+0000","last_change":"2024-04-26T17:53:00.585971+0000","last_active":"2024-04-26T17:53:49.761817+0000","last_peered":"2024-04-26T17:53:49.761817+0000","last_clean":"2024-04-26T17:53:49.761817+0000","last_became_active":"2024-04-26T17:53:00.585648+0000","last_became_peered":"2024-04-26T17:53:00.585648+0000","last_unstale":"2024-04-26T17:53:49.761817+0000","last_undegraded":"2024-04-26T17:53:49.761817+0000","last_fullsized":"2024-04-26T17:53:49.761817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:00:24.082773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.770579+0000","last_change":"2024-04-26T17:53:49.770579+0000","last_active":"2024-04-26T17:53:49.770579+0000","last_peered":"2024-04-26T17:53:49.770579+0000","last_clean":"2024-04-26T17:53:49.770579+0000","last_became_active":"2024-04-26T17:53:49.770250+0000","last_became_peered":"2024-04-26T17:53:49.770250+0000","last_unstale":"2024-04-26T17:53:49.770579+0000","last_undegraded":"2024-04-26T17:53:49.770579+0000","last_fullsized":"2024-04-26T17:53:49.770579+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:35.758790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.766024+0000","last_change":"2024-04-26T17:53:01.145438+0000","last_active":"2024-04-26T17:53:49.766024+0000","last_peered":"2024-04-26T17:53:49.766024+0000","last_clean":"2024-04-26T17:53:49.766024+0000","last_became_active":"2024-04-26T17:53:01.145030+0000","last_became_peered":"2024-04-26T17:53:01.145030+0000","last_unstale":"2024-04-26T17:53:49.766024+0000","last_undegraded":"2024-04-26T17:53:49.766024+0000","last_fullsized":"2024-04-26T17:53:49.766024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:15.650247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765536+0000","last_change":"2024-04-26T17:53:03.622647+0000","last_active":"2024-04-26T17:53:49.765536+0000","last_peered":"2024-04-26T17:53:49.765536+0000","last_clean":"2024-04-26T17:53:49.765536+0000","last_became_active":"2024-04-26T17:53:03.622298+0000","last_became_peered":"2024-04-26T17:53:03.622298+0000","last_unstale":"2024-04-26T17:53:49.765536+0000","last_undegraded":"2024-04-26T17:53:49.765536+0000","last_fullsized":"2024-04-26T17:53:49.765536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:37.230539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.775803+0000","last_change":"2024-04-26T17:53:49.775405+0000","last_active":"2024-04-26T17:53:49.775803+0000","last_peered":"2024-04-26T17:53:49.775803+0000","last_clean":"2024-04-26T17:53:49.775803+0000","last_became_active":"2024-04-26T17:53:49.770536+0000","last_became_peered":"2024-04-26T17:53:49.770536+0000","last_unstale":"2024-04-26T17:53:49.775803+0000","last_undegraded":"2024-04-26T17:53:49.775803+0000","last_fullsized":"2024-04-26T17:53:49.775803+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:34.643452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758674+0000","last_change":"2024-04-26T17:53:03.617462+0000","last_active":"2024-04-26T17:53:49.758674+0000","last_peered":"2024-04-26T17:53:49.758674+0000","last_clean":"2024-04-26T17:53:49.758674+0000","last_became_active":"2024-04-26T17:53:03.617115+0000","last_became_peered":"2024-04-26T17:53:03.617115+0000","last_unstale":"2024-04-26T17:53:49.758674+0000","last_undegraded":"2024-04-26T17:53:49.758674+0000","last_fullsized":"2024-04-26T17:53:49.758674+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:11.406146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767810+0000","last_change":"2024-04-26T17:53:00.583378+0000","last_active":"2024-04-26T17:53:49.767810+0000","last_peered":"2024-04-26T17:53:49.767810+0000","last_clean":"2024-04-26T17:53:49.767810+0000","last_became_active":"2024-04-26T17:53:00.583020+0000","last_became_peered":"2024-04-26T17:53:00.583020+0000","last_unstale":"2024-04-26T17:53:49.767810+0000","last_undegraded":"2024-04-26T17:53:49.767810+0000","last_fullsized":"2024-04-26T17:53:49.767810+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:24.089996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759512+0000","last_change":"2024-04-26T17:53:03.609557+0000","last_active":"2024-04-26T17:53:49.759512+0000","last_peered":"2024-04-26T17:53:49.759512+0000","last_clean":"2024-04-26T17:53:49.759512+0000","last_became_active":"2024-04-26T17:53:03.609028+0000","last_became_peered":"2024-04-26T17:53:03.609028+0000","last_unstale":"2024-04-26T17:53:49.759512+0000","last_undegraded":"2024-04-26T17:53:49.759512+0000","last_fullsized":"2024-04-26T17:53:49.759512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:39:22.860877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:53.743918+0000","last_change":"2024-04-26T17:53:00.586683+0000","last_active":"2024-04-26T17:53:53.743918+0000","last_peered":"2024-04-26T17:53:53.743918+0000","last_clean":"2024-04-26T17:53:53.743918+0000","last_became_active":"2024-04-26T17:53:00.586272+0000","last_became_peered":"2024-04-26T17:53:00.586272+0000","last_unstale":"2024-04-26T17:53:53.743918+0000","last_undegraded":"2024-04-26T17:53:53.743918+0000","last_fullsized":"2024-04-26T17:53:53.743918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:20:29.068607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767148+0000","last_change":"2024-04-26T17:53:03.611073+0000","last_active":"2024-04-26T17:53:49.767148+0000","last_peered":"2024-04-26T17:53:49.767148+0000","last_clean":"2024-04-26T17:53:49.767148+0000","last_became_active":"2024-04-26T17:53:03.610701+0000","last_became_peered":"2024-04-26T17:53:03.610701+0000","last_unstale":"2024-04-26T17:53:49.767148+0000","last_undegraded":"2024-04-26T17:53:49.767148+0000","last_fullsized":"2024-04-26T17:53:49.767148+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:01:00.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764638+0000","last_change":"2024-04-26T17:53:00.586082+0000","last_active":"2024-04-26T17:53:49.764638+0000","last_peered":"2024-04-26T17:53:49.764638+0000","last_clean":"2024-04-26T17:53:49.764638+0000","last_became_active":"2024-04-26T17:53:00.585678+0000","last_became_peered":"2024-04-26T17:53:00.585678+0000","last_unstale":"2024-04-26T17:53:49.764638+0000","last_undegraded":"2024-04-26T17:53:49.764638+0000","last_fullsized":"2024-04-26T17:53:49.764638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:55:25.523846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759383+0000","last_change":"2024-04-26T17:53:03.600230+0000","last_active":"2024-04-26T17:53:49.759383+0000","last_peered":"2024-04-26T17:53:49.759383+0000","last_clean":"2024-04-26T17:53:49.759383+0000","last_became_active":"2024-04-26T17:53:03.599713+0000","last_became_peered":"2024-04-26T17:53:03.599713+0000","last_unstale":"2024-04-26T17:53:49.759383+0000","last_undegraded":"2024-04-26T17:53:49.759383+0000","last_fullsized":"2024-04-26T17:53:49.759383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:45.822561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760523+0000","last_change":"2024-04-26T17:53:00.584588+0000","last_active":"2024-04-26T17:53:49.760523+0000","last_peered":"2024-04-26T17:53:49.760523+0000","last_clean":"2024-04-26T17:53:49.760523+0000","last_became_active":"2024-04-26T17:53:00.584284+0000","last_became_peered":"2024-04-26T17:53:00.584284+0000","last_unstale":"2024-04-26T17:53:49.760523+0000","last_undegraded":"2024-04-26T17:53:49.760523+0000","last_fullsized":"2024-04-26T17:53:49.760523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:08.428904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764864+0000","last_change":"2024-04-26T17:53:03.615237+0000","last_active":"2024-04-26T17:53:49.764864+0000","last_peered":"2024-04-26T17:53:49.764864+0000","last_clean":"2024-04-26T17:53:49.764864+0000","last_became_active":"2024-04-26T17:53:03.614704+0000","last_became_peered":"2024-04-26T17:53:03.614704+0000","last_unstale":"2024-04-26T17:53:49.764864+0000","last_undegraded":"2024-04-26T17:53:49.764864+0000","last_fullsized":"2024-04-26T17:53:49.764864+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:27.251839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758907+0000","last_change":"2024-04-26T17:53:00.589287+0000","last_active":"2024-04-26T17:53:49.758907+0000","last_peered":"2024-04-26T17:53:49.758907+0000","last_clean":"2024-04-26T17:53:49.758907+0000","last_became_active":"2024-04-26T17:53:00.588119+0000","last_became_peered":"2024-04-26T17:53:00.588119+0000","last_unstale":"2024-04-26T17:53:49.758907+0000","last_undegraded":"2024-04-26T17:53:49.758907+0000","last_fullsized":"2024-04-26T17:53:49.758907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:25.389777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.761931+0000","last_change":"2024-04-26T17:53:03.608061+0000","last_active":"2024-04-26T17:53:49.761931+0000","last_peered":"2024-04-26T17:53:49.761931+0000","last_clean":"2024-04-26T17:53:49.761931+0000","last_became_active":"2024-04-26T17:53:03.607624+0000","last_became_peered":"2024-04-26T17:53:03.607624+0000","last_unstale":"2024-04-26T17:53:49.761931+0000","last_undegraded":"2024-04-26T17:53:49.761931+0000","last_fullsized":"2024-04-26T17:53:49.761931+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:15.111011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:54.744837+0000","last_change":"2024-04-26T17:53:00.588621+0000","last_active":"2024-04-26T17:53:54.744837+0000","last_peered":"2024-04-26T17:53:54.744837+0000","last_clean":"2024-04-26T17:53:54.744837+0000","last_became_active":"2024-04-26T17:53:00.587330+0000","last_became_peered":"2024-04-26T17:53:00.587330+0000","last_unstale":"2024-04-26T17:53:54.744837+0000","last_undegraded":"2024-04-26T17:53:54.744837+0000","last_fullsized":"2024-04-26T17:53:54.744837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:01:12.178610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759017+0000","last_change":"2024-04-26T17:53:03.615723+0000","last_active":"2024-04-26T17:53:49.759017+0000","last_peered":"2024-04-26T17:53:49.759017+0000","last_clean":"2024-04-26T17:53:49.759017+0000","last_became_active":"2024-04-26T17:53:03.614931+0000","last_became_peered":"2024-04-26T17:53:03.614931+0000","last_unstale":"2024-04-26T17:53:49.759017+0000","last_undegraded":"2024-04-26T17:53:49.759017+0000","last_fullsized":"2024-04-26T17:53:49.759017+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:09.663876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767693+0000","last_change":"2024-04-26T17:53:00.589232+0000","last_active":"2024-04-26T17:53:49.767693+0000","last_peered":"2024-04-26T17:53:49.767693+0000","last_clean":"2024-04-26T17:53:49.767693+0000","last_became_active":"2024-04-26T17:53:00.588863+0000","last_became_peered":"2024-04-26T17:53:00.588863+0000","last_unstale":"2024-04-26T17:53:49.767693+0000","last_undegraded":"2024-04-26T17:53:49.767693+0000","last_fullsized":"2024-04-26T17:53:49.767693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:14.360782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765501+0000","last_change":"2024-04-26T17:53:03.624923+0000","last_active":"2024-04-26T17:53:49.765501+0000","last_peered":"2024-04-26T17:53:49.765501+0000","last_clean":"2024-04-26T17:53:49.765501+0000","last_became_active":"2024-04-26T17:53:03.624620+0000","last_became_peered":"2024-04-26T17:53:03.624620+0000","last_unstale":"2024-04-26T17:53:49.765501+0000","last_undegraded":"2024-04-26T17:53:49.765501+0000","last_fullsized":"2024-04-26T17:53:49.765501+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:50:51.259563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759526+0000","last_change":"2024-04-26T17:53:01.146950+0000","last_active":"2024-04-26T17:53:49.759526+0000","last_peered":"2024-04-26T17:53:49.759526+0000","last_clean":"2024-04-26T17:53:49.759526+0000","last_became_active":"2024-04-26T17:53:01.146591+0000","last_became_peered":"2024-04-26T17:53:01.146591+0000","last_unstale":"2024-04-26T17:53:49.759526+0000","last_undegraded":"2024-04-26T17:53:49.759526+0000","last_fullsized":"2024-04-26T17:53:49.759526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:17.756660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758787+0000","last_change":"2024-04-26T17:53:03.626944+0000","last_active":"2024-04-26T17:53:49.758787+0000","last_peered":"2024-04-26T17:53:49.758787+0000","last_clean":"2024-04-26T17:53:49.758787+0000","last_became_active":"2024-04-26T17:53:03.626594+0000","last_became_peered":"2024-04-26T17:53:03.626594+0000","last_unstale":"2024-04-26T17:53:49.758787+0000","last_undegraded":"2024-04-26T17:53:49.758787+0000","last_fullsized":"2024-04-26T17:53:49.758787+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:52:42.997251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'30","reported_seq":133,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:55.746477+0000","last_change":"2024-04-26T17:53:00.588353+0000","last_active":"2024-04-26T17:53:55.746477+0000","last_peered":"2024-04-26T17:53:55.746477+0000","last_clean":"2024-04-26T17:53:55.746477+0000","last_became_active":"2024-04-26T17:53:00.586351+0000","last_became_peered":"2024-04-26T17:53:00.586351+0000","last_unstale":"2024-04-26T17:53:55.746477+0000","last_undegraded":"2024-04-26T17:53:55.746477+0000","last_fullsized":"2024-04-26T17:53:55.746477+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:41.660525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62335,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":77,"num_read_kb":99,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768625+0000","last_change":"2024-04-26T17:53:03.612052+0000","last_active":"2024-04-26T17:53:49.768625+0000","last_peered":"2024-04-26T17:53:49.768625+0000","last_clean":"2024-04-26T17:53:49.768625+0000","last_became_active":"2024-04-26T17:53:03.611733+0000","last_became_peered":"2024-04-26T17:53:03.611733+0000","last_unstale":"2024-04-26T17:53:49.768625+0000","last_undegraded":"2024-04-26T17:53:49.768625+0000","last_fullsized":"2024-04-26T17:53:49.768625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:05:23.958317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769171+0000","last_change":"2024-04-26T17:53:00.588284+0000","last_active":"2024-04-26T17:53:49.769171+0000","last_peered":"2024-04-26T17:53:49.769171+0000","last_clean":"2024-04-26T17:53:49.769171+0000","last_became_active":"2024-04-26T17:53:00.587942+0000","last_became_peered":"2024-04-26T17:53:00.587942+0000","last_unstale":"2024-04-26T17:53:49.769171+0000","last_undegraded":"2024-04-26T17:53:49.769171+0000","last_fullsized":"2024-04-26T17:53:49.769171+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:43.782971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760366+0000","last_change":"2024-04-26T17:53:03.613092+0000","last_active":"2024-04-26T17:53:49.760366+0000","last_peered":"2024-04-26T17:53:49.760366+0000","last_clean":"2024-04-26T17:53:49.760366+0000","last_became_active":"2024-04-26T17:53:03.612773+0000","last_became_peered":"2024-04-26T17:53:03.612773+0000","last_unstale":"2024-04-26T17:53:49.760366+0000","last_undegraded":"2024-04-26T17:53:49.760366+0000","last_fullsized":"2024-04-26T17:53:49.760366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:41:21.340375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759054+0000","last_change":"2024-04-26T17:53:00.588777+0000","last_active":"2024-04-26T17:53:49.759054+0000","last_peered":"2024-04-26T17:53:49.759054+0000","last_clean":"2024-04-26T17:53:49.759054+0000","last_became_active":"2024-04-26T17:53:00.588380+0000","last_became_peered":"2024-04-26T17:53:00.588380+0000","last_unstale":"2024-04-26T17:53:49.759054+0000","last_undegraded":"2024-04-26T17:53:49.759054+0000","last_fullsized":"2024-04-26T17:53:49.759054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:44.587381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767750+0000","last_change":"2024-04-26T17:53:03.620344+0000","last_active":"2024-04-26T17:53:49.767750+0000","last_peered":"2024-04-26T17:53:49.767750+0000","last_clean":"2024-04-26T17:53:49.767750+0000","last_became_active":"2024-04-26T17:53:03.619944+0000","last_became_peered":"2024-04-26T17:53:03.619944+0000","last_unstale":"2024-04-26T17:53:49.767750+0000","last_undegraded":"2024-04-26T17:53:49.767750+0000","last_fullsized":"2024-04-26T17:53:49.767750+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:08:31.801342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:54.744969+0000","last_change":"2024-04-26T17:53:00.587733+0000","last_active":"2024-04-26T17:53:54.744969+0000","last_peered":"2024-04-26T17:53:54.744969+0000","last_clean":"2024-04-26T17:53:54.744969+0000","last_became_active":"2024-04-26T17:53:00.587373+0000","last_became_peered":"2024-04-26T17:53:00.587373+0000","last_unstale":"2024-04-26T17:53:54.744969+0000","last_undegraded":"2024-04-26T17:53:54.744969+0000","last_fullsized":"2024-04-26T17:53:54.744969+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:10.978944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767121+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:53:49.767121+0000","last_peered":"2024-04-26T17:53:49.767121+0000","last_clean":"2024-04-26T17:53:49.767121+0000","last_became_active":"2024-04-26T17:53:03.616588+0000","last_became_peered":"2024-04-26T17:53:03.616588+0000","last_unstale":"2024-04-26T17:53:49.767121+0000","last_undegraded":"2024-04-26T17:53:49.767121+0000","last_fullsized":"2024-04-26T17:53:49.767121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:46:38.419613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767233+0000","last_change":"2024-04-26T17:53:01.146922+0000","last_active":"2024-04-26T17:53:49.767233+0000","last_peered":"2024-04-26T17:53:49.767233+0000","last_clean":"2024-04-26T17:53:49.767233+0000","last_became_active":"2024-04-26T17:53:01.146520+0000","last_became_peered":"2024-04-26T17:53:01.146520+0000","last_unstale":"2024-04-26T17:53:49.767233+0000","last_undegraded":"2024-04-26T17:53:49.767233+0000","last_fullsized":"2024-04-26T17:53:49.767233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:12:49.238012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758964+0000","last_change":"2024-04-26T17:53:03.608513+0000","last_active":"2024-04-26T17:53:49.758964+0000","last_peered":"2024-04-26T17:53:49.758964+0000","last_clean":"2024-04-26T17:53:49.758964+0000","last_became_active":"2024-04-26T17:53:03.608167+0000","last_became_peered":"2024-04-26T17:53:03.608167+0000","last_unstale":"2024-04-26T17:53:49.758964+0000","last_undegraded":"2024-04-26T17:53:49.758964+0000","last_fullsized":"2024-04-26T17:53:49.758964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:03.002756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769191+0000","last_change":"2024-04-26T17:53:00.583878+0000","last_active":"2024-04-26T17:53:49.769191+0000","last_peered":"2024-04-26T17:53:49.769191+0000","last_clean":"2024-04-26T17:53:49.769191+0000","last_became_active":"2024-04-26T17:53:00.583159+0000","last_became_peered":"2024-04-26T17:53:00.583159+0000","last_unstale":"2024-04-26T17:53:49.769191+0000","last_undegraded":"2024-04-26T17:53:49.769191+0000","last_fullsized":"2024-04-26T17:53:49.769191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:49.850942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765106+0000","last_change":"2024-04-26T17:53:03.614467+0000","last_active":"2024-04-26T17:53:49.765106+0000","last_peered":"2024-04-26T17:53:49.765106+0000","last_clean":"2024-04-26T17:53:49.765106+0000","last_became_active":"2024-04-26T17:53:03.613972+0000","last_became_peered":"2024-04-26T17:53:03.613972+0000","last_unstale":"2024-04-26T17:53:49.765106+0000","last_undegraded":"2024-04-26T17:53:49.765106+0000","last_fullsized":"2024-04-26T17:53:49.765106+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:27.259777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767174+0000","last_change":"2024-04-26T17:53:49.767174+0000","last_active":"2024-04-26T17:53:49.767174+0000","last_peered":"2024-04-26T17:53:49.767174+0000","last_clean":"2024-04-26T17:53:49.767174+0000","last_became_active":"2024-04-26T17:53:49.766778+0000","last_became_peered":"2024-04-26T17:53:49.766778+0000","last_unstale":"2024-04-26T17:53:49.767174+0000","last_undegraded":"2024-04-26T17:53:49.767174+0000","last_fullsized":"2024-04-26T17:53:49.767174+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:59.240159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764767+0000","last_change":"2024-04-26T17:53:03.613888+0000","last_active":"2024-04-26T17:53:49.764767+0000","last_peered":"2024-04-26T17:53:49.764767+0000","last_clean":"2024-04-26T17:53:49.764767+0000","last_became_active":"2024-04-26T17:53:03.613499+0000","last_became_peered":"2024-04-26T17:53:03.613499+0000","last_unstale":"2024-04-26T17:53:49.764767+0000","last_undegraded":"2024-04-26T17:53:49.764767+0000","last_fullsized":"2024-04-26T17:53:49.764767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:15.549419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767375+0000","last_change":"2024-04-26T17:53:00.587803+0000","last_active":"2024-04-26T17:53:49.767375+0000","last_peered":"2024-04-26T17:53:49.767375+0000","last_clean":"2024-04-26T17:53:49.767375+0000","last_became_active":"2024-04-26T17:53:00.587382+0000","last_became_peered":"2024-04-26T17:53:00.587382+0000","last_unstale":"2024-04-26T17:53:49.767375+0000","last_undegraded":"2024-04-26T17:53:49.767375+0000","last_fullsized":"2024-04-26T17:53:49.767375+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:38.249478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769008+0000","last_change":"2024-04-26T17:53:49.769008+0000","last_active":"2024-04-26T17:53:49.769008+0000","last_peered":"2024-04-26T17:53:49.769008+0000","last_clean":"2024-04-26T17:53:49.769008+0000","last_became_active":"2024-04-26T17:53:49.768625+0000","last_became_peered":"2024-04-26T17:53:49.768625+0000","last_unstale":"2024-04-26T17:53:49.769008+0000","last_undegraded":"2024-04-26T17:53:49.769008+0000","last_fullsized":"2024-04-26T17:53:49.769008+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:27:46.412625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759215+0000","last_change":"2024-04-26T17:53:03.612899+0000","last_active":"2024-04-26T17:53:49.759215+0000","last_peered":"2024-04-26T17:53:49.759215+0000","last_clean":"2024-04-26T17:53:49.759215+0000","last_became_active":"2024-04-26T17:53:03.612564+0000","last_became_peered":"2024-04-26T17:53:03.612564+0000","last_unstale":"2024-04-26T17:53:49.759215+0000","last_undegraded":"2024-04-26T17:53:49.759215+0000","last_fullsized":"2024-04-26T17:53:49.759215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:07.368256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:56.746778+0000","last_change":"2024-04-26T17:53:01.145471+0000","last_active":"2024-04-26T17:53:56.746778+0000","last_peered":"2024-04-26T17:53:56.746778+0000","last_clean":"2024-04-26T17:53:56.746778+0000","last_became_active":"2024-04-26T17:53:01.145040+0000","last_became_peered":"2024-04-26T17:53:01.145040+0000","last_unstale":"2024-04-26T17:53:56.746778+0000","last_undegraded":"2024-04-26T17:53:56.746778+0000","last_fullsized":"2024-04-26T17:53:56.746778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:46:27.437329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765218+0000","last_change":"2024-04-26T17:53:03.618600+0000","last_active":"2024-04-26T17:53:49.765218+0000","last_peered":"2024-04-26T17:53:49.765218+0000","last_clean":"2024-04-26T17:53:49.765218+0000","last_became_active":"2024-04-26T17:53:03.618139+0000","last_became_peered":"2024-04-26T17:53:03.618139+0000","last_unstale":"2024-04-26T17:53:49.765218+0000","last_undegraded":"2024-04-26T17:53:49.765218+0000","last_fullsized":"2024-04-26T17:53:49.765218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:56.007659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891071+0000","last_change":"2024-04-26T17:53:00.586386+0000","last_active":"2024-04-26T17:53:49.891071+0000","last_peered":"2024-04-26T17:53:49.891071+0000","last_clean":"2024-04-26T17:53:49.891071+0000","last_became_active":"2024-04-26T17:53:00.585953+0000","last_became_peered":"2024-04-26T17:53:00.585953+0000","last_unstale":"2024-04-26T17:53:49.891071+0000","last_undegraded":"2024-04-26T17:53:49.891071+0000","last_fullsized":"2024-04-26T17:53:49.891071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:39:57.160983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":357,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767252+0000","last_change":"2024-04-26T17:51:38.839853+0000","last_active":"2024-04-26T17:53:49.767252+0000","last_peered":"2024-04-26T17:53:49.767252+0000","last_clean":"2024-04-26T17:53:49.767252+0000","last_became_active":"2024-04-26T17:51:38.839318+0000","last_became_peered":"2024-04-26T17:51:38.839318+0000","last_unstale":"2024-04-26T17:53:49.767252+0000","last_undegraded":"2024-04-26T17:53:49.767252+0000","last_fullsized":"2024-04-26T17:53:49.767252+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_clean_scrub_stamp":"2024-04-26T17:48:12.577543+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:08:50.683663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760286+0000","last_change":"2024-04-26T17:53:03.614170+0000","last_active":"2024-04-26T17:53:49.760286+0000","last_peered":"2024-04-26T17:53:49.760286+0000","last_clean":"2024-04-26T17:53:49.760286+0000","last_became_active":"2024-04-26T17:53:03.613836+0000","last_became_peered":"2024-04-26T17:53:03.613836+0000","last_unstale":"2024-04-26T17:53:49.760286+0000","last_undegraded":"2024-04-26T17:53:49.760286+0000","last_fullsized":"2024-04-26T17:53:49.760286+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:48.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767374+0000","last_change":"2024-04-26T17:53:00.587319+0000","last_active":"2024-04-26T17:53:49.767374+0000","last_peered":"2024-04-26T17:53:49.767374+0000","last_clean":"2024-04-26T17:53:49.767374+0000","last_became_active":"2024-04-26T17:53:00.586983+0000","last_became_peered":"2024-04-26T17:53:00.586983+0000","last_unstale":"2024-04-26T17:53:49.767374+0000","last_undegraded":"2024-04-26T17:53:49.767374+0000","last_fullsized":"2024-04-26T17:53:49.767374+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:03:43.470013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768739+0000","last_change":"2024-04-26T17:53:01.145672+0000","last_active":"2024-04-26T17:53:49.768739+0000","last_peered":"2024-04-26T17:53:49.768739+0000","last_clean":"2024-04-26T17:53:49.768739+0000","last_became_active":"2024-04-26T17:53:01.145274+0000","last_became_peered":"2024-04-26T17:53:01.145274+0000","last_unstale":"2024-04-26T17:53:49.768739+0000","last_undegraded":"2024-04-26T17:53:49.768739+0000","last_fullsized":"2024-04-26T17:53:49.768739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:21.122352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760603+0000","last_change":"2024-04-26T17:53:03.611232+0000","last_active":"2024-04-26T17:53:49.760603+0000","last_peered":"2024-04-26T17:53:49.760603+0000","last_clean":"2024-04-26T17:53:49.760603+0000","last_became_active":"2024-04-26T17:53:03.610905+0000","last_became_peered":"2024-04-26T17:53:03.610905+0000","last_unstale":"2024-04-26T17:53:49.760603+0000","last_undegraded":"2024-04-26T17:53:49.760603+0000","last_fullsized":"2024-04-26T17:53:49.760603+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:31.426237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'11","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.762463+0000","last_change":"2024-04-26T17:53:00.580220+0000","last_active":"2024-04-26T17:53:49.762463+0000","last_peered":"2024-04-26T17:53:49.762463+0000","last_clean":"2024-04-26T17:53:49.762463+0000","last_became_active":"2024-04-26T17:53:00.579818+0000","last_became_peered":"2024-04-26T17:53:00.579818+0000","last_unstale":"2024-04-26T17:53:49.762463+0000","last_undegraded":"2024-04-26T17:53:49.762463+0000","last_fullsized":"2024-04-26T17:53:49.762463+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:15.260489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765821+0000","last_change":"2024-04-26T17:53:03.610882+0000","last_active":"2024-04-26T17:53:49.765821+0000","last_peered":"2024-04-26T17:53:49.765821+0000","last_clean":"2024-04-26T17:53:49.765821+0000","last_became_active":"2024-04-26T17:53:03.609979+0000","last_became_peered":"2024-04-26T17:53:03.609979+0000","last_unstale":"2024-04-26T17:53:49.765821+0000","last_undegraded":"2024-04-26T17:53:49.765821+0000","last_fullsized":"2024-04-26T17:53:49.765821+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:33.167579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891536+0000","last_change":"2024-04-26T17:53:01.358555+0000","last_active":"2024-04-26T17:53:49.891536+0000","last_peered":"2024-04-26T17:53:49.891536+0000","last_clean":"2024-04-26T17:53:49.891536+0000","last_became_active":"2024-04-26T17:53:01.358138+0000","last_became_peered":"2024-04-26T17:53:01.358138+0000","last_unstale":"2024-04-26T17:53:49.891536+0000","last_undegraded":"2024-04-26T17:53:49.891536+0000","last_fullsized":"2024-04-26T17:53:49.891536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:14.886609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.766115+0000","last_change":"2024-04-26T17:53:03.618816+0000","last_active":"2024-04-26T17:53:49.766115+0000","last_peered":"2024-04-26T17:53:49.766115+0000","last_clean":"2024-04-26T17:53:49.766115+0000","last_became_active":"2024-04-26T17:53:03.618184+0000","last_became_peered":"2024-04-26T17:53:03.618184+0000","last_unstale":"2024-04-26T17:53:49.766115+0000","last_undegraded":"2024-04-26T17:53:49.766115+0000","last_fullsized":"2024-04-26T17:53:49.766115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:07.514516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758599+0000","last_change":"2024-04-26T17:53:01.147081+0000","last_active":"2024-04-26T17:53:49.758599+0000","last_peered":"2024-04-26T17:53:49.758599+0000","last_clean":"2024-04-26T17:53:49.758599+0000","last_became_active":"2024-04-26T17:53:01.146698+0000","last_became_peered":"2024-04-26T17:53:01.146698+0000","last_unstale":"2024-04-26T17:53:49.758599+0000","last_undegraded":"2024-04-26T17:53:49.758599+0000","last_fullsized":"2024-04-26T17:53:49.758599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:44:19.927316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767053+0000","last_change":"2024-04-26T17:53:03.612592+0000","last_active":"2024-04-26T17:53:49.767053+0000","last_peered":"2024-04-26T17:53:49.767053+0000","last_clean":"2024-04-26T17:53:49.767053+0000","last_became_active":"2024-04-26T17:53:03.612274+0000","last_became_peered":"2024-04-26T17:53:03.612274+0000","last_unstale":"2024-04-26T17:53:49.767053+0000","last_undegraded":"2024-04-26T17:53:49.767053+0000","last_fullsized":"2024-04-26T17:53:49.767053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:30:48.461947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764834+0000","last_change":"2024-04-26T17:53:00.586430+0000","last_active":"2024-04-26T17:53:49.764834+0000","last_peered":"2024-04-26T17:53:49.764834+0000","last_clean":"2024-04-26T17:53:49.764834+0000","last_became_active":"2024-04-26T17:53:00.586004+0000","last_became_peered":"2024-04-26T17:53:00.586004+0000","last_unstale":"2024-04-26T17:53:49.764834+0000","last_undegraded":"2024-04-26T17:53:49.764834+0000","last_fullsized":"2024-04-26T17:53:49.764834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:47:18.224890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764726+0000","last_change":"2024-04-26T17:53:03.623081+0000","last_active":"2024-04-26T17:53:49.764726+0000","last_peered":"2024-04-26T17:53:49.764726+0000","last_clean":"2024-04-26T17:53:49.764726+0000","last_became_active":"2024-04-26T17:53:03.622761+0000","last_became_peered":"2024-04-26T17:53:03.622761+0000","last_unstale":"2024-04-26T17:53:49.764726+0000","last_undegraded":"2024-04-26T17:53:49.764726+0000","last_fullsized":"2024-04-26T17:53:49.764726+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:39.520075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768849+0000","last_change":"2024-04-26T17:53:00.583013+0000","last_active":"2024-04-26T17:53:49.768849+0000","last_peered":"2024-04-26T17:53:49.768849+0000","last_clean":"2024-04-26T17:53:49.768849+0000","last_became_active":"2024-04-26T17:53:00.582653+0000","last_became_peered":"2024-04-26T17:53:00.582653+0000","last_unstale":"2024-04-26T17:53:49.768849+0000","last_undegraded":"2024-04-26T17:53:49.768849+0000","last_fullsized":"2024-04-26T17:53:49.768849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:09.448621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765042+0000","last_change":"2024-04-26T17:53:03.618232+0000","last_active":"2024-04-26T17:53:49.765042+0000","last_peered":"2024-04-26T17:53:49.765042+0000","last_clean":"2024-04-26T17:53:49.765042+0000","last_became_active":"2024-04-26T17:53:03.617906+0000","last_became_peered":"2024-04-26T17:53:03.617906+0000","last_unstale":"2024-04-26T17:53:49.765042+0000","last_undegraded":"2024-04-26T17:53:49.765042+0000","last_fullsized":"2024-04-26T17:53:49.765042+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:17.357398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768301+0000","last_change":"2024-04-26T17:53:00.586565+0000","last_active":"2024-04-26T17:53:49.768301+0000","last_peered":"2024-04-26T17:53:49.768301+0000","last_clean":"2024-04-26T17:53:49.768301+0000","last_became_active":"2024-04-26T17:53:00.586161+0000","last_became_peered":"2024-04-26T17:53:00.586161+0000","last_unstale":"2024-04-26T17:53:49.768301+0000","last_undegraded":"2024-04-26T17:53:49.768301+0000","last_fullsized":"2024-04-26T17:53:49.768301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:56:25.294055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767400+0000","last_change":"2024-04-26T17:53:03.620340+0000","last_active":"2024-04-26T17:53:49.767400+0000","last_peered":"2024-04-26T17:53:49.767400+0000","last_clean":"2024-04-26T17:53:49.767400+0000","last_became_active":"2024-04-26T17:53:03.619897+0000","last_became_peered":"2024-04-26T17:53:03.619897+0000","last_unstale":"2024-04-26T17:53:49.767400+0000","last_undegraded":"2024-04-26T17:53:49.767400+0000","last_fullsized":"2024-04-26T17:53:49.767400+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:12:40.662070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769496+0000","last_change":"2024-04-26T17:53:01.358131+0000","last_active":"2024-04-26T17:53:49.769496+0000","last_peered":"2024-04-26T17:53:49.769496+0000","last_clean":"2024-04-26T17:53:49.769496+0000","last_became_active":"2024-04-26T17:53:01.357796+0000","last_became_peered":"2024-04-26T17:53:01.357796+0000","last_unstale":"2024-04-26T17:53:49.769496+0000","last_undegraded":"2024-04-26T17:53:49.769496+0000","last_fullsized":"2024-04-26T17:53:49.769496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:47.849873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767708+0000","last_change":"2024-04-26T17:53:03.616133+0000","last_active":"2024-04-26T17:53:49.767708+0000","last_peered":"2024-04-26T17:53:49.767708+0000","last_clean":"2024-04-26T17:53:49.767708+0000","last_became_active":"2024-04-26T17:53:03.615797+0000","last_became_peered":"2024-04-26T17:53:03.615797+0000","last_unstale":"2024-04-26T17:53:49.767708+0000","last_undegraded":"2024-04-26T17:53:49.767708+0000","last_fullsized":"2024-04-26T17:53:49.767708+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:05.698489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759241+0000","last_change":"2024-04-26T17:53:00.587696+0000","last_active":"2024-04-26T17:53:49.759241+0000","last_peered":"2024-04-26T17:53:49.759241+0000","last_clean":"2024-04-26T17:53:49.759241+0000","last_became_active":"2024-04-26T17:53:00.587305+0000","last_became_peered":"2024-04-26T17:53:00.587305+0000","last_unstale":"2024-04-26T17:53:49.759241+0000","last_undegraded":"2024-04-26T17:53:49.759241+0000","last_fullsized":"2024-04-26T17:53:49.759241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:39.185753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767528+0000","last_change":"2024-04-26T17:53:03.613076+0000","last_active":"2024-04-26T17:53:49.767528+0000","last_peered":"2024-04-26T17:53:49.767528+0000","last_clean":"2024-04-26T17:53:49.767528+0000","last_became_active":"2024-04-26T17:53:03.612746+0000","last_became_peered":"2024-04-26T17:53:03.612746+0000","last_unstale":"2024-04-26T17:53:49.767528+0000","last_undegraded":"2024-04-26T17:53:49.767528+0000","last_fullsized":"2024-04-26T17:53:49.767528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:20.920154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760620+0000","last_change":"2024-04-26T17:53:00.582575+0000","last_active":"2024-04-26T17:53:49.760620+0000","last_peered":"2024-04-26T17:53:49.760620+0000","last_clean":"2024-04-26T17:53:49.760620+0000","last_became_active":"2024-04-26T17:53:00.582215+0000","last_became_peered":"2024-04-26T17:53:00.582215+0000","last_unstale":"2024-04-26T17:53:49.760620+0000","last_undegraded":"2024-04-26T17:53:49.760620+0000","last_fullsized":"2024-04-26T17:53:49.760620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:35:12.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767871+0000","last_change":"2024-04-26T17:53:03.616269+0000","last_active":"2024-04-26T17:53:49.767871+0000","last_peered":"2024-04-26T17:53:49.767871+0000","last_clean":"2024-04-26T17:53:49.767871+0000","last_became_active":"2024-04-26T17:53:03.615896+0000","last_became_peered":"2024-04-26T17:53:03.615896+0000","last_unstale":"2024-04-26T17:53:49.767871+0000","last_undegraded":"2024-04-26T17:53:49.767871+0000","last_fullsized":"2024-04-26T17:53:49.767871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:12.111392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764802+0000","last_change":"2024-04-26T17:53:03.621589+0000","last_active":"2024-04-26T17:53:49.764802+0000","last_peered":"2024-04-26T17:53:49.764802+0000","last_clean":"2024-04-26T17:53:49.764802+0000","last_became_active":"2024-04-26T17:53:03.621211+0000","last_became_peered":"2024-04-26T17:53:03.621211+0000","last_unstale":"2024-04-26T17:53:49.764802+0000","last_undegraded":"2024-04-26T17:53:49.764802+0000","last_fullsized":"2024-04-26T17:53:49.764802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.487476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768626+0000","last_change":"2024-04-26T17:53:03.624536+0000","last_active":"2024-04-26T17:53:49.768626+0000","last_peered":"2024-04-26T17:53:49.768626+0000","last_clean":"2024-04-26T17:53:49.768626+0000","last_became_active":"2024-04-26T17:53:03.624234+0000","last_became_peered":"2024-04-26T17:53:03.624234+0000","last_unstale":"2024-04-26T17:53:49.768626+0000","last_undegraded":"2024-04-26T17:53:49.768626+0000","last_fullsized":"2024-04-26T17:53:49.768626+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:31:36.322688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768748+0000","last_change":"2024-04-26T17:53:03.620096+0000","last_active":"2024-04-26T17:53:49.768748+0000","last_peered":"2024-04-26T17:53:49.768748+0000","last_clean":"2024-04-26T17:53:49.768748+0000","last_became_active":"2024-04-26T17:53:03.619729+0000","last_became_peered":"2024-04-26T17:53:03.619729+0000","last_unstale":"2024-04-26T17:53:49.768748+0000","last_undegraded":"2024-04-26T17:53:49.768748+0000","last_fullsized":"2024-04-26T17:53:49.768748+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:17.987728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768829+0000","last_change":"2024-04-26T17:53:49.768829+0000","last_active":"2024-04-26T17:53:49.768829+0000","last_peered":"2024-04-26T17:53:49.768829+0000","last_clean":"2024-04-26T17:53:49.768829+0000","last_became_active":"2024-04-26T17:53:49.768415+0000","last_became_peered":"2024-04-26T17:53:49.768415+0000","last_unstale":"2024-04-26T17:53:49.768829+0000","last_undegraded":"2024-04-26T17:53:49.768829+0000","last_fullsized":"2024-04-26T17:53:49.768829+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:57.364355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760497+0000","last_change":"2024-04-26T17:53:03.619193+0000","last_active":"2024-04-26T17:53:49.760497+0000","last_peered":"2024-04-26T17:53:49.760497+0000","last_clean":"2024-04-26T17:53:49.760497+0000","last_became_active":"2024-04-26T17:53:03.618776+0000","last_became_peered":"2024-04-26T17:53:03.618776+0000","last_unstale":"2024-04-26T17:53:49.760497+0000","last_undegraded":"2024-04-26T17:53:49.760497+0000","last_fullsized":"2024-04-26T17:53:49.760497+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:07:24.811683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765044+0000","last_change":"2024-04-26T17:53:03.621366+0000","last_active":"2024-04-26T17:53:49.765044+0000","last_peered":"2024-04-26T17:53:49.765044+0000","last_clean":"2024-04-26T17:53:49.765044+0000","last_became_active":"2024-04-26T17:53:03.621048+0000","last_became_peered":"2024-04-26T17:53:03.621048+0000","last_unstale":"2024-04-26T17:53:49.765044+0000","last_undegraded":"2024-04-26T17:53:49.765044+0000","last_fullsized":"2024-04-26T17:53:49.765044+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:31:06.484880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769806+0000","last_change":"2024-04-26T17:53:03.619383+0000","last_active":"2024-04-26T17:53:49.769806+0000","last_peered":"2024-04-26T17:53:49.769806+0000","last_clean":"2024-04-26T17:53:49.769806+0000","last_became_active":"2024-04-26T17:53:03.619017+0000","last_became_peered":"2024-04-26T17:53:03.619017+0000","last_unstale":"2024-04-26T17:53:49.769806+0000","last_undegraded":"2024-04-26T17:53:49.769806+0000","last_fullsized":"2024-04-26T17:53:49.769806+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:20.091959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.770695+0000","last_change":"2024-04-26T17:53:49.770695+0000","last_active":"2024-04-26T17:53:49.770695+0000","last_peered":"2024-04-26T17:53:49.770695+0000","last_clean":"2024-04-26T17:53:49.770695+0000","last_became_active":"2024-04-26T17:53:49.770303+0000","last_became_peered":"2024-04-26T17:53:49.770303+0000","last_unstale":"2024-04-26T17:53:49.770695+0000","last_undegraded":"2024-04-26T17:53:49.770695+0000","last_fullsized":"2024-04-26T17:53:49.770695+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:22.830129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758969+0000","last_change":"2024-04-26T17:53:03.605521+0000","last_active":"2024-04-26T17:53:49.758969+0000","last_peered":"2024-04-26T17:53:49.758969+0000","last_clean":"2024-04-26T17:53:49.758969+0000","last_became_active":"2024-04-26T17:53:03.605145+0000","last_became_peered":"2024-04-26T17:53:03.605145+0000","last_unstale":"2024-04-26T17:53:49.758969+0000","last_undegraded":"2024-04-26T17:53:49.758969+0000","last_fullsized":"2024-04-26T17:53:49.758969+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:58:09.856021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764723+0000","last_change":"2024-04-26T17:53:03.610866+0000","last_active":"2024-04-26T17:53:49.764723+0000","last_peered":"2024-04-26T17:53:49.764723+0000","last_clean":"2024-04-26T17:53:49.764723+0000","last_became_active":"2024-04-26T17:53:03.610326+0000","last_became_peered":"2024-04-26T17:53:03.610326+0000","last_unstale":"2024-04-26T17:53:49.764723+0000","last_undegraded":"2024-04-26T17:53:49.764723+0000","last_fullsized":"2024-04-26T17:53:49.764723+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:20.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758767+0000","last_change":"2024-04-26T17:53:03.615336+0000","last_active":"2024-04-26T17:53:49.758767+0000","last_peered":"2024-04-26T17:53:49.758767+0000","last_clean":"2024-04-26T17:53:49.758767+0000","last_became_active":"2024-04-26T17:53:03.614447+0000","last_became_peered":"2024-04-26T17:53:03.614447+0000","last_unstale":"2024-04-26T17:53:49.758767+0000","last_undegraded":"2024-04-26T17:53:49.758767+0000","last_fullsized":"2024-04-26T17:53:49.758767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:36:10.091929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760835+0000","last_change":"2024-04-26T17:53:03.614863+0000","last_active":"2024-04-26T17:53:49.760835+0000","last_peered":"2024-04-26T17:53:49.760835+0000","last_clean":"2024-04-26T17:53:49.760835+0000","last_became_active":"2024-04-26T17:53:03.614520+0000","last_became_peered":"2024-04-26T17:53:03.614520+0000","last_unstale":"2024-04-26T17:53:49.760835+0000","last_undegraded":"2024-04-26T17:53:49.760835+0000","last_fullsized":"2024-04-26T17:53:49.760835+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:51.367785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764960+0000","last_change":"2024-04-26T17:53:03.618521+0000","last_active":"2024-04-26T17:53:49.764960+0000","last_peered":"2024-04-26T17:53:49.764960+0000","last_clean":"2024-04-26T17:53:49.764960+0000","last_became_active":"2024-04-26T17:53:03.617658+0000","last_became_peered":"2024-04-26T17:53:03.617658+0000","last_unstale":"2024-04-26T17:53:49.764960+0000","last_undegraded":"2024-04-26T17:53:49.764960+0000","last_fullsized":"2024-04-26T17:53:49.764960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:39.813098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280045,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66336,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":233,"num_read_kb":188,"num_write":165,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":199008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383706,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":479802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645343,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":1133618,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808868,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":480038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070504,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":479958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332142,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":1071888,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495668,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":983040,"data_stored":479427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":541555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":480698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":543296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":999424,"data_stored":479738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920829,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":479864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757304,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1056768,"data_stored":1071572,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":63913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T17:54:03.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:02 smithi031 ceph-mon[29164]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:54:03.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:02 smithi097 ceph-mon[32419]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:54:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:02 smithi186 ceph-mon[32797]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:54:04.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:03 smithi031 ceph-mon[29164]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 7 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T17:54:04.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:04.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:03 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:04.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:03 smithi097 ceph-mon[32419]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 7 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T17:54:04.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:04.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:03 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:04.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:03 smithi186 ceph-mon[32797]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 7 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T17:54:04.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:04.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:03 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:05.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:04 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:54:05.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:04 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:54:05.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:04 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:05.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:04 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:54:05.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:04 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:54:05.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:04 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:05.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:04 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:54:05.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:04 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:54:05.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:04 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:54:05.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T17:54:05.947 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:06.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:06 smithi031 ceph-mon[29164]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-04-26T17:54:06.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:06 smithi097 ceph-mon[32419]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-04-26T17:54:06.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:06 smithi186 ceph-mon[32797]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-04-26T17:54:07.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.657+0000 7fb2e62f9700 1 -- 172.21.15.31:0/3968793959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2e01025a0 msgr2=0x7fb2e0102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:07.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.657+0000 7fb2e62f9700 1 --2- 172.21.15.31:0/3968793959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2e01025a0 0x7fb2e0102980 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80099c0 tx=0x7fb2c8009cd0 comp rx=0 tx=0).stop 2024-04-26T17:54:07.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.657+0000 7fb2e62f9700 1 -- 172.21.15.31:0/3968793959 shutdown_connections 2024-04-26T17:54:07.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.657+0000 7fb2e62f9700 1 --2- 172.21.15.31:0/3968793959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2e0107680 0x7fb2e010f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:07.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.657+0000 7fb2e62f9700 1 --2- 172.21.15.31:0/3968793959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2e01025a0 0x7fb2e0102980 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:07.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.657+0000 7fb2e62f9700 1 --2- 172.21.15.31:0/3968793959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e0102f50 0x7fb2e0106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:07.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.657+0000 7fb2e62f9700 1 -- 172.21.15.31:0/3968793959 >> 172.21.15.31:0/3968793959 conn(0x7fb2e00793c0 msgr2=0x7fb2e00797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:07.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.657+0000 7fb2e62f9700 1 -- 172.21.15.31:0/3968793959 shutdown_connections 2024-04-26T17:54:07.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.658+0000 7fb2e62f9700 1 -- 172.21.15.31:0/3968793959 wait complete. 2024-04-26T17:54:07.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.658+0000 7fb2e62f9700 1 Processor -- start 2024-04-26T17:54:07.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2e62f9700 1 -- start start 2024-04-26T17:54:07.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2e62f9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2e01025a0 0x7fb2e019bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:07.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2e62f9700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e0102f50 0x7fb2e019c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:07.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2e62f9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2e0107680 0x7fb2e01a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:07.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2e62f9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e0112560 con 0x7fb2e01025a0 2024-04-26T17:54:07.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2e62f9700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01123e0 con 0x7fb2e0102f50 2024-04-26T17:54:07.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2e62f9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01126e0 con 0x7fb2e0107680 2024-04-26T17:54:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2dffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2e01025a0 0x7fb2e019bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2dffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2e01025a0 0x7fb2e019bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57598/0 (socket says 172.21.15.31:57598) 2024-04-26T17:54:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2dffff700 1 -- 172.21.15.31:0/4016353192 learned_addr learned my addr 172.21.15.31:0/4016353192 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2df7fe700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e0102f50 0x7fb2e019c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2e4896700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2e0107680 0x7fb2e01a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2dffff700 1 -- 172.21.15.31:0/4016353192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2e0107680 msgr2=0x7fb2e01a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2dffff700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2e0107680 0x7fb2e01a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2dffff700 1 -- 172.21.15.31:0/4016353192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e0102f50 msgr2=0x7fb2e019c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2dffff700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e0102f50 0x7fb2e019c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2dffff700 1 -- 172.21.15.31:0/4016353192 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e01a0db0 con 0x7fb2e01025a0 2024-04-26T17:54:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.659+0000 7fb2e4896700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2e0107680 0x7fb2e01a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:54:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.660+0000 7fb2df7fe700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e0102f50 0x7fb2e019c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:54:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.660+0000 7fb2dffff700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2e01025a0 0x7fb2e019bda0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009990 tx=0x7fb2c80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.660+0000 7fb2dd7fa700 1 -- 172.21.15.31:0/4016353192 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8027070 con 0x7fb2e01025a0 2024-04-26T17:54:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.660+0000 7fb2dd7fa700 1 -- 172.21.15.31:0/4016353192 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c8004020 con 0x7fb2e01025a0 2024-04-26T17:54:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.660+0000 7fb2dd7fa700 1 -- 172.21.15.31:0/4016353192 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8022db0 con 0x7fb2e01025a0 2024-04-26T17:54:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.660+0000 7fb2e62f9700 1 -- 172.21.15.31:0/4016353192 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e01a1040 con 0x7fb2e01025a0 2024-04-26T17:54:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.660+0000 7fb2e62f9700 1 -- 172.21.15.31:0/4016353192 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e01a8920 con 0x7fb2e01025a0 2024-04-26T17:54:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.661+0000 7fb2e62f9700 1 -- 172.21.15.31:0/4016353192 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2e006a110 con 0x7fb2e01025a0 2024-04-26T17:54:07.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.662+0000 7fb2dd7fa700 1 -- 172.21.15.31:0/4016353192 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb2c802a070 con 0x7fb2e01025a0 2024-04-26T17:54:07.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.663+0000 7fb2dd7fa700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb2cc0a3d50 0x7fb2cc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:07.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.663+0000 7fb2df7fe700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb2cc0a3d50 0x7fb2cc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:07.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.663+0000 7fb2dd7fa700 1 -- 172.21.15.31:0/4016353192 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fb2c80d12a0 con 0x7fb2e01025a0 2024-04-26T17:54:07.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.664+0000 7fb2df7fe700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb2cc0a3d50 0x7fb2cc0a6210 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0005950 tx=0x7fb2d00058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:07.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.668+0000 7fb2dd7fa700 1 -- 172.21.15.31:0/4016353192 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2c805eff0 con 0x7fb2e01025a0 2024-04-26T17:54:07.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.878+0000 7fb2e62f9700 1 -- 172.21.15.31:0/4016353192 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb2e019d100 con 0x7fb2cc0a3d50 2024-04-26T17:54:07.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.885+0000 7fb2dd7fa700 1 -- 172.21.15.31:0/4016353192 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258886 (secure 0 0 0) 0x7fb2e019d100 con 0x7fb2cc0a3d50 2024-04-26T17:54:07.888 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:54:07.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.888+0000 7fb2e62f9700 1 -- 172.21.15.31:0/4016353192 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb2cc0a3d50 msgr2=0x7fb2cc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:07.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.888+0000 7fb2e62f9700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb2cc0a3d50 0x7fb2cc0a6210 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0005950 tx=0x7fb2d00058e0 comp rx=0 tx=0).stop 2024-04-26T17:54:07.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.888+0000 7fb2e62f9700 1 -- 172.21.15.31:0/4016353192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2e01025a0 msgr2=0x7fb2e019bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:07.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.888+0000 7fb2e62f9700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2e01025a0 0x7fb2e019bda0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009990 tx=0x7fb2c80047b0 comp rx=0 tx=0).stop 2024-04-26T17:54:07.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.888+0000 7fb2e62f9700 1 -- 172.21.15.31:0/4016353192 shutdown_connections 2024-04-26T17:54:07.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.888+0000 7fb2e62f9700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb2cc0a3d50 0x7fb2cc0a6210 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:07.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.888+0000 7fb2e62f9700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2e0107680 0x7fb2e01a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:07.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.888+0000 7fb2e62f9700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2e01025a0 0x7fb2e019bda0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:07.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.888+0000 7fb2e62f9700 1 --2- 172.21.15.31:0/4016353192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e0102f50 0x7fb2e019c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:07.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.888+0000 7fb2e62f9700 1 -- 172.21.15.31:0/4016353192 >> 172.21.15.31:0/4016353192 conn(0x7fb2e00793c0 msgr2=0x7fb2e01009b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:07.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.888+0000 7fb2e62f9700 1 -- 172.21.15.31:0/4016353192 shutdown_connections 2024-04-26T17:54:07.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:07.888+0000 7fb2e62f9700 1 -- 172.21.15.31:0/4016353192 wait complete. 2024-04-26T17:54:07.891 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-26T17:54:08.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:08 smithi031 ceph-mon[29164]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T17:54:08.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:08 smithi097 ceph-mon[32419]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T17:54:08.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:08 smithi186 ceph-mon[32797]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T17:54:08.556 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-04-26T17:54:06.321542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656938,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":419,"num_read_kb":332,"num_write":348,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361212,"kb_used_data":12012,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580612,"statfs":{"total":1151940427776,"available":1151570546688,"internally_reserved":0,"allocated":12300288,"data_stored":7721454,"data_compressed":106863,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":45,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003102"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765218+0000","last_change":"2024-04-26T17:53:03.616592+0000","last_active":"2024-04-26T17:53:49.765218+0000","last_peered":"2024-04-26T17:53:49.765218+0000","last_clean":"2024-04-26T17:53:49.765218+0000","last_became_active":"2024-04-26T17:53:03.616255+0000","last_became_peered":"2024-04-26T17:53:03.616255+0000","last_unstale":"2024-04-26T17:53:49.765218+0000","last_undegraded":"2024-04-26T17:53:49.765218+0000","last_fullsized":"2024-04-26T17:53:49.765218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:43:31.183679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765248+0000","last_change":"2024-04-26T17:53:03.615402+0000","last_active":"2024-04-26T17:53:49.765248+0000","last_peered":"2024-04-26T17:53:49.765248+0000","last_clean":"2024-04-26T17:53:49.765248+0000","last_became_active":"2024-04-26T17:53:03.615048+0000","last_became_peered":"2024-04-26T17:53:03.615048+0000","last_unstale":"2024-04-26T17:53:49.765248+0000","last_undegraded":"2024-04-26T17:53:49.765248+0000","last_fullsized":"2024-04-26T17:53:49.765248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:05:12.454770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765118+0000","last_change":"2024-04-26T17:53:03.622625+0000","last_active":"2024-04-26T17:53:49.765118+0000","last_peered":"2024-04-26T17:53:49.765118+0000","last_clean":"2024-04-26T17:53:49.765118+0000","last_became_active":"2024-04-26T17:53:03.622305+0000","last_became_peered":"2024-04-26T17:53:03.622305+0000","last_unstale":"2024-04-26T17:53:49.765118+0000","last_undegraded":"2024-04-26T17:53:49.765118+0000","last_fullsized":"2024-04-26T17:53:49.765118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:06.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891229+0000","last_change":"2024-04-26T17:53:03.617650+0000","last_active":"2024-04-26T17:53:49.891229+0000","last_peered":"2024-04-26T17:53:49.891229+0000","last_clean":"2024-04-26T17:53:49.891229+0000","last_became_active":"2024-04-26T17:53:03.617220+0000","last_became_peered":"2024-04-26T17:53:03.617220+0000","last_unstale":"2024-04-26T17:53:49.891229+0000","last_undegraded":"2024-04-26T17:53:49.891229+0000","last_fullsized":"2024-04-26T17:53:49.891229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:31.765577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767329+0000","last_change":"2024-04-26T17:53:03.615429+0000","last_active":"2024-04-26T17:53:49.767329+0000","last_peered":"2024-04-26T17:53:49.767329+0000","last_clean":"2024-04-26T17:53:49.767329+0000","last_became_active":"2024-04-26T17:53:03.615079+0000","last_became_peered":"2024-04-26T17:53:03.615079+0000","last_unstale":"2024-04-26T17:53:49.767329+0000","last_undegraded":"2024-04-26T17:53:49.767329+0000","last_fullsized":"2024-04-26T17:53:49.767329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:20.994016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764669+0000","last_change":"2024-04-26T17:53:03.624619+0000","last_active":"2024-04-26T17:53:49.764669+0000","last_peered":"2024-04-26T17:53:49.764669+0000","last_clean":"2024-04-26T17:53:49.764669+0000","last_became_active":"2024-04-26T17:53:03.624251+0000","last_became_peered":"2024-04-26T17:53:03.624251+0000","last_unstale":"2024-04-26T17:53:49.764669+0000","last_undegraded":"2024-04-26T17:53:49.764669+0000","last_fullsized":"2024-04-26T17:53:49.764669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:50:17.535235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767186+0000","last_change":"2024-04-26T17:53:03.619325+0000","last_active":"2024-04-26T17:53:49.767186+0000","last_peered":"2024-04-26T17:53:49.767186+0000","last_clean":"2024-04-26T17:53:49.767186+0000","last_became_active":"2024-04-26T17:53:03.618989+0000","last_became_peered":"2024-04-26T17:53:03.618989+0000","last_unstale":"2024-04-26T17:53:49.767186+0000","last_undegraded":"2024-04-26T17:53:49.767186+0000","last_fullsized":"2024-04-26T17:53:49.767186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:10.195742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764403+0000","last_change":"2024-04-26T17:53:03.609346+0000","last_active":"2024-04-26T17:53:49.764403+0000","last_peered":"2024-04-26T17:53:49.764403+0000","last_clean":"2024-04-26T17:53:49.764403+0000","last_became_active":"2024-04-26T17:53:03.609021+0000","last_became_peered":"2024-04-26T17:53:03.609021+0000","last_unstale":"2024-04-26T17:53:49.764403+0000","last_undegraded":"2024-04-26T17:53:49.764403+0000","last_fullsized":"2024-04-26T17:53:49.764403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:44.328599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759099+0000","last_change":"2024-04-26T17:53:03.618573+0000","last_active":"2024-04-26T17:53:49.759099+0000","last_peered":"2024-04-26T17:53:49.759099+0000","last_clean":"2024-04-26T17:53:49.759099+0000","last_became_active":"2024-04-26T17:53:03.618223+0000","last_became_peered":"2024-04-26T17:53:03.618223+0000","last_unstale":"2024-04-26T17:53:49.759099+0000","last_undegraded":"2024-04-26T17:53:49.759099+0000","last_fullsized":"2024-04-26T17:53:49.759099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:16.749205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765013+0000","last_change":"2024-04-26T17:53:03.621530+0000","last_active":"2024-04-26T17:53:49.765013+0000","last_peered":"2024-04-26T17:53:49.765013+0000","last_clean":"2024-04-26T17:53:49.765013+0000","last_became_active":"2024-04-26T17:53:03.621174+0000","last_became_peered":"2024-04-26T17:53:03.621174+0000","last_unstale":"2024-04-26T17:53:49.765013+0000","last_undegraded":"2024-04-26T17:53:49.765013+0000","last_fullsized":"2024-04-26T17:53:49.765013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:31.388232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767569+0000","last_change":"2024-04-26T17:53:03.617682+0000","last_active":"2024-04-26T17:53:49.767569+0000","last_peered":"2024-04-26T17:53:49.767569+0000","last_clean":"2024-04-26T17:53:49.767569+0000","last_became_active":"2024-04-26T17:53:03.617348+0000","last_became_peered":"2024-04-26T17:53:03.617348+0000","last_unstale":"2024-04-26T17:53:49.767569+0000","last_undegraded":"2024-04-26T17:53:49.767569+0000","last_fullsized":"2024-04-26T17:53:49.767569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:21.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891137+0000","last_change":"2024-04-26T17:53:03.617955+0000","last_active":"2024-04-26T17:53:49.891137+0000","last_peered":"2024-04-26T17:53:49.891137+0000","last_clean":"2024-04-26T17:53:49.891137+0000","last_became_active":"2024-04-26T17:53:03.616765+0000","last_became_peered":"2024-04-26T17:53:03.616765+0000","last_unstale":"2024-04-26T17:53:49.891137+0000","last_undegraded":"2024-04-26T17:53:49.891137+0000","last_fullsized":"2024-04-26T17:53:49.891137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:59.292277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767807+0000","last_change":"2024-04-26T17:53:49.767807+0000","last_active":"2024-04-26T17:53:49.767807+0000","last_peered":"2024-04-26T17:53:49.767807+0000","last_clean":"2024-04-26T17:53:49.767807+0000","last_became_active":"2024-04-26T17:53:49.767404+0000","last_became_peered":"2024-04-26T17:53:49.767404+0000","last_unstale":"2024-04-26T17:53:49.767807+0000","last_undegraded":"2024-04-26T17:53:49.767807+0000","last_fullsized":"2024-04-26T17:53:49.767807+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:20:45.482580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764992+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:53:49.764992+0000","last_peered":"2024-04-26T17:53:49.764992+0000","last_clean":"2024-04-26T17:53:49.764992+0000","last_became_active":"2024-04-26T17:53:03.616623+0000","last_became_peered":"2024-04-26T17:53:03.616623+0000","last_unstale":"2024-04-26T17:53:49.764992+0000","last_undegraded":"2024-04-26T17:53:49.764992+0000","last_fullsized":"2024-04-26T17:53:49.764992+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:04.231549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764782+0000","last_change":"2024-04-26T17:53:03.622147+0000","last_active":"2024-04-26T17:53:49.764782+0000","last_peered":"2024-04-26T17:53:49.764782+0000","last_clean":"2024-04-26T17:53:49.764782+0000","last_became_active":"2024-04-26T17:53:03.621542+0000","last_became_peered":"2024-04-26T17:53:03.621542+0000","last_unstale":"2024-04-26T17:53:49.764782+0000","last_undegraded":"2024-04-26T17:53:49.764782+0000","last_fullsized":"2024-04-26T17:53:49.764782+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:01.746954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758591+0000","last_change":"2024-04-26T17:53:03.624433+0000","last_active":"2024-04-26T17:53:49.758591+0000","last_peered":"2024-04-26T17:53:49.758591+0000","last_clean":"2024-04-26T17:53:49.758591+0000","last_became_active":"2024-04-26T17:53:03.624135+0000","last_became_peered":"2024-04-26T17:53:03.624135+0000","last_unstale":"2024-04-26T17:53:49.758591+0000","last_undegraded":"2024-04-26T17:53:49.758591+0000","last_fullsized":"2024-04-26T17:53:49.758591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:32:20.521917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765321+0000","last_change":"2024-04-26T17:53:03.623607+0000","last_active":"2024-04-26T17:53:49.765321+0000","last_peered":"2024-04-26T17:53:49.765321+0000","last_clean":"2024-04-26T17:53:49.765321+0000","last_became_active":"2024-04-26T17:53:03.621783+0000","last_became_peered":"2024-04-26T17:53:03.621783+0000","last_unstale":"2024-04-26T17:53:49.765321+0000","last_undegraded":"2024-04-26T17:53:49.765321+0000","last_fullsized":"2024-04-26T17:53:49.765321+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:38:59.872868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.762359+0000","last_change":"2024-04-26T17:53:03.614974+0000","last_active":"2024-04-26T17:53:49.762359+0000","last_peered":"2024-04-26T17:53:49.762359+0000","last_clean":"2024-04-26T17:53:49.762359+0000","last_became_active":"2024-04-26T17:53:03.614619+0000","last_became_peered":"2024-04-26T17:53:03.614619+0000","last_unstale":"2024-04-26T17:53:49.762359+0000","last_undegraded":"2024-04-26T17:53:49.762359+0000","last_fullsized":"2024-04-26T17:53:49.762359+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:50.729531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769611+0000","last_change":"2024-04-26T17:53:03.611140+0000","last_active":"2024-04-26T17:53:49.769611+0000","last_peered":"2024-04-26T17:53:49.769611+0000","last_clean":"2024-04-26T17:53:49.769611+0000","last_became_active":"2024-04-26T17:53:03.610549+0000","last_became_peered":"2024-04-26T17:53:03.610549+0000","last_unstale":"2024-04-26T17:53:49.769611+0000","last_undegraded":"2024-04-26T17:53:49.769611+0000","last_fullsized":"2024-04-26T17:53:49.769611+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:46:17.178598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.761817+0000","last_change":"2024-04-26T17:53:00.585971+0000","last_active":"2024-04-26T17:53:49.761817+0000","last_peered":"2024-04-26T17:53:49.761817+0000","last_clean":"2024-04-26T17:53:49.761817+0000","last_became_active":"2024-04-26T17:53:00.585648+0000","last_became_peered":"2024-04-26T17:53:00.585648+0000","last_unstale":"2024-04-26T17:53:49.761817+0000","last_undegraded":"2024-04-26T17:53:49.761817+0000","last_fullsized":"2024-04-26T17:53:49.761817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:00:24.082773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.770579+0000","last_change":"2024-04-26T17:53:49.770579+0000","last_active":"2024-04-26T17:53:49.770579+0000","last_peered":"2024-04-26T17:53:49.770579+0000","last_clean":"2024-04-26T17:53:49.770579+0000","last_became_active":"2024-04-26T17:53:49.770250+0000","last_became_peered":"2024-04-26T17:53:49.770250+0000","last_unstale":"2024-04-26T17:53:49.770579+0000","last_undegraded":"2024-04-26T17:53:49.770579+0000","last_fullsized":"2024-04-26T17:53:49.770579+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:35.758790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.766024+0000","last_change":"2024-04-26T17:53:01.145438+0000","last_active":"2024-04-26T17:53:49.766024+0000","last_peered":"2024-04-26T17:53:49.766024+0000","last_clean":"2024-04-26T17:53:49.766024+0000","last_became_active":"2024-04-26T17:53:01.145030+0000","last_became_peered":"2024-04-26T17:53:01.145030+0000","last_unstale":"2024-04-26T17:53:49.766024+0000","last_undegraded":"2024-04-26T17:53:49.766024+0000","last_fullsized":"2024-04-26T17:53:49.766024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:15.650247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765536+0000","last_change":"2024-04-26T17:53:03.622647+0000","last_active":"2024-04-26T17:53:49.765536+0000","last_peered":"2024-04-26T17:53:49.765536+0000","last_clean":"2024-04-26T17:53:49.765536+0000","last_became_active":"2024-04-26T17:53:03.622298+0000","last_became_peered":"2024-04-26T17:53:03.622298+0000","last_unstale":"2024-04-26T17:53:49.765536+0000","last_undegraded":"2024-04-26T17:53:49.765536+0000","last_fullsized":"2024-04-26T17:53:49.765536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:37.230539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.775803+0000","last_change":"2024-04-26T17:53:49.775405+0000","last_active":"2024-04-26T17:53:49.775803+0000","last_peered":"2024-04-26T17:53:49.775803+0000","last_clean":"2024-04-26T17:53:49.775803+0000","last_became_active":"2024-04-26T17:53:49.770536+0000","last_became_peered":"2024-04-26T17:53:49.770536+0000","last_unstale":"2024-04-26T17:53:49.775803+0000","last_undegraded":"2024-04-26T17:53:49.775803+0000","last_fullsized":"2024-04-26T17:53:49.775803+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:34.643452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758674+0000","last_change":"2024-04-26T17:53:03.617462+0000","last_active":"2024-04-26T17:53:49.758674+0000","last_peered":"2024-04-26T17:53:49.758674+0000","last_clean":"2024-04-26T17:53:49.758674+0000","last_became_active":"2024-04-26T17:53:03.617115+0000","last_became_peered":"2024-04-26T17:53:03.617115+0000","last_unstale":"2024-04-26T17:53:49.758674+0000","last_undegraded":"2024-04-26T17:53:49.758674+0000","last_fullsized":"2024-04-26T17:53:49.758674+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:11.406146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767810+0000","last_change":"2024-04-26T17:53:00.583378+0000","last_active":"2024-04-26T17:53:49.767810+0000","last_peered":"2024-04-26T17:53:49.767810+0000","last_clean":"2024-04-26T17:53:49.767810+0000","last_became_active":"2024-04-26T17:53:00.583020+0000","last_became_peered":"2024-04-26T17:53:00.583020+0000","last_unstale":"2024-04-26T17:53:49.767810+0000","last_undegraded":"2024-04-26T17:53:49.767810+0000","last_fullsized":"2024-04-26T17:53:49.767810+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:24.089996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759512+0000","last_change":"2024-04-26T17:53:03.609557+0000","last_active":"2024-04-26T17:53:49.759512+0000","last_peered":"2024-04-26T17:53:49.759512+0000","last_clean":"2024-04-26T17:53:49.759512+0000","last_became_active":"2024-04-26T17:53:03.609028+0000","last_became_peered":"2024-04-26T17:53:03.609028+0000","last_unstale":"2024-04-26T17:53:49.759512+0000","last_undegraded":"2024-04-26T17:53:49.759512+0000","last_fullsized":"2024-04-26T17:53:49.759512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:39:22.860877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:03.757676+0000","last_change":"2024-04-26T17:53:00.586683+0000","last_active":"2024-04-26T17:54:03.757676+0000","last_peered":"2024-04-26T17:54:03.757676+0000","last_clean":"2024-04-26T17:54:03.757676+0000","last_became_active":"2024-04-26T17:53:00.586272+0000","last_became_peered":"2024-04-26T17:53:00.586272+0000","last_unstale":"2024-04-26T17:54:03.757676+0000","last_undegraded":"2024-04-26T17:54:03.757676+0000","last_fullsized":"2024-04-26T17:54:03.757676+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:20:29.068607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767148+0000","last_change":"2024-04-26T17:53:03.611073+0000","last_active":"2024-04-26T17:53:49.767148+0000","last_peered":"2024-04-26T17:53:49.767148+0000","last_clean":"2024-04-26T17:53:49.767148+0000","last_became_active":"2024-04-26T17:53:03.610701+0000","last_became_peered":"2024-04-26T17:53:03.610701+0000","last_unstale":"2024-04-26T17:53:49.767148+0000","last_undegraded":"2024-04-26T17:53:49.767148+0000","last_fullsized":"2024-04-26T17:53:49.767148+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:01:00.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764638+0000","last_change":"2024-04-26T17:53:00.586082+0000","last_active":"2024-04-26T17:53:49.764638+0000","last_peered":"2024-04-26T17:53:49.764638+0000","last_clean":"2024-04-26T17:53:49.764638+0000","last_became_active":"2024-04-26T17:53:00.585678+0000","last_became_peered":"2024-04-26T17:53:00.585678+0000","last_unstale":"2024-04-26T17:53:49.764638+0000","last_undegraded":"2024-04-26T17:53:49.764638+0000","last_fullsized":"2024-04-26T17:53:49.764638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:55:25.523846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759383+0000","last_change":"2024-04-26T17:53:03.600230+0000","last_active":"2024-04-26T17:53:49.759383+0000","last_peered":"2024-04-26T17:53:49.759383+0000","last_clean":"2024-04-26T17:53:49.759383+0000","last_became_active":"2024-04-26T17:53:03.599713+0000","last_became_peered":"2024-04-26T17:53:03.599713+0000","last_unstale":"2024-04-26T17:53:49.759383+0000","last_undegraded":"2024-04-26T17:53:49.759383+0000","last_fullsized":"2024-04-26T17:53:49.759383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:45.822561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760523+0000","last_change":"2024-04-26T17:53:00.584588+0000","last_active":"2024-04-26T17:53:49.760523+0000","last_peered":"2024-04-26T17:53:49.760523+0000","last_clean":"2024-04-26T17:53:49.760523+0000","last_became_active":"2024-04-26T17:53:00.584284+0000","last_became_peered":"2024-04-26T17:53:00.584284+0000","last_unstale":"2024-04-26T17:53:49.760523+0000","last_undegraded":"2024-04-26T17:53:49.760523+0000","last_fullsized":"2024-04-26T17:53:49.760523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:08.428904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764864+0000","last_change":"2024-04-26T17:53:03.615237+0000","last_active":"2024-04-26T17:53:49.764864+0000","last_peered":"2024-04-26T17:53:49.764864+0000","last_clean":"2024-04-26T17:53:49.764864+0000","last_became_active":"2024-04-26T17:53:03.614704+0000","last_became_peered":"2024-04-26T17:53:03.614704+0000","last_unstale":"2024-04-26T17:53:49.764864+0000","last_undegraded":"2024-04-26T17:53:49.764864+0000","last_fullsized":"2024-04-26T17:53:49.764864+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:27.251839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758907+0000","last_change":"2024-04-26T17:53:00.589287+0000","last_active":"2024-04-26T17:53:49.758907+0000","last_peered":"2024-04-26T17:53:49.758907+0000","last_clean":"2024-04-26T17:53:49.758907+0000","last_became_active":"2024-04-26T17:53:00.588119+0000","last_became_peered":"2024-04-26T17:53:00.588119+0000","last_unstale":"2024-04-26T17:53:49.758907+0000","last_undegraded":"2024-04-26T17:53:49.758907+0000","last_fullsized":"2024-04-26T17:53:49.758907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:25.389777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.761931+0000","last_change":"2024-04-26T17:53:03.608061+0000","last_active":"2024-04-26T17:53:49.761931+0000","last_peered":"2024-04-26T17:53:49.761931+0000","last_clean":"2024-04-26T17:53:49.761931+0000","last_became_active":"2024-04-26T17:53:03.607624+0000","last_became_peered":"2024-04-26T17:53:03.607624+0000","last_unstale":"2024-04-26T17:53:49.761931+0000","last_undegraded":"2024-04-26T17:53:49.761931+0000","last_fullsized":"2024-04-26T17:53:49.761931+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:15.111011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:04.758487+0000","last_change":"2024-04-26T17:53:00.588621+0000","last_active":"2024-04-26T17:54:04.758487+0000","last_peered":"2024-04-26T17:54:04.758487+0000","last_clean":"2024-04-26T17:54:04.758487+0000","last_became_active":"2024-04-26T17:53:00.587330+0000","last_became_peered":"2024-04-26T17:53:00.587330+0000","last_unstale":"2024-04-26T17:54:04.758487+0000","last_undegraded":"2024-04-26T17:54:04.758487+0000","last_fullsized":"2024-04-26T17:54:04.758487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:01:12.178610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":44,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759017+0000","last_change":"2024-04-26T17:53:03.615723+0000","last_active":"2024-04-26T17:53:49.759017+0000","last_peered":"2024-04-26T17:53:49.759017+0000","last_clean":"2024-04-26T17:53:49.759017+0000","last_became_active":"2024-04-26T17:53:03.614931+0000","last_became_peered":"2024-04-26T17:53:03.614931+0000","last_unstale":"2024-04-26T17:53:49.759017+0000","last_undegraded":"2024-04-26T17:53:49.759017+0000","last_fullsized":"2024-04-26T17:53:49.759017+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:09.663876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767693+0000","last_change":"2024-04-26T17:53:00.589232+0000","last_active":"2024-04-26T17:53:49.767693+0000","last_peered":"2024-04-26T17:53:49.767693+0000","last_clean":"2024-04-26T17:53:49.767693+0000","last_became_active":"2024-04-26T17:53:00.588863+0000","last_became_peered":"2024-04-26T17:53:00.588863+0000","last_unstale":"2024-04-26T17:53:49.767693+0000","last_undegraded":"2024-04-26T17:53:49.767693+0000","last_fullsized":"2024-04-26T17:53:49.767693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:14.360782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765501+0000","last_change":"2024-04-26T17:53:03.624923+0000","last_active":"2024-04-26T17:53:49.765501+0000","last_peered":"2024-04-26T17:53:49.765501+0000","last_clean":"2024-04-26T17:53:49.765501+0000","last_became_active":"2024-04-26T17:53:03.624620+0000","last_became_peered":"2024-04-26T17:53:03.624620+0000","last_unstale":"2024-04-26T17:53:49.765501+0000","last_undegraded":"2024-04-26T17:53:49.765501+0000","last_fullsized":"2024-04-26T17:53:49.765501+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:50:51.259563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759526+0000","last_change":"2024-04-26T17:53:01.146950+0000","last_active":"2024-04-26T17:53:49.759526+0000","last_peered":"2024-04-26T17:53:49.759526+0000","last_clean":"2024-04-26T17:53:49.759526+0000","last_became_active":"2024-04-26T17:53:01.146591+0000","last_became_peered":"2024-04-26T17:53:01.146591+0000","last_unstale":"2024-04-26T17:53:49.759526+0000","last_undegraded":"2024-04-26T17:53:49.759526+0000","last_fullsized":"2024-04-26T17:53:49.759526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:17.756660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758787+0000","last_change":"2024-04-26T17:53:03.626944+0000","last_active":"2024-04-26T17:53:49.758787+0000","last_peered":"2024-04-26T17:53:49.758787+0000","last_clean":"2024-04-26T17:53:49.758787+0000","last_became_active":"2024-04-26T17:53:03.626594+0000","last_became_peered":"2024-04-26T17:53:03.626594+0000","last_unstale":"2024-04-26T17:53:49.758787+0000","last_undegraded":"2024-04-26T17:53:49.758787+0000","last_fullsized":"2024-04-26T17:53:49.758787+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:52:42.997251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'30","reported_seq":153,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:05.759730+0000","last_change":"2024-04-26T17:53:00.588353+0000","last_active":"2024-04-26T17:54:05.759730+0000","last_peered":"2024-04-26T17:54:05.759730+0000","last_clean":"2024-04-26T17:54:05.759730+0000","last_became_active":"2024-04-26T17:53:00.586351+0000","last_became_peered":"2024-04-26T17:53:00.586351+0000","last_unstale":"2024-04-26T17:54:05.759730+0000","last_undegraded":"2024-04-26T17:54:05.759730+0000","last_fullsized":"2024-04-26T17:54:05.759730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:41.660525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62335,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":97,"num_read_kb":109,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768625+0000","last_change":"2024-04-26T17:53:03.612052+0000","last_active":"2024-04-26T17:53:49.768625+0000","last_peered":"2024-04-26T17:53:49.768625+0000","last_clean":"2024-04-26T17:53:49.768625+0000","last_became_active":"2024-04-26T17:53:03.611733+0000","last_became_peered":"2024-04-26T17:53:03.611733+0000","last_unstale":"2024-04-26T17:53:49.768625+0000","last_undegraded":"2024-04-26T17:53:49.768625+0000","last_fullsized":"2024-04-26T17:53:49.768625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:05:23.958317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769171+0000","last_change":"2024-04-26T17:53:00.588284+0000","last_active":"2024-04-26T17:53:49.769171+0000","last_peered":"2024-04-26T17:53:49.769171+0000","last_clean":"2024-04-26T17:53:49.769171+0000","last_became_active":"2024-04-26T17:53:00.587942+0000","last_became_peered":"2024-04-26T17:53:00.587942+0000","last_unstale":"2024-04-26T17:53:49.769171+0000","last_undegraded":"2024-04-26T17:53:49.769171+0000","last_fullsized":"2024-04-26T17:53:49.769171+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:43.782971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760366+0000","last_change":"2024-04-26T17:53:03.613092+0000","last_active":"2024-04-26T17:53:49.760366+0000","last_peered":"2024-04-26T17:53:49.760366+0000","last_clean":"2024-04-26T17:53:49.760366+0000","last_became_active":"2024-04-26T17:53:03.612773+0000","last_became_peered":"2024-04-26T17:53:03.612773+0000","last_unstale":"2024-04-26T17:53:49.760366+0000","last_undegraded":"2024-04-26T17:53:49.760366+0000","last_fullsized":"2024-04-26T17:53:49.760366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:41:21.340375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759054+0000","last_change":"2024-04-26T17:53:00.588777+0000","last_active":"2024-04-26T17:53:49.759054+0000","last_peered":"2024-04-26T17:53:49.759054+0000","last_clean":"2024-04-26T17:53:49.759054+0000","last_became_active":"2024-04-26T17:53:00.588380+0000","last_became_peered":"2024-04-26T17:53:00.588380+0000","last_unstale":"2024-04-26T17:53:49.759054+0000","last_undegraded":"2024-04-26T17:53:49.759054+0000","last_fullsized":"2024-04-26T17:53:49.759054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:44.587381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767750+0000","last_change":"2024-04-26T17:53:03.620344+0000","last_active":"2024-04-26T17:53:49.767750+0000","last_peered":"2024-04-26T17:53:49.767750+0000","last_clean":"2024-04-26T17:53:49.767750+0000","last_became_active":"2024-04-26T17:53:03.619944+0000","last_became_peered":"2024-04-26T17:53:03.619944+0000","last_unstale":"2024-04-26T17:53:49.767750+0000","last_undegraded":"2024-04-26T17:53:49.767750+0000","last_fullsized":"2024-04-26T17:53:49.767750+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:08:31.801342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:59.751753+0000","last_change":"2024-04-26T17:53:00.587733+0000","last_active":"2024-04-26T17:53:59.751753+0000","last_peered":"2024-04-26T17:53:59.751753+0000","last_clean":"2024-04-26T17:53:59.751753+0000","last_became_active":"2024-04-26T17:53:00.587373+0000","last_became_peered":"2024-04-26T17:53:00.587373+0000","last_unstale":"2024-04-26T17:53:59.751753+0000","last_undegraded":"2024-04-26T17:53:59.751753+0000","last_fullsized":"2024-04-26T17:53:59.751753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:10.978944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767121+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:53:49.767121+0000","last_peered":"2024-04-26T17:53:49.767121+0000","last_clean":"2024-04-26T17:53:49.767121+0000","last_became_active":"2024-04-26T17:53:03.616588+0000","last_became_peered":"2024-04-26T17:53:03.616588+0000","last_unstale":"2024-04-26T17:53:49.767121+0000","last_undegraded":"2024-04-26T17:53:49.767121+0000","last_fullsized":"2024-04-26T17:53:49.767121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:46:38.419613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767233+0000","last_change":"2024-04-26T17:53:01.146922+0000","last_active":"2024-04-26T17:53:49.767233+0000","last_peered":"2024-04-26T17:53:49.767233+0000","last_clean":"2024-04-26T17:53:49.767233+0000","last_became_active":"2024-04-26T17:53:01.146520+0000","last_became_peered":"2024-04-26T17:53:01.146520+0000","last_unstale":"2024-04-26T17:53:49.767233+0000","last_undegraded":"2024-04-26T17:53:49.767233+0000","last_fullsized":"2024-04-26T17:53:49.767233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:12:49.238012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758964+0000","last_change":"2024-04-26T17:53:03.608513+0000","last_active":"2024-04-26T17:53:49.758964+0000","last_peered":"2024-04-26T17:53:49.758964+0000","last_clean":"2024-04-26T17:53:49.758964+0000","last_became_active":"2024-04-26T17:53:03.608167+0000","last_became_peered":"2024-04-26T17:53:03.608167+0000","last_unstale":"2024-04-26T17:53:49.758964+0000","last_undegraded":"2024-04-26T17:53:49.758964+0000","last_fullsized":"2024-04-26T17:53:49.758964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:03.002756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769191+0000","last_change":"2024-04-26T17:53:00.583878+0000","last_active":"2024-04-26T17:53:49.769191+0000","last_peered":"2024-04-26T17:53:49.769191+0000","last_clean":"2024-04-26T17:53:49.769191+0000","last_became_active":"2024-04-26T17:53:00.583159+0000","last_became_peered":"2024-04-26T17:53:00.583159+0000","last_unstale":"2024-04-26T17:53:49.769191+0000","last_undegraded":"2024-04-26T17:53:49.769191+0000","last_fullsized":"2024-04-26T17:53:49.769191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:49.850942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765106+0000","last_change":"2024-04-26T17:53:03.614467+0000","last_active":"2024-04-26T17:53:49.765106+0000","last_peered":"2024-04-26T17:53:49.765106+0000","last_clean":"2024-04-26T17:53:49.765106+0000","last_became_active":"2024-04-26T17:53:03.613972+0000","last_became_peered":"2024-04-26T17:53:03.613972+0000","last_unstale":"2024-04-26T17:53:49.765106+0000","last_undegraded":"2024-04-26T17:53:49.765106+0000","last_fullsized":"2024-04-26T17:53:49.765106+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:27.259777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767174+0000","last_change":"2024-04-26T17:53:49.767174+0000","last_active":"2024-04-26T17:53:49.767174+0000","last_peered":"2024-04-26T17:53:49.767174+0000","last_clean":"2024-04-26T17:53:49.767174+0000","last_became_active":"2024-04-26T17:53:49.766778+0000","last_became_peered":"2024-04-26T17:53:49.766778+0000","last_unstale":"2024-04-26T17:53:49.767174+0000","last_undegraded":"2024-04-26T17:53:49.767174+0000","last_fullsized":"2024-04-26T17:53:49.767174+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:59.240159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764767+0000","last_change":"2024-04-26T17:53:03.613888+0000","last_active":"2024-04-26T17:53:49.764767+0000","last_peered":"2024-04-26T17:53:49.764767+0000","last_clean":"2024-04-26T17:53:49.764767+0000","last_became_active":"2024-04-26T17:53:03.613499+0000","last_became_peered":"2024-04-26T17:53:03.613499+0000","last_unstale":"2024-04-26T17:53:49.764767+0000","last_undegraded":"2024-04-26T17:53:49.764767+0000","last_fullsized":"2024-04-26T17:53:49.764767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:15.549419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767375+0000","last_change":"2024-04-26T17:53:00.587803+0000","last_active":"2024-04-26T17:53:49.767375+0000","last_peered":"2024-04-26T17:53:49.767375+0000","last_clean":"2024-04-26T17:53:49.767375+0000","last_became_active":"2024-04-26T17:53:00.587382+0000","last_became_peered":"2024-04-26T17:53:00.587382+0000","last_unstale":"2024-04-26T17:53:49.767375+0000","last_undegraded":"2024-04-26T17:53:49.767375+0000","last_fullsized":"2024-04-26T17:53:49.767375+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:38.249478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769008+0000","last_change":"2024-04-26T17:53:49.769008+0000","last_active":"2024-04-26T17:53:49.769008+0000","last_peered":"2024-04-26T17:53:49.769008+0000","last_clean":"2024-04-26T17:53:49.769008+0000","last_became_active":"2024-04-26T17:53:49.768625+0000","last_became_peered":"2024-04-26T17:53:49.768625+0000","last_unstale":"2024-04-26T17:53:49.769008+0000","last_undegraded":"2024-04-26T17:53:49.769008+0000","last_fullsized":"2024-04-26T17:53:49.769008+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:27:46.412625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759215+0000","last_change":"2024-04-26T17:53:03.612899+0000","last_active":"2024-04-26T17:53:49.759215+0000","last_peered":"2024-04-26T17:53:49.759215+0000","last_clean":"2024-04-26T17:53:49.759215+0000","last_became_active":"2024-04-26T17:53:03.612564+0000","last_became_peered":"2024-04-26T17:53:03.612564+0000","last_unstale":"2024-04-26T17:53:49.759215+0000","last_undegraded":"2024-04-26T17:53:49.759215+0000","last_fullsized":"2024-04-26T17:53:49.759215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:07.368256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:01.753375+0000","last_change":"2024-04-26T17:53:01.145471+0000","last_active":"2024-04-26T17:54:01.753375+0000","last_peered":"2024-04-26T17:54:01.753375+0000","last_clean":"2024-04-26T17:54:01.753375+0000","last_became_active":"2024-04-26T17:53:01.145040+0000","last_became_peered":"2024-04-26T17:53:01.145040+0000","last_unstale":"2024-04-26T17:54:01.753375+0000","last_undegraded":"2024-04-26T17:54:01.753375+0000","last_fullsized":"2024-04-26T17:54:01.753375+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:46:27.437329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765218+0000","last_change":"2024-04-26T17:53:03.618600+0000","last_active":"2024-04-26T17:53:49.765218+0000","last_peered":"2024-04-26T17:53:49.765218+0000","last_clean":"2024-04-26T17:53:49.765218+0000","last_became_active":"2024-04-26T17:53:03.618139+0000","last_became_peered":"2024-04-26T17:53:03.618139+0000","last_unstale":"2024-04-26T17:53:49.765218+0000","last_undegraded":"2024-04-26T17:53:49.765218+0000","last_fullsized":"2024-04-26T17:53:49.765218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:56.007659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891071+0000","last_change":"2024-04-26T17:53:00.586386+0000","last_active":"2024-04-26T17:53:49.891071+0000","last_peered":"2024-04-26T17:53:49.891071+0000","last_clean":"2024-04-26T17:53:49.891071+0000","last_became_active":"2024-04-26T17:53:00.585953+0000","last_became_peered":"2024-04-26T17:53:00.585953+0000","last_unstale":"2024-04-26T17:53:49.891071+0000","last_undegraded":"2024-04-26T17:53:49.891071+0000","last_fullsized":"2024-04-26T17:53:49.891071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:39:57.160983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":357,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767252+0000","last_change":"2024-04-26T17:51:38.839853+0000","last_active":"2024-04-26T17:53:49.767252+0000","last_peered":"2024-04-26T17:53:49.767252+0000","last_clean":"2024-04-26T17:53:49.767252+0000","last_became_active":"2024-04-26T17:51:38.839318+0000","last_became_peered":"2024-04-26T17:51:38.839318+0000","last_unstale":"2024-04-26T17:53:49.767252+0000","last_undegraded":"2024-04-26T17:53:49.767252+0000","last_fullsized":"2024-04-26T17:53:49.767252+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_clean_scrub_stamp":"2024-04-26T17:48:12.577543+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:08:50.683663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760286+0000","last_change":"2024-04-26T17:53:03.614170+0000","last_active":"2024-04-26T17:53:49.760286+0000","last_peered":"2024-04-26T17:53:49.760286+0000","last_clean":"2024-04-26T17:53:49.760286+0000","last_became_active":"2024-04-26T17:53:03.613836+0000","last_became_peered":"2024-04-26T17:53:03.613836+0000","last_unstale":"2024-04-26T17:53:49.760286+0000","last_undegraded":"2024-04-26T17:53:49.760286+0000","last_fullsized":"2024-04-26T17:53:49.760286+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:48.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767374+0000","last_change":"2024-04-26T17:53:00.587319+0000","last_active":"2024-04-26T17:53:49.767374+0000","last_peered":"2024-04-26T17:53:49.767374+0000","last_clean":"2024-04-26T17:53:49.767374+0000","last_became_active":"2024-04-26T17:53:00.586983+0000","last_became_peered":"2024-04-26T17:53:00.586983+0000","last_unstale":"2024-04-26T17:53:49.767374+0000","last_undegraded":"2024-04-26T17:53:49.767374+0000","last_fullsized":"2024-04-26T17:53:49.767374+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:03:43.470013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768739+0000","last_change":"2024-04-26T17:53:01.145672+0000","last_active":"2024-04-26T17:53:49.768739+0000","last_peered":"2024-04-26T17:53:49.768739+0000","last_clean":"2024-04-26T17:53:49.768739+0000","last_became_active":"2024-04-26T17:53:01.145274+0000","last_became_peered":"2024-04-26T17:53:01.145274+0000","last_unstale":"2024-04-26T17:53:49.768739+0000","last_undegraded":"2024-04-26T17:53:49.768739+0000","last_fullsized":"2024-04-26T17:53:49.768739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:21.122352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760603+0000","last_change":"2024-04-26T17:53:03.611232+0000","last_active":"2024-04-26T17:53:49.760603+0000","last_peered":"2024-04-26T17:53:49.760603+0000","last_clean":"2024-04-26T17:53:49.760603+0000","last_became_active":"2024-04-26T17:53:03.610905+0000","last_became_peered":"2024-04-26T17:53:03.610905+0000","last_unstale":"2024-04-26T17:53:49.760603+0000","last_undegraded":"2024-04-26T17:53:49.760603+0000","last_fullsized":"2024-04-26T17:53:49.760603+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:31.426237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'11","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.762463+0000","last_change":"2024-04-26T17:53:00.580220+0000","last_active":"2024-04-26T17:53:49.762463+0000","last_peered":"2024-04-26T17:53:49.762463+0000","last_clean":"2024-04-26T17:53:49.762463+0000","last_became_active":"2024-04-26T17:53:00.579818+0000","last_became_peered":"2024-04-26T17:53:00.579818+0000","last_unstale":"2024-04-26T17:53:49.762463+0000","last_undegraded":"2024-04-26T17:53:49.762463+0000","last_fullsized":"2024-04-26T17:53:49.762463+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:15.260489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765821+0000","last_change":"2024-04-26T17:53:03.610882+0000","last_active":"2024-04-26T17:53:49.765821+0000","last_peered":"2024-04-26T17:53:49.765821+0000","last_clean":"2024-04-26T17:53:49.765821+0000","last_became_active":"2024-04-26T17:53:03.609979+0000","last_became_peered":"2024-04-26T17:53:03.609979+0000","last_unstale":"2024-04-26T17:53:49.765821+0000","last_undegraded":"2024-04-26T17:53:49.765821+0000","last_fullsized":"2024-04-26T17:53:49.765821+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:33.167579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891536+0000","last_change":"2024-04-26T17:53:01.358555+0000","last_active":"2024-04-26T17:53:49.891536+0000","last_peered":"2024-04-26T17:53:49.891536+0000","last_clean":"2024-04-26T17:53:49.891536+0000","last_became_active":"2024-04-26T17:53:01.358138+0000","last_became_peered":"2024-04-26T17:53:01.358138+0000","last_unstale":"2024-04-26T17:53:49.891536+0000","last_undegraded":"2024-04-26T17:53:49.891536+0000","last_fullsized":"2024-04-26T17:53:49.891536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:14.886609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.766115+0000","last_change":"2024-04-26T17:53:03.618816+0000","last_active":"2024-04-26T17:53:49.766115+0000","last_peered":"2024-04-26T17:53:49.766115+0000","last_clean":"2024-04-26T17:53:49.766115+0000","last_became_active":"2024-04-26T17:53:03.618184+0000","last_became_peered":"2024-04-26T17:53:03.618184+0000","last_unstale":"2024-04-26T17:53:49.766115+0000","last_undegraded":"2024-04-26T17:53:49.766115+0000","last_fullsized":"2024-04-26T17:53:49.766115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:07.514516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758599+0000","last_change":"2024-04-26T17:53:01.147081+0000","last_active":"2024-04-26T17:53:49.758599+0000","last_peered":"2024-04-26T17:53:49.758599+0000","last_clean":"2024-04-26T17:53:49.758599+0000","last_became_active":"2024-04-26T17:53:01.146698+0000","last_became_peered":"2024-04-26T17:53:01.146698+0000","last_unstale":"2024-04-26T17:53:49.758599+0000","last_undegraded":"2024-04-26T17:53:49.758599+0000","last_fullsized":"2024-04-26T17:53:49.758599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:44:19.927316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767053+0000","last_change":"2024-04-26T17:53:03.612592+0000","last_active":"2024-04-26T17:53:49.767053+0000","last_peered":"2024-04-26T17:53:49.767053+0000","last_clean":"2024-04-26T17:53:49.767053+0000","last_became_active":"2024-04-26T17:53:03.612274+0000","last_became_peered":"2024-04-26T17:53:03.612274+0000","last_unstale":"2024-04-26T17:53:49.767053+0000","last_undegraded":"2024-04-26T17:53:49.767053+0000","last_fullsized":"2024-04-26T17:53:49.767053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:30:48.461947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764834+0000","last_change":"2024-04-26T17:53:00.586430+0000","last_active":"2024-04-26T17:53:49.764834+0000","last_peered":"2024-04-26T17:53:49.764834+0000","last_clean":"2024-04-26T17:53:49.764834+0000","last_became_active":"2024-04-26T17:53:00.586004+0000","last_became_peered":"2024-04-26T17:53:00.586004+0000","last_unstale":"2024-04-26T17:53:49.764834+0000","last_undegraded":"2024-04-26T17:53:49.764834+0000","last_fullsized":"2024-04-26T17:53:49.764834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:47:18.224890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764726+0000","last_change":"2024-04-26T17:53:03.623081+0000","last_active":"2024-04-26T17:53:49.764726+0000","last_peered":"2024-04-26T17:53:49.764726+0000","last_clean":"2024-04-26T17:53:49.764726+0000","last_became_active":"2024-04-26T17:53:03.622761+0000","last_became_peered":"2024-04-26T17:53:03.622761+0000","last_unstale":"2024-04-26T17:53:49.764726+0000","last_undegraded":"2024-04-26T17:53:49.764726+0000","last_fullsized":"2024-04-26T17:53:49.764726+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:39.520075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768849+0000","last_change":"2024-04-26T17:53:00.583013+0000","last_active":"2024-04-26T17:53:49.768849+0000","last_peered":"2024-04-26T17:53:49.768849+0000","last_clean":"2024-04-26T17:53:49.768849+0000","last_became_active":"2024-04-26T17:53:00.582653+0000","last_became_peered":"2024-04-26T17:53:00.582653+0000","last_unstale":"2024-04-26T17:53:49.768849+0000","last_undegraded":"2024-04-26T17:53:49.768849+0000","last_fullsized":"2024-04-26T17:53:49.768849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:09.448621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765042+0000","last_change":"2024-04-26T17:53:03.618232+0000","last_active":"2024-04-26T17:53:49.765042+0000","last_peered":"2024-04-26T17:53:49.765042+0000","last_clean":"2024-04-26T17:53:49.765042+0000","last_became_active":"2024-04-26T17:53:03.617906+0000","last_became_peered":"2024-04-26T17:53:03.617906+0000","last_unstale":"2024-04-26T17:53:49.765042+0000","last_undegraded":"2024-04-26T17:53:49.765042+0000","last_fullsized":"2024-04-26T17:53:49.765042+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:17.357398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768301+0000","last_change":"2024-04-26T17:53:00.586565+0000","last_active":"2024-04-26T17:53:49.768301+0000","last_peered":"2024-04-26T17:53:49.768301+0000","last_clean":"2024-04-26T17:53:49.768301+0000","last_became_active":"2024-04-26T17:53:00.586161+0000","last_became_peered":"2024-04-26T17:53:00.586161+0000","last_unstale":"2024-04-26T17:53:49.768301+0000","last_undegraded":"2024-04-26T17:53:49.768301+0000","last_fullsized":"2024-04-26T17:53:49.768301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:56:25.294055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767400+0000","last_change":"2024-04-26T17:53:03.620340+0000","last_active":"2024-04-26T17:53:49.767400+0000","last_peered":"2024-04-26T17:53:49.767400+0000","last_clean":"2024-04-26T17:53:49.767400+0000","last_became_active":"2024-04-26T17:53:03.619897+0000","last_became_peered":"2024-04-26T17:53:03.619897+0000","last_unstale":"2024-04-26T17:53:49.767400+0000","last_undegraded":"2024-04-26T17:53:49.767400+0000","last_fullsized":"2024-04-26T17:53:49.767400+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:12:40.662070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769496+0000","last_change":"2024-04-26T17:53:01.358131+0000","last_active":"2024-04-26T17:53:49.769496+0000","last_peered":"2024-04-26T17:53:49.769496+0000","last_clean":"2024-04-26T17:53:49.769496+0000","last_became_active":"2024-04-26T17:53:01.357796+0000","last_became_peered":"2024-04-26T17:53:01.357796+0000","last_unstale":"2024-04-26T17:53:49.769496+0000","last_undegraded":"2024-04-26T17:53:49.769496+0000","last_fullsized":"2024-04-26T17:53:49.769496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:47.849873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767708+0000","last_change":"2024-04-26T17:53:03.616133+0000","last_active":"2024-04-26T17:53:49.767708+0000","last_peered":"2024-04-26T17:53:49.767708+0000","last_clean":"2024-04-26T17:53:49.767708+0000","last_became_active":"2024-04-26T17:53:03.615797+0000","last_became_peered":"2024-04-26T17:53:03.615797+0000","last_unstale":"2024-04-26T17:53:49.767708+0000","last_undegraded":"2024-04-26T17:53:49.767708+0000","last_fullsized":"2024-04-26T17:53:49.767708+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:05.698489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759241+0000","last_change":"2024-04-26T17:53:00.587696+0000","last_active":"2024-04-26T17:53:49.759241+0000","last_peered":"2024-04-26T17:53:49.759241+0000","last_clean":"2024-04-26T17:53:49.759241+0000","last_became_active":"2024-04-26T17:53:00.587305+0000","last_became_peered":"2024-04-26T17:53:00.587305+0000","last_unstale":"2024-04-26T17:53:49.759241+0000","last_undegraded":"2024-04-26T17:53:49.759241+0000","last_fullsized":"2024-04-26T17:53:49.759241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:39.185753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767528+0000","last_change":"2024-04-26T17:53:03.613076+0000","last_active":"2024-04-26T17:53:49.767528+0000","last_peered":"2024-04-26T17:53:49.767528+0000","last_clean":"2024-04-26T17:53:49.767528+0000","last_became_active":"2024-04-26T17:53:03.612746+0000","last_became_peered":"2024-04-26T17:53:03.612746+0000","last_unstale":"2024-04-26T17:53:49.767528+0000","last_undegraded":"2024-04-26T17:53:49.767528+0000","last_fullsized":"2024-04-26T17:53:49.767528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:20.920154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760620+0000","last_change":"2024-04-26T17:53:00.582575+0000","last_active":"2024-04-26T17:53:49.760620+0000","last_peered":"2024-04-26T17:53:49.760620+0000","last_clean":"2024-04-26T17:53:49.760620+0000","last_became_active":"2024-04-26T17:53:00.582215+0000","last_became_peered":"2024-04-26T17:53:00.582215+0000","last_unstale":"2024-04-26T17:53:49.760620+0000","last_undegraded":"2024-04-26T17:53:49.760620+0000","last_fullsized":"2024-04-26T17:53:49.760620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:35:12.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767871+0000","last_change":"2024-04-26T17:53:03.616269+0000","last_active":"2024-04-26T17:53:49.767871+0000","last_peered":"2024-04-26T17:53:49.767871+0000","last_clean":"2024-04-26T17:53:49.767871+0000","last_became_active":"2024-04-26T17:53:03.615896+0000","last_became_peered":"2024-04-26T17:53:03.615896+0000","last_unstale":"2024-04-26T17:53:49.767871+0000","last_undegraded":"2024-04-26T17:53:49.767871+0000","last_fullsized":"2024-04-26T17:53:49.767871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:12.111392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764802+0000","last_change":"2024-04-26T17:53:03.621589+0000","last_active":"2024-04-26T17:53:49.764802+0000","last_peered":"2024-04-26T17:53:49.764802+0000","last_clean":"2024-04-26T17:53:49.764802+0000","last_became_active":"2024-04-26T17:53:03.621211+0000","last_became_peered":"2024-04-26T17:53:03.621211+0000","last_unstale":"2024-04-26T17:53:49.764802+0000","last_undegraded":"2024-04-26T17:53:49.764802+0000","last_fullsized":"2024-04-26T17:53:49.764802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.487476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768626+0000","last_change":"2024-04-26T17:53:03.624536+0000","last_active":"2024-04-26T17:53:49.768626+0000","last_peered":"2024-04-26T17:53:49.768626+0000","last_clean":"2024-04-26T17:53:49.768626+0000","last_became_active":"2024-04-26T17:53:03.624234+0000","last_became_peered":"2024-04-26T17:53:03.624234+0000","last_unstale":"2024-04-26T17:53:49.768626+0000","last_undegraded":"2024-04-26T17:53:49.768626+0000","last_fullsized":"2024-04-26T17:53:49.768626+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:31:36.322688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768748+0000","last_change":"2024-04-26T17:53:03.620096+0000","last_active":"2024-04-26T17:53:49.768748+0000","last_peered":"2024-04-26T17:53:49.768748+0000","last_clean":"2024-04-26T17:53:49.768748+0000","last_became_active":"2024-04-26T17:53:03.619729+0000","last_became_peered":"2024-04-26T17:53:03.619729+0000","last_unstale":"2024-04-26T17:53:49.768748+0000","last_undegraded":"2024-04-26T17:53:49.768748+0000","last_fullsized":"2024-04-26T17:53:49.768748+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:17.987728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768829+0000","last_change":"2024-04-26T17:53:49.768829+0000","last_active":"2024-04-26T17:53:49.768829+0000","last_peered":"2024-04-26T17:53:49.768829+0000","last_clean":"2024-04-26T17:53:49.768829+0000","last_became_active":"2024-04-26T17:53:49.768415+0000","last_became_peered":"2024-04-26T17:53:49.768415+0000","last_unstale":"2024-04-26T17:53:49.768829+0000","last_undegraded":"2024-04-26T17:53:49.768829+0000","last_fullsized":"2024-04-26T17:53:49.768829+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:57.364355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760497+0000","last_change":"2024-04-26T17:53:03.619193+0000","last_active":"2024-04-26T17:53:49.760497+0000","last_peered":"2024-04-26T17:53:49.760497+0000","last_clean":"2024-04-26T17:53:49.760497+0000","last_became_active":"2024-04-26T17:53:03.618776+0000","last_became_peered":"2024-04-26T17:53:03.618776+0000","last_unstale":"2024-04-26T17:53:49.760497+0000","last_undegraded":"2024-04-26T17:53:49.760497+0000","last_fullsized":"2024-04-26T17:53:49.760497+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:07:24.811683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.765044+0000","last_change":"2024-04-26T17:53:03.621366+0000","last_active":"2024-04-26T17:53:49.765044+0000","last_peered":"2024-04-26T17:53:49.765044+0000","last_clean":"2024-04-26T17:53:49.765044+0000","last_became_active":"2024-04-26T17:53:03.621048+0000","last_became_peered":"2024-04-26T17:53:03.621048+0000","last_unstale":"2024-04-26T17:53:49.765044+0000","last_undegraded":"2024-04-26T17:53:49.765044+0000","last_fullsized":"2024-04-26T17:53:49.765044+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:31:06.484880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769806+0000","last_change":"2024-04-26T17:53:03.619383+0000","last_active":"2024-04-26T17:53:49.769806+0000","last_peered":"2024-04-26T17:53:49.769806+0000","last_clean":"2024-04-26T17:53:49.769806+0000","last_became_active":"2024-04-26T17:53:03.619017+0000","last_became_peered":"2024-04-26T17:53:03.619017+0000","last_unstale":"2024-04-26T17:53:49.769806+0000","last_undegraded":"2024-04-26T17:53:49.769806+0000","last_fullsized":"2024-04-26T17:53:49.769806+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:20.091959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.770695+0000","last_change":"2024-04-26T17:53:49.770695+0000","last_active":"2024-04-26T17:53:49.770695+0000","last_peered":"2024-04-26T17:53:49.770695+0000","last_clean":"2024-04-26T17:53:49.770695+0000","last_became_active":"2024-04-26T17:53:49.770303+0000","last_became_peered":"2024-04-26T17:53:49.770303+0000","last_unstale":"2024-04-26T17:53:49.770695+0000","last_undegraded":"2024-04-26T17:53:49.770695+0000","last_fullsized":"2024-04-26T17:53:49.770695+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:22.830129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758969+0000","last_change":"2024-04-26T17:53:03.605521+0000","last_active":"2024-04-26T17:53:49.758969+0000","last_peered":"2024-04-26T17:53:49.758969+0000","last_clean":"2024-04-26T17:53:49.758969+0000","last_became_active":"2024-04-26T17:53:03.605145+0000","last_became_peered":"2024-04-26T17:53:03.605145+0000","last_unstale":"2024-04-26T17:53:49.758969+0000","last_undegraded":"2024-04-26T17:53:49.758969+0000","last_fullsized":"2024-04-26T17:53:49.758969+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:58:09.856021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764723+0000","last_change":"2024-04-26T17:53:03.610866+0000","last_active":"2024-04-26T17:53:49.764723+0000","last_peered":"2024-04-26T17:53:49.764723+0000","last_clean":"2024-04-26T17:53:49.764723+0000","last_became_active":"2024-04-26T17:53:03.610326+0000","last_became_peered":"2024-04-26T17:53:03.610326+0000","last_unstale":"2024-04-26T17:53:49.764723+0000","last_undegraded":"2024-04-26T17:53:49.764723+0000","last_fullsized":"2024-04-26T17:53:49.764723+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:20.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758767+0000","last_change":"2024-04-26T17:53:03.615336+0000","last_active":"2024-04-26T17:53:49.758767+0000","last_peered":"2024-04-26T17:53:49.758767+0000","last_clean":"2024-04-26T17:53:49.758767+0000","last_became_active":"2024-04-26T17:53:03.614447+0000","last_became_peered":"2024-04-26T17:53:03.614447+0000","last_unstale":"2024-04-26T17:53:49.758767+0000","last_undegraded":"2024-04-26T17:53:49.758767+0000","last_fullsized":"2024-04-26T17:53:49.758767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:36:10.091929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760835+0000","last_change":"2024-04-26T17:53:03.614863+0000","last_active":"2024-04-26T17:53:49.760835+0000","last_peered":"2024-04-26T17:53:49.760835+0000","last_clean":"2024-04-26T17:53:49.760835+0000","last_became_active":"2024-04-26T17:53:03.614520+0000","last_became_peered":"2024-04-26T17:53:03.614520+0000","last_unstale":"2024-04-26T17:53:49.760835+0000","last_undegraded":"2024-04-26T17:53:49.760835+0000","last_fullsized":"2024-04-26T17:53:49.760835+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:51.367785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764960+0000","last_change":"2024-04-26T17:53:03.618521+0000","last_active":"2024-04-26T17:53:49.764960+0000","last_peered":"2024-04-26T17:53:49.764960+0000","last_clean":"2024-04-26T17:53:49.764960+0000","last_became_active":"2024-04-26T17:53:03.617658+0000","last_became_peered":"2024-04-26T17:53:03.617658+0000","last_unstale":"2024-04-26T17:53:49.764960+0000","last_undegraded":"2024-04-26T17:53:49.764960+0000","last_fullsized":"2024-04-26T17:53:49.764960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:39.813098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280045,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66336,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":283,"num_read_kb":213,"num_write":165,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":199008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383707,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":479802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645345,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":1133618,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808869,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":480038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070505,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":479958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332143,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":1071888,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495669,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":983040,"data_stored":479427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":541555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":480698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":543296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182467,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":999424,"data_stored":479738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920830,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":479864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757306,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1056768,"data_stored":1071572,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":63913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T17:54:08.561 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T17:54:08.562 INFO:tasks.cephadm:Setting up client nodes... 2024-04-26T17:54:08.562 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T17:54:08.819 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:09.101 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:09 smithi031 ceph-mon[29164]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:54:09.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:09 smithi097 ceph-mon[32419]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:54:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:09 smithi186 ceph-mon[32797]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:54:10.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:10 smithi031 ceph-mon[29164]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:10.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:10 smithi097 ceph-mon[32419]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:10.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:10 smithi186 ceph-mon[32797]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:10.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.603+0000 7fb00570f700 1 -- 172.21.15.31:0/879569615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb000105070 msgr2=0x7fb00010b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:10.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.603+0000 7fb00570f700 1 --2- 172.21.15.31:0/879569615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb000105070 0x7fb00010b920 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7faff800b0d0 tx=0x7faff800b3e0 comp rx=0 tx=0).stop 2024-04-26T17:54:10.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.603+0000 7fb00570f700 1 -- 172.21.15.31:0/879569615 shutdown_connections 2024-04-26T17:54:10.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.603+0000 7fb00570f700 1 --2- 172.21.15.31:0/879569615 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0001034d0 0x7fb0001038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:10.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.603+0000 7fb00570f700 1 --2- 172.21.15.31:0/879569615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb000105070 0x7fb00010b920 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:10.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.603+0000 7fb00570f700 1 --2- 172.21.15.31:0/879569615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0001046d0 0x7fb000104b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:10.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.603+0000 7fb00570f700 1 -- 172.21.15.31:0/879569615 >> 172.21.15.31:0/879569615 conn(0x7fb0000fea70 msgr2=0x7fb000100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:10.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.603+0000 7fb00570f700 1 -- 172.21.15.31:0/879569615 shutdown_connections 2024-04-26T17:54:10.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.603+0000 7fb00570f700 1 -- 172.21.15.31:0/879569615 wait complete. 2024-04-26T17:54:10.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.603+0000 7fb00570f700 1 Processor -- start 2024-04-26T17:54:10.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fb00570f700 1 -- start start 2024-04-26T17:54:10.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fb00570f700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0001034d0 0x7fb0000771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:10.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fb00570f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0001046d0 0x7fb000075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:10.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fb00570f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb000105070 0x7fb000075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:10.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fb00570f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb00010efc0 con 0x7fb000105070 2024-04-26T17:54:10.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fb00570f700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb00010ee40 con 0x7fb0001034d0 2024-04-26T17:54:10.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fb00570f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb00010f140 con 0x7fb0001046d0 2024-04-26T17:54:10.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fafff7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb000105070 0x7fb000075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:10.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fafff7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb000105070 0x7fb000075d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57618/0 (socket says 172.21.15.31:57618) 2024-04-26T17:54:10.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fafff7fe700 1 -- 172.21.15.31:0/1624704137 learned_addr learned my addr 172.21.15.31:0/1624704137 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:10.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7faffe7fc700 1 --2- 172.21.15.31:0/1624704137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0001046d0 0x7fb000075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:10.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7faffeffd700 1 --2- 172.21.15.31:0/1624704137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0001034d0 0x7fb0000771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:10.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fafff7fe700 1 -- 172.21.15.31:0/1624704137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0001046d0 msgr2=0x7fb000075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fafff7fe700 1 --2- 172.21.15.31:0/1624704137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0001046d0 0x7fb000075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fafff7fe700 1 -- 172.21.15.31:0/1624704137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0001034d0 msgr2=0x7fb0000771c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fafff7fe700 1 --2- 172.21.15.31:0/1624704137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0001034d0 0x7fb0000771c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.604+0000 7fafff7fe700 1 -- 172.21.15.31:0/1624704137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb000076600 con 0x7fb000105070 2024-04-26T17:54:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.605+0000 7fafff7fe700 1 --2- 172.21.15.31:0/1624704137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb000105070 0x7fb000075d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faff8009040 tx=0x7faff8004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.605+0000 7fafe7fff700 1 -- 172.21.15.31:0/1624704137 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff801f030 con 0x7fb000105070 2024-04-26T17:54:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.605+0000 7fb00570f700 1 -- 172.21.15.31:0/1624704137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0001a3590 con 0x7fb000105070 2024-04-26T17:54:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.605+0000 7fafe7fff700 1 -- 172.21.15.31:0/1624704137 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faff8003a00 con 0x7fb000105070 2024-04-26T17:54:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.605+0000 7fafe7fff700 1 -- 172.21.15.31:0/1624704137 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff80046e0 con 0x7fb000105070 2024-04-26T17:54:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.605+0000 7fb00570f700 1 -- 172.21.15.31:0/1624704137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0001a3a60 con 0x7fb000105070 2024-04-26T17:54:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.606+0000 7fb00570f700 1 -- 172.21.15.31:0/1624704137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb000051960 con 0x7fb000105070 2024-04-26T17:54:10.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.607+0000 7fafe7fff700 1 -- 172.21.15.31:0/1624704137 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faff8003bb0 con 0x7fb000105070 2024-04-26T17:54:10.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.608+0000 7fafe7fff700 1 --2- 172.21.15.31:0/1624704137 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fafe80a3d80 0x7fafe80a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:10.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.608+0000 7faffeffd700 1 --2- 172.21.15.31:0/1624704137 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fafe80a3d80 0x7fafe80a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:10.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.608+0000 7fafe7fff700 1 -- 172.21.15.31:0/1624704137 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7faff800f020 con 0x7fb000105070 2024-04-26T17:54:10.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.608+0000 7faffeffd700 1 --2- 172.21.15.31:0/1624704137 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fafe80a3d80 0x7fafe80a6240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb000104530 tx=0x7faff400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:10.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.612+0000 7fafe7fff700 1 -- 172.21.15.31:0/1624704137 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faff8029020 con 0x7fb000105070 2024-04-26T17:54:10.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.908+0000 7fb00570f700 1 -- 172.21.15.31:0/1624704137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fb00002e530 con 0x7fb000105070 2024-04-26T17:54:10.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.911+0000 7fafe7fff700 1 -- 172.21.15.31:0/1624704137 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fb00002e530 con 0x7fb000105070 2024-04-26T17:54:10.911 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-04-26T17:54:10.912 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQBC6itmxt1FNhAA8yrjn7YC9Gb1mqmY3J0iBA== 2024-04-26T17:54:10.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.913+0000 7fb00570f700 1 -- 172.21.15.31:0/1624704137 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fafe80a3d80 msgr2=0x7fafe80a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:10.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.913+0000 7fb00570f700 1 --2- 172.21.15.31:0/1624704137 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fafe80a3d80 0x7fafe80a6240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb000104530 tx=0x7faff400b410 comp rx=0 tx=0).stop 2024-04-26T17:54:10.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.913+0000 7fb00570f700 1 -- 172.21.15.31:0/1624704137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb000105070 msgr2=0x7fb000075d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:10.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.913+0000 7fb00570f700 1 --2- 172.21.15.31:0/1624704137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb000105070 0x7fb000075d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faff8009040 tx=0x7faff8004d50 comp rx=0 tx=0).stop 2024-04-26T17:54:10.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.913+0000 7fb00570f700 1 -- 172.21.15.31:0/1624704137 shutdown_connections 2024-04-26T17:54:10.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.913+0000 7fb00570f700 1 --2- 172.21.15.31:0/1624704137 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fafe80a3d80 0x7fafe80a6240 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:10.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.913+0000 7fb00570f700 1 --2- 172.21.15.31:0/1624704137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb0001034d0 0x7fb0000771c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:10.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.913+0000 7fb00570f700 1 --2- 172.21.15.31:0/1624704137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb000105070 0x7fb000075d50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:10.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.913+0000 7fb00570f700 1 --2- 172.21.15.31:0/1624704137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0001046d0 0x7fb000075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:10.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.913+0000 7fb00570f700 1 -- 172.21.15.31:0/1624704137 >> 172.21.15.31:0/1624704137 conn(0x7fb0000fea70 msgr2=0x7fb000107900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:10.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.914+0000 7fb00570f700 1 -- 172.21.15.31:0/1624704137 shutdown_connections 2024-04-26T17:54:10.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:10.914+0000 7fb00570f700 1 -- 172.21.15.31:0/1624704137 wait complete. 2024-04-26T17:54:11.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:11 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1624704137' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T17:54:11.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:11 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1624704137' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T17:54:11.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:11 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1624704137' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T17:54:11.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:11 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1624704137' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T17:54:11.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:11 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1624704137' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T17:54:11.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:11 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1624704137' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T17:54:11.548 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:54:11.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-26T17:54:11.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T17:54:11.588 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T17:54:11.588 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-26T17:54:11.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-26T17:54:11.877 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:12.251 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:12 smithi031 ceph-mon[29164]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:12.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:12 smithi097 ceph-mon[32419]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:12.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:12 smithi186 ceph-mon[32797]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:13.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.602+0000 7ffbaba70700 1 -- 172.21.15.31:0/2215443274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba4103470 msgr2=0x7ffba41038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:13.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.602+0000 7ffbaba70700 1 --2- 172.21.15.31:0/2215443274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba4103470 0x7ffba41038f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffb94009990 tx=0x7ffb94009ca0 comp rx=0 tx=0).stop 2024-04-26T17:54:13.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.602+0000 7ffbaba70700 1 -- 172.21.15.31:0/2215443274 shutdown_connections 2024-04-26T17:54:13.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.602+0000 7ffbaba70700 1 --2- 172.21.15.31:0/2215443274 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffba41094c0 0x7ffba41098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:13.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.602+0000 7ffbaba70700 1 --2- 172.21.15.31:0/2215443274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffba4103e30 0x7ffba4110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:13.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.602+0000 7ffbaba70700 1 --2- 172.21.15.31:0/2215443274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba4103470 0x7ffba41038f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:13.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.602+0000 7ffbaba70700 1 -- 172.21.15.31:0/2215443274 >> 172.21.15.31:0/2215443274 conn(0x7ffba407a580 msgr2=0x7ffba407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.602+0000 7ffbaba70700 1 -- 172.21.15.31:0/2215443274 shutdown_connections 2024-04-26T17:54:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.602+0000 7ffbaba70700 1 -- 172.21.15.31:0/2215443274 wait complete. 2024-04-26T17:54:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffbaba70700 1 Processor -- start 2024-04-26T17:54:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffbaba70700 1 -- start start 2024-04-26T17:54:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffbaba70700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffba4103470 0x7ffba419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffbaba70700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba4103e30 0x7ffba419fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffbaba70700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffba41094c0 0x7ffba41998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffbaba70700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffba41135f0 con 0x7ffba4103e30 2024-04-26T17:54:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffbaba70700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ffba4113470 con 0x7ffba4103470 2024-04-26T17:54:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffbaba70700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffba4113770 con 0x7ffba41094c0 2024-04-26T17:54:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffba900b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba4103e30 0x7ffba419fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffba900b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba4103e30 0x7ffba419fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57636/0 (socket says 172.21.15.31:57636) 2024-04-26T17:54:13.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffba900b700 1 -- 172.21.15.31:0/2742583083 learned_addr learned my addr 172.21.15.31:0/2742583083 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffba980c700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffba4103470 0x7ffba419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffbaa00d700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffba41094c0 0x7ffba41998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffba900b700 1 -- 172.21.15.31:0/2742583083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffba41094c0 msgr2=0x7ffba41998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffba900b700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffba41094c0 0x7ffba41998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffba900b700 1 -- 172.21.15.31:0/2742583083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffba4103470 msgr2=0x7ffba419f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffba900b700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffba4103470 0x7ffba419f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffba900b700 1 -- 172.21.15.31:0/2742583083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffba419a030 con 0x7ffba4103e30 2024-04-26T17:54:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.603+0000 7ffba980c700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffba4103470 0x7ffba419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:54:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.604+0000 7ffba900b700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba4103e30 0x7ffba419fc50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ffb94000c00 tx=0x7ffb940047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.604+0000 7ffb9affd700 1 -- 172.21.15.31:0/2742583083 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb94027070 con 0x7ffba4103e30 2024-04-26T17:54:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.604+0000 7ffb9affd700 1 -- 172.21.15.31:0/2742583083 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb940041d0 con 0x7ffba4103e30 2024-04-26T17:54:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.604+0000 7ffbaba70700 1 -- 172.21.15.31:0/2742583083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffba419a2c0 con 0x7ffba4103e30 2024-04-26T17:54:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.604+0000 7ffb9affd700 1 -- 172.21.15.31:0/2742583083 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9400f460 con 0x7ffba4103e30 2024-04-26T17:54:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.604+0000 7ffbaba70700 1 -- 172.21.15.31:0/2742583083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffba41a65e0 con 0x7ffba4103e30 2024-04-26T17:54:13.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.605+0000 7ffbaba70700 1 -- 172.21.15.31:0/2742583083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffba410b930 con 0x7ffba4103e30 2024-04-26T17:54:13.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.607+0000 7ffb9affd700 1 -- 172.21.15.31:0/2742583083 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ffb9402a070 con 0x7ffba4103e30 2024-04-26T17:54:13.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.607+0000 7ffb9affd700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ffb900a3d50 0x7ffb900a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:13.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.607+0000 7ffba980c700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ffb900a3d50 0x7ffb900a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:13.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.607+0000 7ffb9affd700 1 -- 172.21.15.31:0/2742583083 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7ffb940d18f0 con 0x7ffba4103e30 2024-04-26T17:54:13.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.608+0000 7ffba980c700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ffb900a3d50 0x7ffb900a6210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ffba0005ea0 tx=0x7ffba0005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:13.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.611+0000 7ffb9affd700 1 -- 172.21.15.31:0/2742583083 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffb94031070 con 0x7ffba4103e30 2024-04-26T17:54:13.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.891+0000 7ffbaba70700 1 -- 172.21.15.31:0/2742583083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ffba419ae00 con 0x7ffba4103e30 2024-04-26T17:54:13.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.896+0000 7ffb9affd700 1 -- 172.21.15.31:0/2742583083 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264996 (secure 0 0 0) 0x7ffb94013070 con 0x7ffba4103e30 2024-04-26T17:54:13.899 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:54:13.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.899+0000 7ffbaba70700 1 -- 172.21.15.31:0/2742583083 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ffb900a3d50 msgr2=0x7ffb900a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:13.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.899+0000 7ffbaba70700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ffb900a3d50 0x7ffb900a6210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ffba0005ea0 tx=0x7ffba0005e30 comp rx=0 tx=0).stop 2024-04-26T17:54:13.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.899+0000 7ffbaba70700 1 -- 172.21.15.31:0/2742583083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba4103e30 msgr2=0x7ffba419fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:13.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.899+0000 7ffbaba70700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba4103e30 0x7ffba419fc50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ffb94000c00 tx=0x7ffb940047b0 comp rx=0 tx=0).stop 2024-04-26T17:54:13.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.899+0000 7ffbaba70700 1 -- 172.21.15.31:0/2742583083 shutdown_connections 2024-04-26T17:54:13.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.899+0000 7ffbaba70700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ffb900a3d50 0x7ffb900a6210 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:13.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.899+0000 7ffbaba70700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffba4103470 0x7ffba419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:13.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.899+0000 7ffbaba70700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffba41094c0 0x7ffba41998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:13.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.899+0000 7ffbaba70700 1 --2- 172.21.15.31:0/2742583083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba4103e30 0x7ffba419fc50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:13.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.899+0000 7ffbaba70700 1 -- 172.21.15.31:0/2742583083 >> 172.21.15.31:0/2742583083 conn(0x7ffba407a580 msgr2=0x7ffba4102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:13.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.899+0000 7ffbaba70700 1 -- 172.21.15.31:0/2742583083 shutdown_connections 2024-04-26T17:54:13.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:13.899+0000 7ffbaba70700 1 -- 172.21.15.31:0/2742583083 wait complete. 2024-04-26T17:54:14.224 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:14 smithi031 ceph-mon[29164]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:54:14.224 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:14 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2742583083' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T17:54:14.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:14 smithi097 ceph-mon[32419]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:54:14.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:14 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2742583083' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T17:54:14.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:14 smithi186 ceph-mon[32797]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:54:14.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:14 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2742583083' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T17:54:14.636 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2410008001},{"type":"v1","addr":"172.21.15.31:6801","nonce":2410008001}]},"active_addr":"172.21.15.31:6801/2410008001","active_change":"2024-04-26T17:45:48.218927+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.31:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":540352687}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":614760140}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2095760002}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":3429421652}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2225674161}]}]} 2024-04-26T17:54:14.642 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-26T17:54:14.642 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T17:54:14.642 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T17:54:14.899 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:16.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:16 smithi031 ceph-mon[29164]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:16.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:16 smithi097 ceph-mon[32419]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:16.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:16 smithi186 ceph-mon[32797]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:16.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.549+0000 7f7cacd75700 1 -- 172.21.15.31:0/2722885731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ca8103470 msgr2=0x7f7ca81038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:16.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.549+0000 7f7cacd75700 1 --2- 172.21.15.31:0/2722885731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ca8103470 0x7f7ca81038f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009990 tx=0x7f7c9c009ca0 comp rx=0 tx=0).stop 2024-04-26T17:54:16.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.549+0000 7f7cacd75700 1 -- 172.21.15.31:0/2722885731 shutdown_connections 2024-04-26T17:54:16.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.549+0000 7f7cacd75700 1 --2- 172.21.15.31:0/2722885731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ca8103e30 0x7f7ca8110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:16.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.549+0000 7f7cacd75700 1 --2- 172.21.15.31:0/2722885731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ca81094c0 0x7f7ca81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:16.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.549+0000 7f7cacd75700 1 --2- 172.21.15.31:0/2722885731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ca8103470 0x7f7ca81038f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:16.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.549+0000 7f7cacd75700 1 -- 172.21.15.31:0/2722885731 >> 172.21.15.31:0/2722885731 conn(0x7f7ca807a580 msgr2=0x7f7ca807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:16.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.549+0000 7f7cacd75700 1 -- 172.21.15.31:0/2722885731 shutdown_connections 2024-04-26T17:54:16.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.549+0000 7f7cacd75700 1 -- 172.21.15.31:0/2722885731 wait complete. 2024-04-26T17:54:16.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7cacd75700 1 Processor -- start 2024-04-26T17:54:16.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7cacd75700 1 -- start start 2024-04-26T17:54:16.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7cacd75700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ca8103470 0x7f7ca8075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:16.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7cacd75700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ca8103e30 0x7f7ca8075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:16.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7cacd75700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ca81094c0 0x7f7ca80761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:16.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7cacd75700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca8113710 con 0x7f7ca8103470 2024-04-26T17:54:16.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7cacd75700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca8113590 con 0x7f7ca81094c0 2024-04-26T17:54:16.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7cacd75700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca8113890 con 0x7f7ca8103e30 2024-04-26T17:54:16.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7ca659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ca8103470 0x7f7ca8075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:16.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7ca659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ca8103470 0x7f7ca8075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57074/0 (socket says 172.21.15.31:57074) 2024-04-26T17:54:16.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7ca659c700 1 -- 172.21.15.31:0/3208424461 learned_addr learned my addr 172.21.15.31:0/3208424461 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:16.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7ca6d9d700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ca81094c0 0x7f7ca80761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:16.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7ca5d9b700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ca8103e30 0x7f7ca8075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:16.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7ca659c700 1 -- 172.21.15.31:0/3208424461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ca8103e30 msgr2=0x7f7ca8075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:16.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7ca659c700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ca8103e30 0x7f7ca8075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:16.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7ca659c700 1 -- 172.21.15.31:0/3208424461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ca81094c0 msgr2=0x7f7ca80761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:16.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7ca659c700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ca81094c0 0x7f7ca80761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:16.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7ca659c700 1 -- 172.21.15.31:0/3208424461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca81a8210 con 0x7f7ca8103470 2024-04-26T17:54:16.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7ca6d9d700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ca81094c0 0x7f7ca80761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:54:16.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.550+0000 7f7ca659c700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ca8103470 0x7f7ca8075720 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800b530 tx=0x7f7c9800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:16.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.551+0000 7f7c977fe700 1 -- 172.21.15.31:0/3208424461 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c98015020 con 0x7f7ca8103470 2024-04-26T17:54:16.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.551+0000 7f7cacd75700 1 -- 172.21.15.31:0/3208424461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca81a8500 con 0x7f7ca8103470 2024-04-26T17:54:16.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.551+0000 7f7c977fe700 1 -- 172.21.15.31:0/3208424461 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c98011ce0 con 0x7f7ca8103470 2024-04-26T17:54:16.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.551+0000 7f7c977fe700 1 -- 172.21.15.31:0/3208424461 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c98010d20 con 0x7f7ca8103470 2024-04-26T17:54:16.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.551+0000 7f7cacd75700 1 -- 172.21.15.31:0/3208424461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca81a8a40 con 0x7f7ca8103470 2024-04-26T17:54:16.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.552+0000 7f7cacd75700 1 -- 172.21.15.31:0/3208424461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ca806a110 con 0x7f7ca8103470 2024-04-26T17:54:16.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.553+0000 7f7c977fe700 1 -- 172.21.15.31:0/3208424461 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7c98018020 con 0x7f7ca8103470 2024-04-26T17:54:16.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.553+0000 7f7c977fe700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f7c900a3cd0 0x7f7c900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:16.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.554+0000 7f7c977fe700 1 -- 172.21.15.31:0/3208424461 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f7c980cfd20 con 0x7f7ca8103470 2024-04-26T17:54:16.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.554+0000 7f7ca5d9b700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f7c900a3cd0 0x7f7c900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:16.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.554+0000 7f7ca5d9b700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f7c900a3cd0 0x7f7c900a6190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00b5c0 tx=0x7f7c9c01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:16.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.558+0000 7f7c977fe700 1 -- 172.21.15.31:0/3208424461 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c9801f020 con 0x7f7ca8103470 2024-04-26T17:54:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.771+0000 7f7cacd75700 1 -- 172.21.15.31:0/3208424461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ca802e490 con 0x7f7ca8103470 2024-04-26T17:54:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.772+0000 7f7c977fe700 1 -- 172.21.15.31:0/3208424461 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22577 (secure 0 0 0) 0x7f7c9809c6e0 con 0x7f7ca8103470 2024-04-26T17:54:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:54:16.773 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","created":"2024-04-26T17:44:48.618250+0000","modified":"2024-04-26T17:53:49.754504+0000","last_up_change":"2024-04-26T17:52:00.798886+0000","last_in_change":"2024-04-26T17:51:40.613593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T17:48:12.307250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T17:52:58.650338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T17:53:02.488144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f684ee5-eeee-454a-a00b-e5d6d2901572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6803","nonce":1957398328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6805","nonce":1957398328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6809","nonce":1957398328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6807","nonce":1957398328}]},"public_addr":"172.21.15.31:6803/1957398328","cluster_addr":"172.21.15.31:6805/1957398328","heartbeat_back_addr":"172.21.15.31:6809/1957398328","heartbeat_front_addr":"172.21.15.31:6807/1957398328","state":["exists","up"]},{"osd":1,"uuid":"08be6b26-cb60-41f7-be8b-01a0c93ebafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6801","nonce":4209866427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6803","nonce":4209866427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6807","nonce":4209866427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6805","nonce":4209866427}]},"public_addr":"172.21.15.97:6801/4209866427","cluster_addr":"172.21.15.97:6803/4209866427","heartbeat_back_addr":"172.21.15.97:6807/4209866427","heartbeat_front_addr":"172.21.15.97:6805/4209866427","state":["exists","up"]},{"osd":2,"uuid":"498216b1-a0ac-4923-ade8-ce50fb97d472","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6801","nonce":223535147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6803","nonce":223535147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6807","nonce":223535147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6805","nonce":223535147}]},"public_addr":"172.21.15.186:6801/223535147","cluster_addr":"172.21.15.186:6803/223535147","heartbeat_back_addr":"172.21.15.186:6807/223535147","heartbeat_front_addr":"172.21.15.186:6805/223535147","state":["exists","up"]},{"osd":3,"uuid":"568a352d-a430-4f0f-9a49-6f992e0ceae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6811","nonce":2640112341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6813","nonce":2640112341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6817","nonce":2640112341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6815","nonce":2640112341}]},"public_addr":"172.21.15.31:6811/2640112341","cluster_addr":"172.21.15.31:6813/2640112341","heartbeat_back_addr":"172.21.15.31:6817/2640112341","heartbeat_front_addr":"172.21.15.31:6815/2640112341","state":["exists","up"]},{"osd":4,"uuid":"112ccbf7-24e8-473c-9852-2e7167dd3108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6809","nonce":1359263143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6811","nonce":1359263143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6815","nonce":1359263143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6813","nonce":1359263143}]},"public_addr":"172.21.15.97:6809/1359263143","cluster_addr":"172.21.15.97:6811/1359263143","heartbeat_back_addr":"172.21.15.97:6815/1359263143","heartbeat_front_addr":"172.21.15.97:6813/1359263143","state":["exists","up"]},{"osd":5,"uuid":"eb17759e-1ad8-4bda-9f61-54d2a9781535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6809","nonce":2006134626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6811","nonce":2006134626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6815","nonce":2006134626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6813","nonce":2006134626}]},"public_addr":"172.21.15.186:6809/2006134626","cluster_addr":"172.21.15.186:6811/2006134626","heartbeat_back_addr":"172.21.15.186:6815/2006134626","heartbeat_front_addr":"172.21.15.186:6813/2006134626","state":["exists","up"]},{"osd":6,"uuid":"f1d11e38-40e9-4db4-868f-145f1ffc0d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6819","nonce":2955759395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6821","nonce":2955759395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6825","nonce":2955759395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6823","nonce":2955759395}]},"public_addr":"172.21.15.31:6819/2955759395","cluster_addr":"172.21.15.31:6821/2955759395","heartbeat_back_addr":"172.21.15.31:6825/2955759395","heartbeat_front_addr":"172.21.15.31:6823/2955759395","state":["exists","up"]},{"osd":7,"uuid":"4e95f728-23ff-4bca-9b94-afde2662a5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6817","nonce":3940083075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6819","nonce":3940083075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6823","nonce":3940083075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6821","nonce":3940083075}]},"public_addr":"172.21.15.97:6817/3940083075","cluster_addr":"172.21.15.97:6819/3940083075","heartbeat_back_addr":"172.21.15.97:6823/3940083075","heartbeat_front_addr":"172.21.15.97:6821/3940083075","state":["exists","up"]},{"osd":8,"uuid":"5f91c915-2568-4d20-b645-00dad2efe952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6817","nonce":3193558170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6819","nonce":3193558170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6823","nonce":3193558170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6821","nonce":3193558170}]},"public_addr":"172.21.15.186:6817/3193558170","cluster_addr":"172.21.15.186:6819/3193558170","heartbeat_back_addr":"172.21.15.186:6823/3193558170","heartbeat_front_addr":"172.21.15.186:6821/3193558170","state":["exists","up"]},{"osd":9,"uuid":"12fda87d-830a-461f-a6cb-4699c52dc5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6827","nonce":3660801905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6829","nonce":3660801905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6833","nonce":3660801905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6831","nonce":3660801905}]},"public_addr":"172.21.15.31:6827/3660801905","cluster_addr":"172.21.15.31:6829/3660801905","heartbeat_back_addr":"172.21.15.31:6833/3660801905","heartbeat_front_addr":"172.21.15.31:6831/3660801905","state":["exists","up"]},{"osd":10,"uuid":"92a639b3-918b-485c-bd30-258b4224993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6825","nonce":4097344903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6827","nonce":4097344903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6831","nonce":4097344903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6829","nonce":4097344903}]},"public_addr":"172.21.15.97:6825/4097344903","cluster_addr":"172.21.15.97:6827/4097344903","heartbeat_back_addr":"172.21.15.97:6831/4097344903","heartbeat_front_addr":"172.21.15.97:6829/4097344903","state":["exists","up"]},{"osd":11,"uuid":"842aa599-d21d-4e4a-b872-e9ac20d8c0ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6825","nonce":1063973352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6827","nonce":1063973352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6831","nonce":1063973352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6829","nonce":1063973352}]},"public_addr":"172.21.15.186:6825/1063973352","cluster_addr":"172.21.15.186:6827/1063973352","heartbeat_back_addr":"172.21.15.186:6831/1063973352","heartbeat_front_addr":"172.21.15.186:6829/1063973352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:21.305331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:44.877932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:08.784665+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:34.381762+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:58.999358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:23.620237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:49.961719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:15.363702+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:40.599703+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:07.391645+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:32.800851+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:59.384148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/718966776":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/2035551671":"2024-04-27T17:45:48.218816+0000","172.21.15.31:6801/153341213":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/810461143":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/3148082955":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6801/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/288292084":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/3101596726":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/153341213":"2024-04-27T17:45:48.218816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T17:54:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.775+0000 7f7cacd75700 1 -- 172.21.15.31:0/3208424461 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f7c900a3cd0 msgr2=0x7f7c900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.775+0000 7f7cacd75700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f7c900a3cd0 0x7f7c900a6190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00b5c0 tx=0x7f7c9c01a040 comp rx=0 tx=0).stop 2024-04-26T17:54:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.775+0000 7f7cacd75700 1 -- 172.21.15.31:0/3208424461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ca8103470 msgr2=0x7f7ca8075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.775+0000 7f7cacd75700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ca8103470 0x7f7ca8075720 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800b530 tx=0x7f7c9800b8f0 comp rx=0 tx=0).stop 2024-04-26T17:54:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.775+0000 7f7cacd75700 1 -- 172.21.15.31:0/3208424461 shutdown_connections 2024-04-26T17:54:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.775+0000 7f7cacd75700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f7c900a3cd0 0x7f7c900a6190 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.775+0000 7f7cacd75700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ca81094c0 0x7f7ca80761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.775+0000 7f7cacd75700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ca8103470 0x7f7ca8075720 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.775+0000 7f7cacd75700 1 --2- 172.21.15.31:0/3208424461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ca8103e30 0x7f7ca8075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.775+0000 7f7cacd75700 1 -- 172.21.15.31:0/3208424461 >> 172.21.15.31:0/3208424461 conn(0x7f7ca807a580 msgr2=0x7f7ca8102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.775+0000 7f7cacd75700 1 -- 172.21.15.31:0/3208424461 shutdown_connections 2024-04-26T17:54:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:16.775+0000 7f7cacd75700 1 -- 172.21.15.31:0/3208424461 wait complete. 2024-04-26T17:54:17.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:17 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3208424461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:54:17.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:17 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3208424461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:54:17.420 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T17:54:17.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T17:54:17.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:17 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3208424461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:54:17.669 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:18.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:18 smithi031 ceph-mon[29164]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:54:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:18 smithi097 ceph-mon[32419]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:54:18.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:18 smithi186 ceph-mon[32797]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:54:19.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.383+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/2573295022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e41034a0 msgr2=0x7fe8e4103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:19.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.383+0000 7fe8eb5ac700 1 --2- 172.21.15.31:0/2573295022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e41034a0 0x7fe8e4103920 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4009990 tx=0x7fe8d4009ca0 comp rx=0 tx=0).stop 2024-04-26T17:54:19.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/2573295022 shutdown_connections 2024-04-26T17:54:19.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 --2- 172.21.15.31:0/2573295022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8e4103e60 0x7fe8e41102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 --2- 172.21.15.31:0/2573295022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8e41094e0 0x7fe8e41098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 --2- 172.21.15.31:0/2573295022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e41034a0 0x7fe8e4103920 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/2573295022 >> 172.21.15.31:0/2573295022 conn(0x7fe8e407a590 msgr2=0x7fe8e407a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/2573295022 shutdown_connections 2024-04-26T17:54:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/2573295022 wait complete. 2024-04-26T17:54:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 Processor -- start 2024-04-26T17:54:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 -- start start 2024-04-26T17:54:19.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8e41034a0 0x7fe8e419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:19.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8e4103e60 0x7fe8e419fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:19.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e41094e0 0x7fe8e41999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:19.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e4113660 con 0x7fe8e41094e0 2024-04-26T17:54:19.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e41134e0 con 0x7fe8e41034a0 2024-04-26T17:54:19.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.384+0000 7fe8eb5ac700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e41137e0 con 0x7fe8e4103e60 2024-04-26T17:54:19.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8e9b49700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e41094e0 0x7fe8e41999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:19.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8e9b49700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e41094e0 0x7fe8e41999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57098/0 (socket says 172.21.15.31:57098) 2024-04-26T17:54:19.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8e9b49700 1 -- 172.21.15.31:0/287593708 learned_addr learned my addr 172.21.15.31:0/287593708 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:19.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8e8b47700 1 --2- 172.21.15.31:0/287593708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8e4103e60 0x7fe8e419fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:19.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8e9348700 1 --2- 172.21.15.31:0/287593708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8e41034a0 0x7fe8e419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:19.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8e9b49700 1 -- 172.21.15.31:0/287593708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8e4103e60 msgr2=0x7fe8e419fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:19.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8e9b49700 1 --2- 172.21.15.31:0/287593708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8e4103e60 0x7fe8e419fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:19.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8e9b49700 1 -- 172.21.15.31:0/287593708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8e41034a0 msgr2=0x7fe8e419f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:19.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8e9b49700 1 --2- 172.21.15.31:0/287593708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8e41034a0 0x7fe8e419f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:19.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8e9b49700 1 -- 172.21.15.31:0/287593708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e419a140 con 0x7fe8e41094e0 2024-04-26T17:54:19.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8e9b49700 1 --2- 172.21.15.31:0/287593708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e41094e0 0x7fe8e41999e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc00ada0 tx=0x7fe8dc00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:19.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8da7fc700 1 -- 172.21.15.31:0/287593708 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc017020 con 0x7fe8e41094e0 2024-04-26T17:54:19.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8da7fc700 1 -- 172.21.15.31:0/287593708 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8dc00ce30 con 0x7fe8e41094e0 2024-04-26T17:54:19.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/287593708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e419a430 con 0x7fe8e41094e0 2024-04-26T17:54:19.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8da7fc700 1 -- 172.21.15.31:0/287593708 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc005350 con 0x7fe8e41094e0 2024-04-26T17:54:19.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.385+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/287593708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8e41a66a0 con 0x7fe8e41094e0 2024-04-26T17:54:19.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.386+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/287593708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8e406a110 con 0x7fe8e41094e0 2024-04-26T17:54:19.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.388+0000 7fe8da7fc700 1 -- 172.21.15.31:0/287593708 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe8dc00c980 con 0x7fe8e41094e0 2024-04-26T17:54:19.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.388+0000 7fe8da7fc700 1 --2- 172.21.15.31:0/287593708 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe8d00a3d50 0x7fe8d00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:19.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.388+0000 7fe8e9348700 1 --2- 172.21.15.31:0/287593708 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe8d00a3d50 0x7fe8d00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:19.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.389+0000 7fe8da7fc700 1 -- 172.21.15.31:0/287593708 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fe8dc0d0240 con 0x7fe8e41094e0 2024-04-26T17:54:19.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.389+0000 7fe8e9348700 1 --2- 172.21.15.31:0/287593708 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe8d00a3d50 0x7fe8d00a6210 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0006fd0 tx=0x7fe8e0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:19.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.394+0000 7fe8da7fc700 1 -- 172.21.15.31:0/287593708 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe8dc021020 con 0x7fe8e41094e0 2024-04-26T17:54:19.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.606+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/287593708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe8e402e490 con 0x7fe8e41094e0 2024-04-26T17:54:19.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.608+0000 7fe8da7fc700 1 -- 172.21.15.31:0/287593708 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22577 (secure 0 0 0) 0x7fe8dc01a310 con 0x7fe8e41094e0 2024-04-26T17:54:19.608 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:54:19.609 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","created":"2024-04-26T17:44:48.618250+0000","modified":"2024-04-26T17:53:49.754504+0000","last_up_change":"2024-04-26T17:52:00.798886+0000","last_in_change":"2024-04-26T17:51:40.613593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T17:48:12.307250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T17:52:58.650338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T17:53:02.488144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f684ee5-eeee-454a-a00b-e5d6d2901572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6803","nonce":1957398328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6805","nonce":1957398328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6809","nonce":1957398328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6807","nonce":1957398328}]},"public_addr":"172.21.15.31:6803/1957398328","cluster_addr":"172.21.15.31:6805/1957398328","heartbeat_back_addr":"172.21.15.31:6809/1957398328","heartbeat_front_addr":"172.21.15.31:6807/1957398328","state":["exists","up"]},{"osd":1,"uuid":"08be6b26-cb60-41f7-be8b-01a0c93ebafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6801","nonce":4209866427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6803","nonce":4209866427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6807","nonce":4209866427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6805","nonce":4209866427}]},"public_addr":"172.21.15.97:6801/4209866427","cluster_addr":"172.21.15.97:6803/4209866427","heartbeat_back_addr":"172.21.15.97:6807/4209866427","heartbeat_front_addr":"172.21.15.97:6805/4209866427","state":["exists","up"]},{"osd":2,"uuid":"498216b1-a0ac-4923-ade8-ce50fb97d472","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6801","nonce":223535147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6803","nonce":223535147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6807","nonce":223535147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6805","nonce":223535147}]},"public_addr":"172.21.15.186:6801/223535147","cluster_addr":"172.21.15.186:6803/223535147","heartbeat_back_addr":"172.21.15.186:6807/223535147","heartbeat_front_addr":"172.21.15.186:6805/223535147","state":["exists","up"]},{"osd":3,"uuid":"568a352d-a430-4f0f-9a49-6f992e0ceae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6811","nonce":2640112341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6813","nonce":2640112341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6817","nonce":2640112341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6815","nonce":2640112341}]},"public_addr":"172.21.15.31:6811/2640112341","cluster_addr":"172.21.15.31:6813/2640112341","heartbeat_back_addr":"172.21.15.31:6817/2640112341","heartbeat_front_addr":"172.21.15.31:6815/2640112341","state":["exists","up"]},{"osd":4,"uuid":"112ccbf7-24e8-473c-9852-2e7167dd3108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6809","nonce":1359263143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6811","nonce":1359263143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6815","nonce":1359263143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6813","nonce":1359263143}]},"public_addr":"172.21.15.97:6809/1359263143","cluster_addr":"172.21.15.97:6811/1359263143","heartbeat_back_addr":"172.21.15.97:6815/1359263143","heartbeat_front_addr":"172.21.15.97:6813/1359263143","state":["exists","up"]},{"osd":5,"uuid":"eb17759e-1ad8-4bda-9f61-54d2a9781535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6809","nonce":2006134626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6811","nonce":2006134626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6815","nonce":2006134626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6813","nonce":2006134626}]},"public_addr":"172.21.15.186:6809/2006134626","cluster_addr":"172.21.15.186:6811/2006134626","heartbeat_back_addr":"172.21.15.186:6815/2006134626","heartbeat_front_addr":"172.21.15.186:6813/2006134626","state":["exists","up"]},{"osd":6,"uuid":"f1d11e38-40e9-4db4-868f-145f1ffc0d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6819","nonce":2955759395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6821","nonce":2955759395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6825","nonce":2955759395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6823","nonce":2955759395}]},"public_addr":"172.21.15.31:6819/2955759395","cluster_addr":"172.21.15.31:6821/2955759395","heartbeat_back_addr":"172.21.15.31:6825/2955759395","heartbeat_front_addr":"172.21.15.31:6823/2955759395","state":["exists","up"]},{"osd":7,"uuid":"4e95f728-23ff-4bca-9b94-afde2662a5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6817","nonce":3940083075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6819","nonce":3940083075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6823","nonce":3940083075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6821","nonce":3940083075}]},"public_addr":"172.21.15.97:6817/3940083075","cluster_addr":"172.21.15.97:6819/3940083075","heartbeat_back_addr":"172.21.15.97:6823/3940083075","heartbeat_front_addr":"172.21.15.97:6821/3940083075","state":["exists","up"]},{"osd":8,"uuid":"5f91c915-2568-4d20-b645-00dad2efe952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6817","nonce":3193558170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6819","nonce":3193558170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6823","nonce":3193558170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6821","nonce":3193558170}]},"public_addr":"172.21.15.186:6817/3193558170","cluster_addr":"172.21.15.186:6819/3193558170","heartbeat_back_addr":"172.21.15.186:6823/3193558170","heartbeat_front_addr":"172.21.15.186:6821/3193558170","state":["exists","up"]},{"osd":9,"uuid":"12fda87d-830a-461f-a6cb-4699c52dc5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6827","nonce":3660801905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6829","nonce":3660801905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6833","nonce":3660801905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6831","nonce":3660801905}]},"public_addr":"172.21.15.31:6827/3660801905","cluster_addr":"172.21.15.31:6829/3660801905","heartbeat_back_addr":"172.21.15.31:6833/3660801905","heartbeat_front_addr":"172.21.15.31:6831/3660801905","state":["exists","up"]},{"osd":10,"uuid":"92a639b3-918b-485c-bd30-258b4224993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6825","nonce":4097344903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6827","nonce":4097344903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6831","nonce":4097344903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6829","nonce":4097344903}]},"public_addr":"172.21.15.97:6825/4097344903","cluster_addr":"172.21.15.97:6827/4097344903","heartbeat_back_addr":"172.21.15.97:6831/4097344903","heartbeat_front_addr":"172.21.15.97:6829/4097344903","state":["exists","up"]},{"osd":11,"uuid":"842aa599-d21d-4e4a-b872-e9ac20d8c0ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6825","nonce":1063973352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6827","nonce":1063973352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6831","nonce":1063973352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6829","nonce":1063973352}]},"public_addr":"172.21.15.186:6825/1063973352","cluster_addr":"172.21.15.186:6827/1063973352","heartbeat_back_addr":"172.21.15.186:6831/1063973352","heartbeat_front_addr":"172.21.15.186:6829/1063973352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:21.305331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:44.877932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:08.784665+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:34.381762+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:58.999358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:23.620237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:49.961719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:15.363702+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:40.599703+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:07.391645+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:32.800851+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:59.384148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/718966776":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/2035551671":"2024-04-27T17:45:48.218816+0000","172.21.15.31:6801/153341213":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/810461143":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/3148082955":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6801/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/288292084":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/3101596726":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/153341213":"2024-04-27T17:45:48.218816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T17:54:19.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.610+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/287593708 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe8d00a3d50 msgr2=0x7fe8d00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:19.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.610+0000 7fe8eb5ac700 1 --2- 172.21.15.31:0/287593708 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe8d00a3d50 0x7fe8d00a6210 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0006fd0 tx=0x7fe8e0008040 comp rx=0 tx=0).stop 2024-04-26T17:54:19.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.610+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/287593708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e41094e0 msgr2=0x7fe8e41999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:19.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.610+0000 7fe8eb5ac700 1 --2- 172.21.15.31:0/287593708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e41094e0 0x7fe8e41999e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc00ada0 tx=0x7fe8dc00c330 comp rx=0 tx=0).stop 2024-04-26T17:54:19.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.610+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/287593708 shutdown_connections 2024-04-26T17:54:19.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.610+0000 7fe8eb5ac700 1 --2- 172.21.15.31:0/287593708 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe8d00a3d50 0x7fe8d00a6210 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:19.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.610+0000 7fe8eb5ac700 1 --2- 172.21.15.31:0/287593708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe8e41034a0 0x7fe8e419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:19.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.610+0000 7fe8eb5ac700 1 --2- 172.21.15.31:0/287593708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e41094e0 0x7fe8e41999e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:19.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.610+0000 7fe8eb5ac700 1 --2- 172.21.15.31:0/287593708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8e4103e60 0x7fe8e419fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:19.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.610+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/287593708 >> 172.21.15.31:0/287593708 conn(0x7fe8e407a590 msgr2=0x7fe8e41022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:19.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.610+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/287593708 shutdown_connections 2024-04-26T17:54:19.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:19.610+0000 7fe8eb5ac700 1 -- 172.21.15.31:0/287593708 wait complete. 2024-04-26T17:54:20.293 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T17:54:20.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T17:54:20.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T17:54:20.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-26T17:54:20.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-26T17:54:20.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-26T17:54:20.297 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-26T17:54:20.297 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-26T17:54:20.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-26T17:54:20.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-26T17:54:20.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-26T17:54:20.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-26T17:54:20.339 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:20 smithi031 ceph-mon[29164]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:20.339 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:20 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/287593708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:54:20.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:20 smithi097 ceph-mon[32419]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:20.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:20 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/287593708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:54:20.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:20 smithi186 ceph-mon[32797]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:20.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:20 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/287593708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:54:20.986 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:20.990 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:21.317 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:21.317 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:21.318 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:21.318 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:21.322 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:21.323 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:21.330 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:21.331 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:22.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:22 smithi097 ceph-mon[32419]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:22.469 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:22 smithi031 ceph-mon[29164]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:22.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:22 smithi186 ceph-mon[32797]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:22.597 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:22.598 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:24.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:24 smithi031 ceph-mon[29164]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:54:24.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:24 smithi097 ceph-mon[32419]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:54:24.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:24 smithi186 ceph-mon[32797]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:54:26.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:26 smithi031 ceph-mon[29164]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:26.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:26 smithi097 ceph-mon[32419]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:26.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:26 smithi186 ceph-mon[32797]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:28.280 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:28 smithi031 ceph-mon[29164]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:54:28.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:28 smithi097 ceph-mon[32419]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:54:28.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:28 smithi186 ceph-mon[32797]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:54:30.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.073+0000 7f12d979c700 1 -- 172.21.15.31:0/2757517072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12d41025b0 msgr2=0x7f12d4102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:30.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.073+0000 7f12d979c700 1 --2- 172.21.15.31:0/2757517072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12d41025b0 0x7f12d4102990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f12c40099c0 tx=0x7f12c4009cd0 comp rx=0 tx=0).stop 2024-04-26T17:54:30.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.073+0000 7f12d979c700 1 -- 172.21.15.31:0/2757517072 shutdown_connections 2024-04-26T17:54:30.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.073+0000 7f12d979c700 1 --2- 172.21.15.31:0/2757517072 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12d4107640 0x7f12d410f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:30.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.073+0000 7f12d979c700 1 --2- 172.21.15.31:0/2757517072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12d41025b0 0x7f12d4102990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:30.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.073+0000 7f12d979c700 1 --2- 172.21.15.31:0/2757517072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12d4102f60 0x7f12d4106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:30.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.073+0000 7f12d979c700 1 -- 172.21.15.31:0/2757517072 >> 172.21.15.31:0/2757517072 conn(0x7f12d40792d0 msgr2=0x7f12d40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:30.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.073+0000 7f12d979c700 1 -- 172.21.15.31:0/2757517072 shutdown_connections 2024-04-26T17:54:30.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.073+0000 7f12d979c700 1 -- 172.21.15.31:0/2757517072 wait complete. 2024-04-26T17:54:30.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.074+0000 7f12d979c700 1 Processor -- start 2024-04-26T17:54:30.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.074+0000 7f12d979c700 1 -- start start 2024-04-26T17:54:30.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.074+0000 7f12d979c700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12d41025b0 0x7f12d419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:30.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.074+0000 7f12d979c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12d4102f60 0x7f12d419fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:30.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.074+0000 7f12d979c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12d4107640 0x7f12d4199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:30.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.074+0000 7f12d979c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f12d4112550 con 0x7f12d4107640 2024-04-26T17:54:30.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.074+0000 7f12d979c700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f12d41123d0 con 0x7f12d41025b0 2024-04-26T17:54:30.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.074+0000 7f12d979c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f12d41126d0 con 0x7f12d4102f60 2024-04-26T17:54:30.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.074+0000 7f12d37fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12d4107640 0x7f12d4199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:30.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.074+0000 7f12d37fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12d4107640 0x7f12d4199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52192/0 (socket says 172.21.15.31:52192) 2024-04-26T17:54:30.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.074+0000 7f12d37fe700 1 -- 172.21.15.31:0/2915482606 learned_addr learned my addr 172.21.15.31:0/2915482606 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:30.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.075+0000 7f12d27fc700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12d4102f60 0x7f12d419fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:30.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.075+0000 7f12d2ffd700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12d41025b0 0x7f12d419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:30.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.075+0000 7f12d37fe700 1 -- 172.21.15.31:0/2915482606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12d4102f60 msgr2=0x7f12d419fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:30.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.075+0000 7f12d37fe700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12d4102f60 0x7f12d419fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:30.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.075+0000 7f12d37fe700 1 -- 172.21.15.31:0/2915482606 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12d41025b0 msgr2=0x7f12d419f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:30.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.075+0000 7f12d37fe700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12d41025b0 0x7f12d419f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:30.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.075+0000 7f12d37fe700 1 -- 172.21.15.31:0/2915482606 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d419a0f0 con 0x7f12d4107640 2024-04-26T17:54:30.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.075+0000 7f12d2ffd700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12d41025b0 0x7f12d419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:54:30.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.075+0000 7f12d37fe700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12d4107640 0x7f12d4199990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f12cc00b970 tx=0x7f12cc00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:30.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.075+0000 7f12bbfff700 1 -- 172.21.15.31:0/2915482606 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12cc019070 con 0x7f12d4107640 2024-04-26T17:54:30.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.075+0000 7f12d979c700 1 -- 172.21.15.31:0/2915482606 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12d419a3e0 con 0x7f12d4107640 2024-04-26T17:54:30.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.075+0000 7f12bbfff700 1 -- 172.21.15.31:0/2915482606 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12cc00cde0 con 0x7f12d4107640 2024-04-26T17:54:30.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.075+0000 7f12bbfff700 1 -- 172.21.15.31:0/2915482606 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12cc007370 con 0x7f12d4107640 2024-04-26T17:54:30.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.075+0000 7f12d979c700 1 -- 172.21.15.31:0/2915482606 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12d41a66a0 con 0x7f12d4107640 2024-04-26T17:54:30.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.076+0000 7f12d979c700 1 -- 172.21.15.31:0/2915482606 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f12d4063bb0 con 0x7f12d4107640 2024-04-26T17:54:30.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.077+0000 7f12bbfff700 1 -- 172.21.15.31:0/2915482606 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f12cc01c070 con 0x7f12d4107640 2024-04-26T17:54:30.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.078+0000 7f12bbfff700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f12bc0a3cd0 0x7f12bc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:30.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.078+0000 7f12d2ffd700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f12bc0a3cd0 0x7f12bc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:30.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.078+0000 7f12bbfff700 1 -- 172.21.15.31:0/2915482606 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f12cc0d0350 con 0x7f12d4107640 2024-04-26T17:54:30.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.079+0000 7f12bbfff700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328] conn(0x7f12bc0ad790 0x7f12bc0afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:30.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.079+0000 7f12bbfff700 1 -- 172.21.15.31:0/2915482606 --> [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12cc004860 con 0x7f12bc0ad790 2024-04-26T17:54:30.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.079+0000 7f12bbfff700 1 -- 172.21.15.31:0/2915482606 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f12cc0d07a0 con 0x7f12d4107640 2024-04-26T17:54:30.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.079+0000 7f12d27fc700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328] conn(0x7f12bc0ad790 0x7f12bc0afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:30.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.079+0000 7f12d2ffd700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f12bc0a3cd0 0x7f12bc0a6190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f12c400b580 tx=0x7f12c4009530 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:30.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.079+0000 7f12d27fc700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328] conn(0x7f12bc0ad790 0x7f12bc0afc30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:30.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.080+0000 7f12bbfff700 1 -- 172.21.15.31:0/2915482606 <== osd.0 v2:172.21.15.31:6802/1957398328 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f12cc004860 con 0x7f12bc0ad790 2024-04-26T17:54:30.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.097+0000 7f12d979c700 1 -- 172.21.15.31:0/2915482606 --> [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f12d4063fb0 con 0x7f12bc0ad790 2024-04-26T17:54:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.097+0000 7f12bbfff700 1 -- 172.21.15.31:0/2915482606 <== osd.0 v2:172.21.15.31:6802/1957398328 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f12d4063fb0 con 0x7f12bc0ad790 2024-04-26T17:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.097+0000 7f12d979c700 1 -- 172.21.15.31:0/2915482606 >> [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328] conn(0x7f12bc0ad790 msgr2=0x7f12bc0afc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.097+0000 7f12d979c700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328] conn(0x7f12bc0ad790 0x7f12bc0afc30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.097+0000 7f12d979c700 1 -- 172.21.15.31:0/2915482606 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f12bc0a3cd0 msgr2=0x7f12bc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.097+0000 7f12d979c700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f12bc0a3cd0 0x7f12bc0a6190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f12c400b580 tx=0x7f12c4009530 comp rx=0 tx=0).stop 2024-04-26T17:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.097+0000 7f12d979c700 1 -- 172.21.15.31:0/2915482606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12d4107640 msgr2=0x7f12d4199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.097+0000 7f12d979c700 1 --2- 172.21.15.31:0/2915482606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12d4107640 0x7f12d4199990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f12cc00b970 tx=0x7f12cc00bc80 comp rx=0 tx=0).stop 2024-04-26T17:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.098+0000 7f12d37fe700 1 -- 172.21.15.31:0/2915482606 reap_dead start 2024-04-26T17:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.098+0000 7f12d979c700 1 -- 172.21.15.31:0/2915482606 shutdown_connections 2024-04-26T17:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.098+0000 7f12d979c700 1 -- 172.21.15.31:0/2915482606 >> 172.21.15.31:0/2915482606 conn(0x7f12d40792d0 msgr2=0x7f12d406c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.098+0000 7f12d979c700 1 -- 172.21.15.31:0/2915482606 shutdown_connections 2024-04-26T17:54:30.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:30.098+0000 7f12d979c700 1 -- 172.21.15.31:0/2915482606 wait complete. 2024-04-26T17:54:30.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:30 smithi097 ceph-mon[32419]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:30.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:30 smithi186 ceph-mon[32797]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:30.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:30 smithi031 ceph-mon[29164]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:32.135 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:32 smithi031 ceph-mon[29164]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:32.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:32 smithi097 ceph-mon[32419]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:32.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:32 smithi186 ceph-mon[32797]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:54:33.026 INFO:teuthology.orchestra.run.smithi031.stdout:34359738457 2024-04-26T17:54:33.027 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-26T17:54:33.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.202+0000 7f52a2a06700 1 -- 172.21.15.31:0/3545768740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f529c06b130 msgr2=0x7f529c06b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:33.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.202+0000 7f52a2a06700 1 --2- 172.21.15.31:0/3545768740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f529c06b130 0x7f529c06b590 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5290009990 tx=0x7f5290009ca0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.202+0000 7f52a2a06700 1 -- 172.21.15.31:0/3545768740 shutdown_connections 2024-04-26T17:54:33.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.202+0000 7f52a2a06700 1 --2- 172.21.15.31:0/3545768740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f529c1094a0 0x7f529c109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.202+0000 7f52a2a06700 1 --2- 172.21.15.31:0/3545768740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f529c06bad0 0x7f529c1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.202+0000 7f52a2a06700 1 --2- 172.21.15.31:0/3545768740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f529c06b130 0x7f529c06b590 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.202+0000 7f52a2a06700 1 -- 172.21.15.31:0/3545768740 >> 172.21.15.31:0/3545768740 conn(0x7f529c0792d0 msgr2=0x7f529c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:33.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.202+0000 7f52a2a06700 1 -- 172.21.15.31:0/3545768740 shutdown_connections 2024-04-26T17:54:33.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.202+0000 7f52a2a06700 1 -- 172.21.15.31:0/3545768740 wait complete. 2024-04-26T17:54:33.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.203+0000 7f52a2a06700 1 Processor -- start 2024-04-26T17:54:33.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.203+0000 7f52a2a06700 1 -- start start 2024-04-26T17:54:33.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.203+0000 7f52a2a06700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f529c06b130 0x7f529c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:33.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.203+0000 7f52a2a06700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f529c06bad0 0x7f529c19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:33.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.203+0000 7f52a2a06700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f529c1094a0 0x7f529c199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:33.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.203+0000 7f52a2a06700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f529c1135d0 con 0x7f529c06b130 2024-04-26T17:54:33.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.203+0000 7f52a2a06700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f529c113450 con 0x7f529c1094a0 2024-04-26T17:54:33.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.203+0000 7f52a2a06700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f529c113750 con 0x7f529c06bad0 2024-04-26T17:54:33.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.203+0000 7f529bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f529c06b130 0x7f529c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:33.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.203+0000 7f529bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f529c06b130 0x7f529c19f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52214/0 (socket says 172.21.15.31:52214) 2024-04-26T17:54:33.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.203+0000 7f529bfff700 1 -- 172.21.15.31:0/2040025787 learned_addr learned my addr 172.21.15.31:0/2040025787 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:33.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.203+0000 7f529b7fe700 1 --2- 172.21.15.31:0/2040025787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f529c06bad0 0x7f529c19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:33.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.204+0000 7f52a0fa3700 1 --2- 172.21.15.31:0/2040025787 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f529c1094a0 0x7f529c199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:33.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.204+0000 7f529bfff700 1 -- 172.21.15.31:0/2040025787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f529c06bad0 msgr2=0x7f529c19fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:33.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.204+0000 7f529bfff700 1 --2- 172.21.15.31:0/2040025787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f529c06bad0 0x7f529c19fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.204+0000 7f529bfff700 1 -- 172.21.15.31:0/2040025787 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f529c1094a0 msgr2=0x7f529c199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:33.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.204+0000 7f529bfff700 1 --2- 172.21.15.31:0/2040025787 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f529c1094a0 0x7f529c199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.204+0000 7f529bfff700 1 -- 172.21.15.31:0/2040025787 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529c19a090 con 0x7f529c06b130 2024-04-26T17:54:33.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.204+0000 7f529bfff700 1 --2- 172.21.15.31:0/2040025787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f529c06b130 0x7f529c19f770 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f528c00c920 tx=0x7f528c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:33.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.204+0000 7f52997fa700 1 -- 172.21.15.31:0/2040025787 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c007b00 con 0x7f529c06b130 2024-04-26T17:54:33.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.204+0000 7f52a2a06700 1 -- 172.21.15.31:0/2040025787 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529c19a380 con 0x7f529c06b130 2024-04-26T17:54:33.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.204+0000 7f52a2a06700 1 -- 172.21.15.31:0/2040025787 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f529c1a6640 con 0x7f529c06b130 2024-04-26T17:54:33.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.204+0000 7f52997fa700 1 -- 172.21.15.31:0/2040025787 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f528c004ce0 con 0x7f529c06b130 2024-04-26T17:54:33.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.204+0000 7f52997fa700 1 -- 172.21.15.31:0/2040025787 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c01d060 con 0x7f529c06b130 2024-04-26T17:54:33.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.205+0000 7f52a2a06700 1 -- 172.21.15.31:0/2040025787 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5288001030 con 0x7f529c06b130 2024-04-26T17:54:33.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.207+0000 7f52997fa700 1 -- 172.21.15.31:0/2040025787 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f528c005440 con 0x7f529c06b130 2024-04-26T17:54:33.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.208+0000 7f52997fa700 1 --2- 172.21.15.31:0/2040025787 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f52840a3dd0 0x7f52840a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:33.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.208+0000 7f529b7fe700 1 --2- 172.21.15.31:0/2040025787 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f52840a3dd0 0x7f52840a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:33.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.208+0000 7f52997fa700 1 -- 172.21.15.31:0/2040025787 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f528c0d0360 con 0x7f529c06b130 2024-04-26T17:54:33.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.208+0000 7f52997fa700 1 --2- 172.21.15.31:0/2040025787 >> [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147] conn(0x7f52840ad890 0x7f52840afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:33.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.208+0000 7f52997fa700 1 -- 172.21.15.31:0/2040025787 --> [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f528c012ec0 con 0x7f52840ad890 2024-04-26T17:54:33.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.208+0000 7f52997fa700 1 -- 172.21.15.31:0/2040025787 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f528c0d0750 con 0x7f529c06b130 2024-04-26T17:54:33.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.208+0000 7f529b7fe700 1 --2- 172.21.15.31:0/2040025787 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f52840a3dd0 0x7f52840a6290 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f529000b5c0 tx=0x7f5290005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:33.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.209+0000 7f52a0fa3700 1 --2- 172.21.15.31:0/2040025787 >> [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147] conn(0x7f52840ad890 0x7f52840afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:33.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.209+0000 7f52a0fa3700 1 --2- 172.21.15.31:0/2040025787 >> [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147] conn(0x7f52840ad890 0x7f52840afd30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:33.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.210+0000 7f52997fa700 1 -- 172.21.15.31:0/2040025787 <== osd.2 v2:172.21.15.186:6800/223535147 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f528c012ec0 con 0x7f52840ad890 2024-04-26T17:54:33.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.228+0000 7f52a2a06700 1 -- 172.21.15.31:0/2040025787 --> [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5288002e60 con 0x7f52840ad890 2024-04-26T17:54:33.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.229+0000 7f52997fa700 1 -- 172.21.15.31:0/2040025787 <== osd.2 v2:172.21.15.186:6800/223535147 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5288002e60 con 0x7f52840ad890 2024-04-26T17:54:33.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.229+0000 7f52a2a06700 1 -- 172.21.15.31:0/2040025787 >> [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147] conn(0x7f52840ad890 msgr2=0x7f52840afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:33.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.229+0000 7f52a2a06700 1 --2- 172.21.15.31:0/2040025787 >> [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147] conn(0x7f52840ad890 0x7f52840afd30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.229+0000 7f52a2a06700 1 -- 172.21.15.31:0/2040025787 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f52840a3dd0 msgr2=0x7f52840a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:33.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.229+0000 7f52a2a06700 1 --2- 172.21.15.31:0/2040025787 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f52840a3dd0 0x7f52840a6290 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f529000b5c0 tx=0x7f5290005960 comp rx=0 tx=0).stop 2024-04-26T17:54:33.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.229+0000 7f52a2a06700 1 -- 172.21.15.31:0/2040025787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f529c06b130 msgr2=0x7f529c19f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:33.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.229+0000 7f52a2a06700 1 --2- 172.21.15.31:0/2040025787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f529c06b130 0x7f529c19f770 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f528c00c920 tx=0x7f528c00cc30 comp rx=0 tx=0).stop 2024-04-26T17:54:33.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.229+0000 7f52a0fa3700 1 -- 172.21.15.31:0/2040025787 reap_dead start 2024-04-26T17:54:33.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.229+0000 7f52a2a06700 1 -- 172.21.15.31:0/2040025787 shutdown_connections 2024-04-26T17:54:33.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.229+0000 7f52a2a06700 1 -- 172.21.15.31:0/2040025787 >> 172.21.15.31:0/2040025787 conn(0x7f529c0792d0 msgr2=0x7f529c100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:33.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.229+0000 7f52a2a06700 1 -- 172.21.15.31:0/2040025787 shutdown_connections 2024-04-26T17:54:33.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.229+0000 7f52a2a06700 1 -- 172.21.15.31:0/2040025787 wait complete. 2024-04-26T17:54:33.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.514+0000 7fd3afca8700 1 -- 172.21.15.31:0/725190266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8102320 msgr2=0x7fd3a81027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:33.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.514+0000 7fd3afca8700 1 --2- 172.21.15.31:0/725190266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8102320 0x7fd3a81027a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd398009990 tx=0x7fd398009ca0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.514+0000 7fd3afca8700 1 -- 172.21.15.31:0/725190266 shutdown_connections 2024-04-26T17:54:33.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.514+0000 7fd3afca8700 1 --2- 172.21.15.31:0/725190266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3a8102ce0 0x7fd3a810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.514+0000 7fd3afca8700 1 --2- 172.21.15.31:0/725190266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a8108390 0x7fd3a8108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.514+0000 7fd3afca8700 1 --2- 172.21.15.31:0/725190266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8102320 0x7fd3a81027a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.514+0000 7fd3afca8700 1 -- 172.21.15.31:0/725190266 >> 172.21.15.31:0/725190266 conn(0x7fd3a80792d0 msgr2=0x7fd3a80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:33.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.514+0000 7fd3afca8700 1 -- 172.21.15.31:0/725190266 shutdown_connections 2024-04-26T17:54:33.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.514+0000 7fd3afca8700 1 -- 172.21.15.31:0/725190266 wait complete. 2024-04-26T17:54:33.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.515+0000 7fd3afca8700 1 Processor -- start 2024-04-26T17:54:33.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.515+0000 7fd3afca8700 1 -- start start 2024-04-26T17:54:33.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.515+0000 7fd3afca8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a8102320 0x7fd3a8104000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:33.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.515+0000 7fd3afca8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8102ce0 0x7fd3a8104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:33.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.515+0000 7fd3afca8700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3a8108390 0x7fd3a8104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:33.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.515+0000 7fd3afca8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a81124b0 con 0x7fd3a8102ce0 2024-04-26T17:54:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.515+0000 7fd3afca8700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a8112330 con 0x7fd3a8108390 2024-04-26T17:54:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.515+0000 7fd3afca8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a8112630 con 0x7fd3a8102320 2024-04-26T17:54:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.515+0000 7fd3ad243700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8102ce0 0x7fd3a8104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd3ad243700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8102ce0 0x7fd3a8104540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52232/0 (socket says 172.21.15.31:52232) 2024-04-26T17:54:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd3ad243700 1 -- 172.21.15.31:0/1180002911 learned_addr learned my addr 172.21.15.31:0/1180002911 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd3ae245700 1 --2- 172.21.15.31:0/1180002911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3a8108390 0x7fd3a8104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd3ada44700 1 --2- 172.21.15.31:0/1180002911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a8102320 0x7fd3a8104000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd3ad243700 1 -- 172.21.15.31:0/1180002911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a8102320 msgr2=0x7fd3a8104000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd3ad243700 1 --2- 172.21.15.31:0/1180002911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a8102320 0x7fd3a8104000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd3ad243700 1 -- 172.21.15.31:0/1180002911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3a8108390 msgr2=0x7fd3a8104a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd3ad243700 1 --2- 172.21.15.31:0/1180002911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd3a8108390 0x7fd3a8104a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd3ad243700 1 -- 172.21.15.31:0/1180002911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a81a8040 con 0x7fd3a8102ce0 2024-04-26T17:54:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd3ad243700 1 --2- 172.21.15.31:0/1180002911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8102ce0 0x7fd3a8104540 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd398000c00 tx=0x7fd3980047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd39effd700 1 -- 172.21.15.31:0/1180002911 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd398027070 con 0x7fd3a8102ce0 2024-04-26T17:54:33.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd39effd700 1 -- 172.21.15.31:0/1180002911 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd398003f20 con 0x7fd3a8102ce0 2024-04-26T17:54:33.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd39effd700 1 -- 172.21.15.31:0/1180002911 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd398022ca0 con 0x7fd3a8102ce0 2024-04-26T17:54:33.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd3afca8700 1 -- 172.21.15.31:0/1180002911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3a81a82d0 con 0x7fd3a8102ce0 2024-04-26T17:54:33.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.516+0000 7fd3afca8700 1 -- 172.21.15.31:0/1180002911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3a81a8720 con 0x7fd3a8102ce0 2024-04-26T17:54:33.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.517+0000 7fd3afca8700 1 -- 172.21.15.31:0/1180002911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd3a806b3d0 con 0x7fd3a8102ce0 2024-04-26T17:54:33.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.519+0000 7fd39effd700 1 -- 172.21.15.31:0/1180002911 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd39802a070 con 0x7fd3a8102ce0 2024-04-26T17:54:33.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.519+0000 7fd39effd700 1 --2- 172.21.15.31:0/1180002911 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd3940a3cd0 0x7fd3940a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:33.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.519+0000 7fd3ada44700 1 --2- 172.21.15.31:0/1180002911 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd3940a3cd0 0x7fd3940a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:33.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.519+0000 7fd39effd700 1 -- 172.21.15.31:0/1180002911 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fd3980d1a90 con 0x7fd3a8102ce0 2024-04-26T17:54:33.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.520+0000 7fd39effd700 1 --2- 172.21.15.31:0/1180002911 >> [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143] conn(0x7fd3940ad7f0 0x7fd3940afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:33.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.520+0000 7fd39effd700 1 -- 172.21.15.31:0/1180002911 --> [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3980095d0 con 0x7fd3940ad7f0 2024-04-26T17:54:33.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.520+0000 7fd39effd700 1 -- 172.21.15.31:0/1180002911 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fd3980d1ee0 con 0x7fd3a8102ce0 2024-04-26T17:54:33.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.520+0000 7fd3ada44700 1 --2- 172.21.15.31:0/1180002911 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd3940a3cd0 0x7fd3940a6190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4005950 tx=0x7fd3a400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:33.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.520+0000 7fd3ae245700 1 --2- 172.21.15.31:0/1180002911 >> [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143] conn(0x7fd3940ad7f0 0x7fd3940afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:33.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.521+0000 7fd3ae245700 1 --2- 172.21.15.31:0/1180002911 >> [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143] conn(0x7fd3940ad7f0 0x7fd3940afc90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:33.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.521+0000 7fd39effd700 1 -- 172.21.15.31:0/1180002911 <== osd.4 v2:172.21.15.97:6808/1359263143 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd3980095d0 con 0x7fd3940ad7f0 2024-04-26T17:54:33.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.538+0000 7fd3afca8700 1 -- 172.21.15.31:0/1180002911 --> [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd3a8103a40 con 0x7fd3940ad7f0 2024-04-26T17:54:33.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.539+0000 7fd39effd700 1 -- 172.21.15.31:0/1180002911 <== osd.4 v2:172.21.15.97:6808/1359263143 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd3a8103a40 con 0x7fd3940ad7f0 2024-04-26T17:54:33.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.539+0000 7fd3afca8700 1 -- 172.21.15.31:0/1180002911 >> [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143] conn(0x7fd3940ad7f0 msgr2=0x7fd3940afc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:33.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.539+0000 7fd3afca8700 1 --2- 172.21.15.31:0/1180002911 >> [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143] conn(0x7fd3940ad7f0 0x7fd3940afc90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.539+0000 7fd3afca8700 1 -- 172.21.15.31:0/1180002911 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd3940a3cd0 msgr2=0x7fd3940a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:33.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.539+0000 7fd3afca8700 1 --2- 172.21.15.31:0/1180002911 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd3940a3cd0 0x7fd3940a6190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4005950 tx=0x7fd3a400b410 comp rx=0 tx=0).stop 2024-04-26T17:54:33.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.539+0000 7fd3afca8700 1 -- 172.21.15.31:0/1180002911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8102ce0 msgr2=0x7fd3a8104540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:33.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.539+0000 7fd3afca8700 1 --2- 172.21.15.31:0/1180002911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8102ce0 0x7fd3a8104540 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd398000c00 tx=0x7fd3980047b0 comp rx=0 tx=0).stop 2024-04-26T17:54:33.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.539+0000 7fd3ae245700 1 -- 172.21.15.31:0/1180002911 reap_dead start 2024-04-26T17:54:33.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.539+0000 7fd3afca8700 1 -- 172.21.15.31:0/1180002911 shutdown_connections 2024-04-26T17:54:33.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.539+0000 7fd3afca8700 1 -- 172.21.15.31:0/1180002911 >> 172.21.15.31:0/1180002911 conn(0x7fd3a80792d0 msgr2=0x7fd3a8101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:33.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.539+0000 7fd3afca8700 1 -- 172.21.15.31:0/1180002911 shutdown_connections 2024-04-26T17:54:33.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:33.539+0000 7fd3afca8700 1 -- 172.21.15.31:0/1180002911 wait complete. 2024-04-26T17:54:34.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:34 smithi031 ceph-mon[29164]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:54:34.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:34 smithi097 ceph-mon[32419]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:54:34.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:34 smithi186 ceph-mon[32797]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:54:35.427 INFO:teuthology.orchestra.run.smithi031.stdout:141733920837 2024-04-26T17:54:35.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-26T17:54:35.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.621+0000 7f65432ca700 1 -- 172.21.15.31:0/1653012111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f653c06c360 msgr2=0x7f653c1102d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:35.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.621+0000 7f65432ca700 1 --2- 172.21.15.31:0/1653012111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f653c06c360 0x7f653c1102d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f653400b0a0 tx=0x7f653400b3b0 comp rx=0 tx=0).stop 2024-04-26T17:54:35.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.621+0000 7f65432ca700 1 -- 172.21.15.31:0/1653012111 shutdown_connections 2024-04-26T17:54:35.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.621+0000 7f65432ca700 1 --2- 172.21.15.31:0/1653012111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f653c06c360 0x7f653c1102d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:35.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.621+0000 7f65432ca700 1 --2- 172.21.15.31:0/1653012111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f653c109490 0x7f653c109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:35.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.621+0000 7f65432ca700 1 --2- 172.21.15.31:0/1653012111 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f653c06b9c0 0x7f653c06be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:35.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.621+0000 7f65432ca700 1 -- 172.21.15.31:0/1653012111 >> 172.21.15.31:0/1653012111 conn(0x7f653c079290 msgr2=0x7f653c0796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:35.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.621+0000 7f65432ca700 1 -- 172.21.15.31:0/1653012111 shutdown_connections 2024-04-26T17:54:35.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.621+0000 7f65432ca700 1 -- 172.21.15.31:0/1653012111 wait complete. 2024-04-26T17:54:35.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.622+0000 7f65432ca700 1 Processor -- start 2024-04-26T17:54:35.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.622+0000 7f65432ca700 1 -- start start 2024-04-26T17:54:35.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.622+0000 7f65432ca700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f653c06b9c0 0x7f653c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:35.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.622+0000 7f65432ca700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f653c06c360 0x7f653c19fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.622+0000 7f65432ca700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f653c109490 0x7f653c199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.622+0000 7f65432ca700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f653c1135b0 con 0x7f653c109490 2024-04-26T17:54:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.622+0000 7f65432ca700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f653c113430 con 0x7f653c06c360 2024-04-26T17:54:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.622+0000 7f65432ca700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f653c113730 con 0x7f653c06b9c0 2024-04-26T17:54:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.622+0000 7f6541066700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f653c06b9c0 0x7f653c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.622+0000 7f6541066700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f653c06b9c0 0x7f653c19f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56424/0 (socket says 172.21.15.31:56424) 2024-04-26T17:54:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.622+0000 7f6541066700 1 -- 172.21.15.31:0/934605877 learned_addr learned my addr 172.21.15.31:0/934605877 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.623+0000 7f6540865700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f653c06c360 0x7f653c19fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.623+0000 7f6541867700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f653c109490 0x7f653c199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.623+0000 7f6541066700 1 -- 172.21.15.31:0/934605877 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f653c06c360 msgr2=0x7f653c19fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:35.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.623+0000 7f6541066700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f653c06c360 0x7f653c19fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.623+0000 7f6541066700 1 -- 172.21.15.31:0/934605877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f653c109490 msgr2=0x7f653c199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.623+0000 7f6541066700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f653c109490 0x7f653c199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.623+0000 7f6541066700 1 -- 172.21.15.31:0/934605877 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653c19a0d0 con 0x7f653c06b9c0 2024-04-26T17:54:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.623+0000 7f6541867700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f653c109490 0x7f653c199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:54:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.623+0000 7f6541066700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f653c06b9c0 0x7f653c19f7b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f653800e9c0 tx=0x7f653800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.623+0000 7f65327fc700 1 -- 172.21.15.31:0/934605877 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653800cbd0 con 0x7f653c06b9c0 2024-04-26T17:54:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.623+0000 7f65432ca700 1 -- 172.21.15.31:0/934605877 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f653c19a3c0 con 0x7f653c06b9c0 2024-04-26T17:54:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.623+0000 7f65327fc700 1 -- 172.21.15.31:0/934605877 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f653800ee60 con 0x7f653c06b9c0 2024-04-26T17:54:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.623+0000 7f65327fc700 1 -- 172.21.15.31:0/934605877 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65380055c0 con 0x7f653c06b9c0 2024-04-26T17:54:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.623+0000 7f65432ca700 1 -- 172.21.15.31:0/934605877 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f653c1a6680 con 0x7f653c06b9c0 2024-04-26T17:54:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.624+0000 7f65432ca700 1 -- 172.21.15.31:0/934605877 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f653c063af0 con 0x7f653c06b9c0 2024-04-26T17:54:35.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.626+0000 7f65327fc700 1 -- 172.21.15.31:0/934605877 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6538004750 con 0x7f653c06b9c0 2024-04-26T17:54:35.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.627+0000 7f65327fc700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f65280a3cd0 0x7f65280a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:35.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.627+0000 7f6540865700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f65280a3cd0 0x7f65280a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:35.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.627+0000 7f65327fc700 1 -- 172.21.15.31:0/934605877 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f65380d0bc0 con 0x7f653c06b9c0 2024-04-26T17:54:35.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.627+0000 7f65327fc700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170] conn(0x7f65280ad790 0x7f65280afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.627+0000 7f65327fc700 1 -- 172.21.15.31:0/934605877 --> [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f653801ee50 con 0x7f65280ad790 2024-04-26T17:54:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.627+0000 7f65327fc700 1 -- 172.21.15.31:0/934605877 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f65380d0f70 con 0x7f653c06b9c0 2024-04-26T17:54:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.627+0000 7f6540865700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f65280a3cd0 0x7f65280a6190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f653c079990 tx=0x7f652c0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.628+0000 7f6541867700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170] conn(0x7f65280ad790 0x7f65280afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.628+0000 7f6541867700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170] conn(0x7f65280ad790 0x7f65280afc30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.629+0000 7f65327fc700 1 -- 172.21.15.31:0/934605877 <== osd.8 v2:172.21.15.186:6816/3193558170 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f653801ee50 con 0x7f65280ad790 2024-04-26T17:54:35.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.649+0000 7f65432ca700 1 -- 172.21.15.31:0/934605877 --> [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f653c063f20 con 0x7f65280ad790 2024-04-26T17:54:35.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.650+0000 7f65327fc700 1 -- 172.21.15.31:0/934605877 <== osd.8 v2:172.21.15.186:6816/3193558170 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f653c063f20 con 0x7f65280ad790 2024-04-26T17:54:35.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.650+0000 7f65432ca700 1 -- 172.21.15.31:0/934605877 >> [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170] conn(0x7f65280ad790 msgr2=0x7f65280afc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:35.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.650+0000 7f65432ca700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170] conn(0x7f65280ad790 0x7f65280afc30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:35.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.650+0000 7f65432ca700 1 -- 172.21.15.31:0/934605877 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f65280a3cd0 msgr2=0x7f65280a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:35.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.650+0000 7f65432ca700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f65280a3cd0 0x7f65280a6190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f653c079990 tx=0x7f652c0073d0 comp rx=0 tx=0).stop 2024-04-26T17:54:35.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.650+0000 7f65432ca700 1 -- 172.21.15.31:0/934605877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f653c06b9c0 msgr2=0x7f653c19f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:35.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.650+0000 7f65432ca700 1 --2- 172.21.15.31:0/934605877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f653c06b9c0 0x7f653c19f7b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f653800e9c0 tx=0x7f653800ecd0 comp rx=0 tx=0).stop 2024-04-26T17:54:35.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.650+0000 7f6541867700 1 -- 172.21.15.31:0/934605877 reap_dead start 2024-04-26T17:54:35.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.651+0000 7f65432ca700 1 -- 172.21.15.31:0/934605877 shutdown_connections 2024-04-26T17:54:35.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.651+0000 7f65432ca700 1 -- 172.21.15.31:0/934605877 >> 172.21.15.31:0/934605877 conn(0x7f653c079290 msgr2=0x7f653c1004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:35.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.651+0000 7f65432ca700 1 -- 172.21.15.31:0/934605877 shutdown_connections 2024-04-26T17:54:35.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:35.651+0000 7f65432ca700 1 -- 172.21.15.31:0/934605877 wait complete. 2024-04-26T17:54:36.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.302+0000 7f451fe27700 1 -- 172.21.15.31:0/1043179686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45181034c0 msgr2=0x7f4518103940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:36.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.302+0000 7f451fe27700 1 --2- 172.21.15.31:0/1043179686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45181034c0 0x7f4518103940 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f450c0099c0 tx=0x7f450c009cd0 comp rx=0 tx=0).stop 2024-04-26T17:54:36.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.302+0000 7f451fe27700 1 -- 172.21.15.31:0/1043179686 shutdown_connections 2024-04-26T17:54:36.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.302+0000 7f451fe27700 1 --2- 172.21.15.31:0/1043179686 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4518109560 0x7f4518109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:36.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.302+0000 7f451fe27700 1 --2- 172.21.15.31:0/1043179686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4518103e80 0x7f4518110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:36.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.302+0000 7f451fe27700 1 --2- 172.21.15.31:0/1043179686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45181034c0 0x7f4518103940 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:36.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.302+0000 7f451fe27700 1 -- 172.21.15.31:0/1043179686 >> 172.21.15.31:0/1043179686 conn(0x7f451807a5d0 msgr2=0x7f451807a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:36.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.302+0000 7f451fe27700 1 -- 172.21.15.31:0/1043179686 shutdown_connections 2024-04-26T17:54:36.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.302+0000 7f451fe27700 1 -- 172.21.15.31:0/1043179686 wait complete. 2024-04-26T17:54:36.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451fe27700 1 Processor -- start 2024-04-26T17:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451fe27700 1 -- start start 2024-04-26T17:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451fe27700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45181034c0 0x7f451819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451fe27700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4518103e80 0x7f451819fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451fe27700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4518109560 0x7f4518199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451fe27700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4518113670 con 0x7f4518109560 2024-04-26T17:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451fe27700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f45181134f0 con 0x7f4518103e80 2024-04-26T17:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451fe27700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45181137f0 con 0x7f45181034c0 2024-04-26T17:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451e3c4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4518109560 0x7f4518199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451d3c2700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4518103e80 0x7f451819fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451e3c4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4518109560 0x7f4518199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50870/0 (socket says 172.21.15.31:50870) 2024-04-26T17:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451e3c4700 1 -- 172.21.15.31:0/2033443031 learned_addr learned my addr 172.21.15.31:0/2033443031 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451dbc3700 1 --2- 172.21.15.31:0/2033443031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45181034c0 0x7f451819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451e3c4700 1 -- 172.21.15.31:0/2033443031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45181034c0 msgr2=0x7f451819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451e3c4700 1 --2- 172.21.15.31:0/2033443031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45181034c0 0x7f451819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451e3c4700 1 -- 172.21.15.31:0/2033443031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4518103e80 msgr2=0x7f451819fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.303+0000 7f451e3c4700 1 --2- 172.21.15.31:0/2033443031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4518103e80 0x7f451819fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.304+0000 7f451e3c4700 1 -- 172.21.15.31:0/2033443031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f451819a0b0 con 0x7f4518109560 2024-04-26T17:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.304+0000 7f451e3c4700 1 --2- 172.21.15.31:0/2033443031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4518109560 0x7f4518199950 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f450800d980 tx=0x7f450800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.304+0000 7f4516ffd700 1 -- 172.21.15.31:0/2033443031 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4508017070 con 0x7f4518109560 2024-04-26T17:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.304+0000 7f4516ffd700 1 -- 172.21.15.31:0/2033443031 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4508005020 con 0x7f4518109560 2024-04-26T17:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.304+0000 7f451fe27700 1 -- 172.21.15.31:0/2033443031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f451819a3a0 con 0x7f4518109560 2024-04-26T17:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.304+0000 7f4516ffd700 1 -- 172.21.15.31:0/2033443031 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450800fa10 con 0x7f4518109560 2024-04-26T17:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.304+0000 7f451fe27700 1 -- 172.21.15.31:0/2033443031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45181a6660 con 0x7f4518109560 2024-04-26T17:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.305+0000 7f451fe27700 1 -- 172.21.15.31:0/2033443031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4518063af0 con 0x7f4518109560 2024-04-26T17:54:36.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.306+0000 7f4516ffd700 1 -- 172.21.15.31:0/2033443031 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4508004a30 con 0x7f4518109560 2024-04-26T17:54:36.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.307+0000 7f4516ffd700 1 --2- 172.21.15.31:0/2033443031 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f45000a3cd0 0x7f45000a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:36.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.307+0000 7f451dbc3700 1 --2- 172.21.15.31:0/2033443031 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f45000a3cd0 0x7f45000a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:36.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.307+0000 7f4516ffd700 1 -- 172.21.15.31:0/2033443031 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f45080d0940 con 0x7f4518109560 2024-04-26T17:54:36.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.307+0000 7f4516ffd700 1 --2- 172.21.15.31:0/2033443031 >> [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626] conn(0x7f45000ad790 0x7f45000afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:36.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.307+0000 7f4516ffd700 1 -- 172.21.15.31:0/2033443031 --> [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45000b0300 con 0x7f45000ad790 2024-04-26T17:54:36.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.307+0000 7f4516ffd700 1 -- 172.21.15.31:0/2033443031 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f45080d0cf0 con 0x7f4518109560 2024-04-26T17:54:36.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.307+0000 7f451dbc3700 1 --2- 172.21.15.31:0/2033443031 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f45000a3cd0 0x7f45000a6190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4510009550 tx=0x7f45100091b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:36.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.307+0000 7f451d3c2700 1 --2- 172.21.15.31:0/2033443031 >> [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626] conn(0x7f45000ad790 0x7f45000afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:36.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.308+0000 7f451d3c2700 1 --2- 172.21.15.31:0/2033443031 >> [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626] conn(0x7f45000ad790 0x7f45000afc30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:36.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.309+0000 7f4516ffd700 1 -- 172.21.15.31:0/2033443031 <== osd.5 v2:172.21.15.186:6808/2006134626 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45000b0300 con 0x7f45000ad790 2024-04-26T17:54:36.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.326+0000 7f451fe27700 1 -- 172.21.15.31:0/2033443031 --> [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4518063ef0 con 0x7f45000ad790 2024-04-26T17:54:36.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.327+0000 7f4516ffd700 1 -- 172.21.15.31:0/2033443031 <== osd.5 v2:172.21.15.186:6808/2006134626 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4518063ef0 con 0x7f45000ad790 2024-04-26T17:54:36.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.327+0000 7f451fe27700 1 -- 172.21.15.31:0/2033443031 >> [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626] conn(0x7f45000ad790 msgr2=0x7f45000afc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:36.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.327+0000 7f451fe27700 1 --2- 172.21.15.31:0/2033443031 >> [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626] conn(0x7f45000ad790 0x7f45000afc30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:36.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.327+0000 7f451fe27700 1 -- 172.21.15.31:0/2033443031 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f45000a3cd0 msgr2=0x7f45000a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:36.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.327+0000 7f451fe27700 1 --2- 172.21.15.31:0/2033443031 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f45000a3cd0 0x7f45000a6190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4510009550 tx=0x7f45100091b0 comp rx=0 tx=0).stop 2024-04-26T17:54:36.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.327+0000 7f451fe27700 1 -- 172.21.15.31:0/2033443031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4518109560 msgr2=0x7f4518199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:36.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.327+0000 7f451fe27700 1 --2- 172.21.15.31:0/2033443031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4518109560 0x7f4518199950 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f450800d980 tx=0x7f450800dc90 comp rx=0 tx=0).stop 2024-04-26T17:54:36.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.327+0000 7f451e3c4700 1 -- 172.21.15.31:0/2033443031 reap_dead start 2024-04-26T17:54:36.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.327+0000 7f451fe27700 1 -- 172.21.15.31:0/2033443031 shutdown_connections 2024-04-26T17:54:36.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.327+0000 7f451fe27700 1 -- 172.21.15.31:0/2033443031 >> 172.21.15.31:0/2033443031 conn(0x7f451807a5d0 msgr2=0x7f4518102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:36.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.327+0000 7f451fe27700 1 -- 172.21.15.31:0/2033443031 shutdown_connections 2024-04-26T17:54:36.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:36.327+0000 7f451fe27700 1 -- 172.21.15.31:0/2033443031 wait complete. 2024-04-26T17:54:36.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:36 smithi031 ceph-mon[29164]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:36.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:36 smithi097 ceph-mon[32419]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:36.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:36 smithi186 ceph-mon[32797]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:38.047 INFO:teuthology.orchestra.run.smithi031.stdout:253403070512 2024-04-26T17:54:38.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-26T17:54:38.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.498+0000 7fdeb3668700 1 -- 172.21.15.31:0/1628832438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdeac1094a0 msgr2=0x7fdeac109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:38.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.498+0000 7fdeb3668700 1 --2- 172.21.15.31:0/1628832438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdeac1094a0 0x7fdeac109880 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009990 tx=0x7fdea8009ca0 comp rx=0 tx=0).stop 2024-04-26T17:54:38.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.498+0000 7fdeb3668700 1 -- 172.21.15.31:0/1628832438 shutdown_connections 2024-04-26T17:54:38.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.498+0000 7fdeb3668700 1 --2- 172.21.15.31:0/1628832438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdeac06bad0 0x7fdeac1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:38.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.498+0000 7fdeb3668700 1 --2- 172.21.15.31:0/1628832438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdeac1094a0 0x7fdeac109880 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:38.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.498+0000 7fdeb3668700 1 --2- 172.21.15.31:0/1628832438 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdeac06b130 0x7fdeac06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:38.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.498+0000 7fdeb3668700 1 -- 172.21.15.31:0/1628832438 >> 172.21.15.31:0/1628832438 conn(0x7fdeac0792d0 msgr2=0x7fdeac0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:38.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.498+0000 7fdeb3668700 1 -- 172.21.15.31:0/1628832438 shutdown_connections 2024-04-26T17:54:38.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.498+0000 7fdeb3668700 1 -- 172.21.15.31:0/1628832438 wait complete. 2024-04-26T17:54:38.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb3668700 1 Processor -- start 2024-04-26T17:54:38.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb3668700 1 -- start start 2024-04-26T17:54:38.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb3668700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdeac06b130 0x7fdeac1023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:38.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb3668700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdeac06bad0 0x7fdeac102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:38.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb3668700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdeac1094a0 0x7fdeac102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:38.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb3668700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac1135b0 con 0x7fdeac1094a0 2024-04-26T17:54:38.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb3668700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac113430 con 0x7fdeac06b130 2024-04-26T17:54:38.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb3668700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac113730 con 0x7fdeac06bad0 2024-04-26T17:54:38.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb1c05700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdeac1094a0 0x7fdeac102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:38.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb1c05700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdeac1094a0 0x7fdeac102e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50886/0 (socket says 172.21.15.31:50886) 2024-04-26T17:54:38.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb1c05700 1 -- 172.21.15.31:0/3399637448 learned_addr learned my addr 172.21.15.31:0/3399637448 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:38.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb1404700 1 --2- 172.21.15.31:0/3399637448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdeac06b130 0x7fdeac1023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:38.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb0c03700 1 --2- 172.21.15.31:0/3399637448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdeac06bad0 0x7fdeac102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:38.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb1c05700 1 -- 172.21.15.31:0/3399637448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdeac06bad0 msgr2=0x7fdeac102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:38.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb1c05700 1 --2- 172.21.15.31:0/3399637448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdeac06bad0 0x7fdeac102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:38.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb1c05700 1 -- 172.21.15.31:0/3399637448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdeac06b130 msgr2=0x7fdeac1023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:38.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb1c05700 1 --2- 172.21.15.31:0/3399637448 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdeac06b130 0x7fdeac1023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:38.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb1c05700 1 -- 172.21.15.31:0/3399637448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeac1a8030 con 0x7fdeac1094a0 2024-04-26T17:54:38.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.499+0000 7fdeb1c05700 1 --2- 172.21.15.31:0/3399637448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdeac1094a0 0x7fdeac102e70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdea400ef30 tx=0x7fdea400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:38.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.500+0000 7fdea27fc700 1 -- 172.21.15.31:0/3399637448 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea40092e0 con 0x7fdeac1094a0 2024-04-26T17:54:38.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.500+0000 7fdeb3668700 1 -- 172.21.15.31:0/3399637448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeac1a82c0 con 0x7fdeac1094a0 2024-04-26T17:54:38.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.500+0000 7fdea27fc700 1 -- 172.21.15.31:0/3399637448 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdea4004bb0 con 0x7fdeac1094a0 2024-04-26T17:54:38.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.500+0000 7fdea27fc700 1 -- 172.21.15.31:0/3399637448 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea4004020 con 0x7fdeac1094a0 2024-04-26T17:54:38.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.500+0000 7fdeb3668700 1 -- 172.21.15.31:0/3399637448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeac1a8800 con 0x7fdeac1094a0 2024-04-26T17:54:38.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.500+0000 7fdeb3668700 1 -- 172.21.15.31:0/3399637448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdeac063b30 con 0x7fdeac1094a0 2024-04-26T17:54:38.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.502+0000 7fdea27fc700 1 -- 172.21.15.31:0/3399637448 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdea401b030 con 0x7fdeac1094a0 2024-04-26T17:54:38.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.502+0000 7fdea27fc700 1 --2- 172.21.15.31:0/3399637448 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fde980a3cd0 0x7fde980a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:38.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.502+0000 7fdeb1404700 1 --2- 172.21.15.31:0/3399637448 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fde980a3cd0 0x7fde980a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:38.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.502+0000 7fdea27fc700 1 -- 172.21.15.31:0/3399637448 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fdea40d08c0 con 0x7fdeac1094a0 2024-04-26T17:54:38.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.503+0000 7fdea27fc700 1 --2- 172.21.15.31:0/3399637448 >> [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427] conn(0x7fde980ad790 0x7fde980afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:38.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.503+0000 7fdea27fc700 1 -- 172.21.15.31:0/3399637448 --> [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde980b0300 con 0x7fde980ad790 2024-04-26T17:54:38.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.503+0000 7fdea27fc700 1 -- 172.21.15.31:0/3399637448 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fdea40d0d10 con 0x7fdeac1094a0 2024-04-26T17:54:38.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.503+0000 7fdeb1404700 1 --2- 172.21.15.31:0/3399637448 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fde980a3cd0 0x7fde980a6190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdea800b2e0 tx=0x7fdea8005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:38.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.503+0000 7fdeb0c03700 1 --2- 172.21.15.31:0/3399637448 >> [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427] conn(0x7fde980ad790 0x7fde980afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:38.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.503+0000 7fdeb0c03700 1 --2- 172.21.15.31:0/3399637448 >> [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427] conn(0x7fde980ad790 0x7fde980afc30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:38.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.504+0000 7fdea27fc700 1 -- 172.21.15.31:0/3399637448 <== osd.1 v2:172.21.15.97:6800/4209866427 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde980b0300 con 0x7fde980ad790 2024-04-26T17:54:38.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.521+0000 7fdeb3668700 1 -- 172.21.15.31:0/3399637448 --> [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdeac063fb0 con 0x7fde980ad790 2024-04-26T17:54:38.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.521+0000 7fdea27fc700 1 -- 172.21.15.31:0/3399637448 <== osd.1 v2:172.21.15.97:6800/4209866427 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdeac063fb0 con 0x7fde980ad790 2024-04-26T17:54:38.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.522+0000 7fdeb3668700 1 -- 172.21.15.31:0/3399637448 >> [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427] conn(0x7fde980ad790 msgr2=0x7fde980afc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:38.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.522+0000 7fdeb3668700 1 --2- 172.21.15.31:0/3399637448 >> [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427] conn(0x7fde980ad790 0x7fde980afc30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:38.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.522+0000 7fdeb3668700 1 -- 172.21.15.31:0/3399637448 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fde980a3cd0 msgr2=0x7fde980a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:38.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.522+0000 7fdeb3668700 1 --2- 172.21.15.31:0/3399637448 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fde980a3cd0 0x7fde980a6190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdea800b2e0 tx=0x7fdea8005960 comp rx=0 tx=0).stop 2024-04-26T17:54:38.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.522+0000 7fdeb3668700 1 -- 172.21.15.31:0/3399637448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdeac1094a0 msgr2=0x7fdeac102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:38.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.522+0000 7fdeb3668700 1 --2- 172.21.15.31:0/3399637448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdeac1094a0 0x7fdeac102e70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdea400ef30 tx=0x7fdea400c3a0 comp rx=0 tx=0).stop 2024-04-26T17:54:38.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.522+0000 7fdeb1c05700 1 -- 172.21.15.31:0/3399637448 reap_dead start 2024-04-26T17:54:38.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.522+0000 7fdeb3668700 1 -- 172.21.15.31:0/3399637448 shutdown_connections 2024-04-26T17:54:38.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.522+0000 7fdeb3668700 1 -- 172.21.15.31:0/3399637448 >> 172.21.15.31:0/3399637448 conn(0x7fdeac0792d0 msgr2=0x7fdeac100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:38.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.522+0000 7fdeb3668700 1 -- 172.21.15.31:0/3399637448 shutdown_connections 2024-04-26T17:54:38.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:37.522+0000 7fdeb3668700 1 -- 172.21.15.31:0/3399637448 wait complete. 2024-04-26T17:54:38.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:38 smithi031 ceph-mon[29164]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:54:38.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:38 smithi097 ceph-mon[32419]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:54:38.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:38 smithi186 ceph-mon[32797]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:54:39.111 INFO:teuthology.orchestra.run.smithi031.stdout:77309411408 2024-04-26T17:54:39.111 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-26T17:54:40.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.263+0000 7ff025f9f700 1 -- 172.21.15.31:0/3751923546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020102310 msgr2=0x7ff020102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:40.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.263+0000 7ff025f9f700 1 --2- 172.21.15.31:0/3751923546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020102310 0x7ff020102790 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff010009990 tx=0x7ff010009ca0 comp rx=0 tx=0).stop 2024-04-26T17:54:40.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.263+0000 7ff025f9f700 1 -- 172.21.15.31:0/3751923546 shutdown_connections 2024-04-26T17:54:40.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.263+0000 7ff025f9f700 1 --2- 172.21.15.31:0/3751923546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff020102cd0 0x7ff02010f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:40.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.263+0000 7ff025f9f700 1 --2- 172.21.15.31:0/3751923546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff020108380 0x7ff020108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:40.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.263+0000 7ff025f9f700 1 --2- 172.21.15.31:0/3751923546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020102310 0x7ff020102790 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:40.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.263+0000 7ff025f9f700 1 -- 172.21.15.31:0/3751923546 >> 172.21.15.31:0/3751923546 conn(0x7ff020079290 msgr2=0x7ff0200796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:40.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.263+0000 7ff025f9f700 1 -- 172.21.15.31:0/3751923546 shutdown_connections 2024-04-26T17:54:40.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.263+0000 7ff025f9f700 1 -- 172.21.15.31:0/3751923546 wait complete. 2024-04-26T17:54:40.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.264+0000 7ff025f9f700 1 Processor -- start 2024-04-26T17:54:40.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.264+0000 7ff025f9f700 1 -- start start 2024-04-26T17:54:40.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.264+0000 7ff025f9f700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff020102310 0x7ff020103ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:40.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.264+0000 7ff025f9f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020102cd0 0x7ff020104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:40.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.264+0000 7ff025f9f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff020108380 0x7ff020104a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:40.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.264+0000 7ff025f9f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff020112520 con 0x7ff020102cd0 2024-04-26T17:54:40.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.264+0000 7ff025f9f700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff0201123a0 con 0x7ff020102310 2024-04-26T17:54:40.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.264+0000 7ff025f9f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff0201126a0 con 0x7ff020108380 2024-04-26T17:54:40.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.264+0000 7ff01effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020102cd0 0x7ff020104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:40.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.264+0000 7ff01effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020102cd0 0x7ff020104530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50898/0 (socket says 172.21.15.31:50898) 2024-04-26T17:54:40.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.264+0000 7ff01effd700 1 -- 172.21.15.31:0/1105531080 learned_addr learned my addr 172.21.15.31:0/1105531080 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:40.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.264+0000 7ff01f7fe700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff020102310 0x7ff020103ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:40.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.264+0000 7ff01ffff700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff020108380 0x7ff020104a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:40.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.265+0000 7ff01effd700 1 -- 172.21.15.31:0/1105531080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff020108380 msgr2=0x7ff020104a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:40.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.265+0000 7ff01effd700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff020108380 0x7ff020104a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:40.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.265+0000 7ff01effd700 1 -- 172.21.15.31:0/1105531080 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff020102310 msgr2=0x7ff020103ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:40.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.265+0000 7ff01effd700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff020102310 0x7ff020103ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:40.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.265+0000 7ff01effd700 1 -- 172.21.15.31:0/1105531080 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0201a80b0 con 0x7ff020102cd0 2024-04-26T17:54:40.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.265+0000 7ff01ffff700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff020108380 0x7ff020104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:54:40.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.265+0000 7ff01effd700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020102cd0 0x7ff020104530 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff010000c00 tx=0x7ff0100047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:40.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.265+0000 7ff01cff9700 1 -- 172.21.15.31:0/1105531080 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff010027070 con 0x7ff020102cd0 2024-04-26T17:54:40.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.265+0000 7ff025f9f700 1 -- 172.21.15.31:0/1105531080 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0201a8340 con 0x7ff020102cd0 2024-04-26T17:54:40.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.265+0000 7ff01cff9700 1 -- 172.21.15.31:0/1105531080 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0100040e0 con 0x7ff020102cd0 2024-04-26T17:54:40.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.265+0000 7ff01cff9700 1 -- 172.21.15.31:0/1105531080 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01000f460 con 0x7ff020102cd0 2024-04-26T17:54:40.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.265+0000 7ff025f9f700 1 -- 172.21.15.31:0/1105531080 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0201a8790 con 0x7ff020102cd0 2024-04-26T17:54:40.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.266+0000 7ff025f9f700 1 -- 172.21.15.31:0/1105531080 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff020106400 con 0x7ff020102cd0 2024-04-26T17:54:40.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.267+0000 7ff01cff9700 1 -- 172.21.15.31:0/1105531080 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff01002a070 con 0x7ff020102cd0 2024-04-26T17:54:40.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.268+0000 7ff01cff9700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff0080a3cd0 0x7ff0080a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:40.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.268+0000 7ff01f7fe700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff0080a3cd0 0x7ff0080a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:40.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.268+0000 7ff01cff9700 1 -- 172.21.15.31:0/1105531080 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7ff0100d1c00 con 0x7ff020102cd0 2024-04-26T17:54:40.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.268+0000 7ff01cff9700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341] conn(0x7ff0080ad7f0 0x7ff0080afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:40.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.268+0000 7ff01cff9700 1 -- 172.21.15.31:0/1105531080 --> [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff010014e50 con 0x7ff0080ad7f0 2024-04-26T17:54:40.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.268+0000 7ff01cff9700 1 -- 172.21.15.31:0/1105531080 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7ff0100d2050 con 0x7ff020102cd0 2024-04-26T17:54:40.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.268+0000 7ff01ffff700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341] conn(0x7ff0080ad7f0 0x7ff0080afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:40.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.269+0000 7ff01f7fe700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff0080a3cd0 0x7ff0080a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff014009a10 tx=0x7ff014009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:40.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.269+0000 7ff01ffff700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341] conn(0x7ff0080ad7f0 0x7ff0080afc90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:40.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.270+0000 7ff01cff9700 1 -- 172.21.15.31:0/1105531080 <== osd.3 v2:172.21.15.31:6810/2640112341 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff010014e50 con 0x7ff0080ad7f0 2024-04-26T17:54:40.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.287+0000 7ff025f9f700 1 -- 172.21.15.31:0/1105531080 --> [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff020103a30 con 0x7ff0080ad7f0 2024-04-26T17:54:40.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.287+0000 7ff01cff9700 1 -- 172.21.15.31:0/1105531080 <== osd.3 v2:172.21.15.31:6810/2640112341 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff020103a30 con 0x7ff0080ad7f0 2024-04-26T17:54:40.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.287+0000 7ff025f9f700 1 -- 172.21.15.31:0/1105531080 >> [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341] conn(0x7ff0080ad7f0 msgr2=0x7ff0080afc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:40.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.287+0000 7ff025f9f700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341] conn(0x7ff0080ad7f0 0x7ff0080afc90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:40.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.287+0000 7ff025f9f700 1 -- 172.21.15.31:0/1105531080 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff0080a3cd0 msgr2=0x7ff0080a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:40.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.287+0000 7ff025f9f700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ff0080a3cd0 0x7ff0080a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff014009a10 tx=0x7ff014009190 comp rx=0 tx=0).stop 2024-04-26T17:54:40.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.287+0000 7ff025f9f700 1 -- 172.21.15.31:0/1105531080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020102cd0 msgr2=0x7ff020104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:40.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.287+0000 7ff025f9f700 1 --2- 172.21.15.31:0/1105531080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff020102cd0 0x7ff020104530 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff010000c00 tx=0x7ff0100047b0 comp rx=0 tx=0).stop 2024-04-26T17:54:40.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.287+0000 7ff01ffff700 1 -- 172.21.15.31:0/1105531080 reap_dead start 2024-04-26T17:54:40.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.287+0000 7ff025f9f700 1 -- 172.21.15.31:0/1105531080 shutdown_connections 2024-04-26T17:54:40.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.287+0000 7ff025f9f700 1 -- 172.21.15.31:0/1105531080 >> 172.21.15.31:0/1105531080 conn(0x7ff020079290 msgr2=0x7ff0201015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:40.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.287+0000 7ff025f9f700 1 -- 172.21.15.31:0/1105531080 shutdown_connections 2024-04-26T17:54:40.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:40.288+0000 7ff025f9f700 1 -- 172.21.15.31:0/1105531080 wait complete. 2024-04-26T17:54:40.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:40 smithi031 ceph-mon[29164]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:40.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:40 smithi097 ceph-mon[32419]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:40.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:40 smithi186 ceph-mon[32797]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:54:40.720 INFO:teuthology.orchestra.run.smithi031.stdout:55834574933 2024-04-26T17:54:40.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-26T17:54:41.925 INFO:teuthology.orchestra.run.smithi031.stdout:107374182474 2024-04-26T17:54:41.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-26T17:54:42.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:42 smithi031 ceph-mon[29164]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T17:54:42.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:42 smithi097 ceph-mon[32419]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T17:54:42.385 INFO:teuthology.orchestra.run.smithi031.stdout:163208757313 2024-04-26T17:54:42.385 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-26T17:54:42.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:42 smithi186 ceph-mon[32797]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T17:54:42.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.903+0000 7f46a704d700 1 -- 172.21.15.31:0/2854208868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46a006c490 msgr2=0x7f46a0110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:42.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.903+0000 7f46a704d700 1 --2- 172.21.15.31:0/2854208868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46a006c490 0x7f46a0110300 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f469800b0a0 tx=0x7f469800b3b0 comp rx=0 tx=0).stop 2024-04-26T17:54:42.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.903+0000 7f46a704d700 1 -- 172.21.15.31:0/2854208868 shutdown_connections 2024-04-26T17:54:42.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.903+0000 7f46a704d700 1 --2- 172.21.15.31:0/2854208868 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a01094c0 0x7f46a01098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:42.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.903+0000 7f46a704d700 1 --2- 172.21.15.31:0/2854208868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46a006c490 0x7f46a0110300 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:42.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.903+0000 7f46a704d700 1 --2- 172.21.15.31:0/2854208868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46a006baf0 0x7f46a006bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:42.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.903+0000 7f46a704d700 1 -- 172.21.15.31:0/2854208868 >> 172.21.15.31:0/2854208868 conn(0x7f46a00792d0 msgr2=0x7f46a00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:42.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.903+0000 7f46a704d700 1 -- 172.21.15.31:0/2854208868 shutdown_connections 2024-04-26T17:54:42.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.903+0000 7f46a704d700 1 -- 172.21.15.31:0/2854208868 wait complete. 2024-04-26T17:54:42.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.903+0000 7f46a704d700 1 Processor -- start 2024-04-26T17:54:42.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.903+0000 7f46a704d700 1 -- start start 2024-04-26T17:54:42.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.903+0000 7f46a704d700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a006baf0 0x7f46a01027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:42.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.903+0000 7f46a704d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46a006c490 0x7f46a0102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:42.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f46a704d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46a01094c0 0x7f46a0103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:42.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f46a704d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f46a01135e0 con 0x7f46a006c490 2024-04-26T17:54:42.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f46a704d700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f46a0113460 con 0x7f46a006baf0 2024-04-26T17:54:42.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f46a704d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f46a0113760 con 0x7f46a01094c0 2024-04-26T17:54:42.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f469ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46a006c490 0x7f46a0102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:42.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f469ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46a006c490 0x7f46a0102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50934/0 (socket says 172.21.15.31:50934) 2024-04-26T17:54:42.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f469ffff700 1 -- 172.21.15.31:0/1586102552 learned_addr learned my addr 172.21.15.31:0/1586102552 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:42.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f46a55ea700 1 --2- 172.21.15.31:0/1586102552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46a01094c0 0x7f46a0103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:42.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f46a4de9700 1 --2- 172.21.15.31:0/1586102552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a006baf0 0x7f46a01027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:42.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f469ffff700 1 -- 172.21.15.31:0/1586102552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46a01094c0 msgr2=0x7f46a0103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:42.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f469ffff700 1 --2- 172.21.15.31:0/1586102552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46a01094c0 0x7f46a0103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:42.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f469ffff700 1 -- 172.21.15.31:0/1586102552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a006baf0 msgr2=0x7f46a01027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:42.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f469ffff700 1 --2- 172.21.15.31:0/1586102552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f46a006baf0 0x7f46a01027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:42.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f469ffff700 1 -- 172.21.15.31:0/1586102552 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a01a8020 con 0x7f46a006c490 2024-04-26T17:54:42.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f469ffff700 1 --2- 172.21.15.31:0/1586102552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46a006c490 0x7f46a0102d10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f468c00c970 tx=0x7f468c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:42.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f469dffb700 1 -- 172.21.15.31:0/1586102552 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c007aa0 con 0x7f46a006c490 2024-04-26T17:54:42.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f46a704d700 1 -- 172.21.15.31:0/1586102552 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a01a8310 con 0x7f46a006c490 2024-04-26T17:54:42.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f469dffb700 1 -- 172.21.15.31:0/1586102552 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f468c004ce0 con 0x7f46a006c490 2024-04-26T17:54:42.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.904+0000 7f469dffb700 1 -- 172.21.15.31:0/1586102552 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c0055c0 con 0x7f46a006c490 2024-04-26T17:54:42.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.905+0000 7f46a704d700 1 -- 172.21.15.31:0/1586102552 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a01a8850 con 0x7f46a006c490 2024-04-26T17:54:42.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.905+0000 7f46a704d700 1 -- 172.21.15.31:0/1586102552 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f46a0063af0 con 0x7f46a006c490 2024-04-26T17:54:42.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.907+0000 7f469dffb700 1 -- 172.21.15.31:0/1586102552 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f468c007c40 con 0x7f46a006c490 2024-04-26T17:54:42.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.907+0000 7f469dffb700 1 --2- 172.21.15.31:0/1586102552 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f46900a3cd0 0x7f46900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:42.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.907+0000 7f46a4de9700 1 --2- 172.21.15.31:0/1586102552 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f46900a3cd0 0x7f46900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:42.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.908+0000 7f469dffb700 1 -- 172.21.15.31:0/1586102552 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f468c0d0840 con 0x7f46a006c490 2024-04-26T17:54:42.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.908+0000 7f469dffb700 1 --2- 172.21.15.31:0/1586102552 >> [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905] conn(0x7f46900ad790 0x7f46900afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:42.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.908+0000 7f469dffb700 1 -- 172.21.15.31:0/1586102552 --> [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f468c00eec0 con 0x7f46900ad790 2024-04-26T17:54:42.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.908+0000 7f469dffb700 1 -- 172.21.15.31:0/1586102552 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f468c0d0bf0 con 0x7f46a006c490 2024-04-26T17:54:42.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.908+0000 7f46a55ea700 1 --2- 172.21.15.31:0/1586102552 >> [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905] conn(0x7f46900ad790 0x7f46900afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:42.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.908+0000 7f46a4de9700 1 --2- 172.21.15.31:0/1586102552 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f46900a3cd0 0x7f46900a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4694005950 tx=0x7f46940058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:42.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.908+0000 7f46a55ea700 1 --2- 172.21.15.31:0/1586102552 >> [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905] conn(0x7f46900ad790 0x7f46900afc30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:42.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.909+0000 7f469dffb700 1 -- 172.21.15.31:0/1586102552 <== osd.9 v2:172.21.15.31:6826/3660801905 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f468c00eec0 con 0x7f46900ad790 2024-04-26T17:54:42.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.927+0000 7f46a704d700 1 -- 172.21.15.31:0/1586102552 --> [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f46a0063f70 con 0x7f46900ad790 2024-04-26T17:54:42.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.928+0000 7f469dffb700 1 -- 172.21.15.31:0/1586102552 <== osd.9 v2:172.21.15.31:6826/3660801905 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f46a0063f70 con 0x7f46900ad790 2024-04-26T17:54:42.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.928+0000 7f46a704d700 1 -- 172.21.15.31:0/1586102552 >> [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905] conn(0x7f46900ad790 msgr2=0x7f46900afc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:42.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.928+0000 7f46a704d700 1 --2- 172.21.15.31:0/1586102552 >> [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905] conn(0x7f46900ad790 0x7f46900afc30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:42.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.928+0000 7f46a704d700 1 -- 172.21.15.31:0/1586102552 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f46900a3cd0 msgr2=0x7f46900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:42.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.928+0000 7f46a704d700 1 --2- 172.21.15.31:0/1586102552 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f46900a3cd0 0x7f46900a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4694005950 tx=0x7f46940058e0 comp rx=0 tx=0).stop 2024-04-26T17:54:42.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.928+0000 7f46a704d700 1 -- 172.21.15.31:0/1586102552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46a006c490 msgr2=0x7f46a0102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:42.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.928+0000 7f46a704d700 1 --2- 172.21.15.31:0/1586102552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46a006c490 0x7f46a0102d10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f468c00c970 tx=0x7f468c00cd30 comp rx=0 tx=0).stop 2024-04-26T17:54:42.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.928+0000 7f46a55ea700 1 -- 172.21.15.31:0/1586102552 reap_dead start 2024-04-26T17:54:42.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.928+0000 7f46a704d700 1 -- 172.21.15.31:0/1586102552 shutdown_connections 2024-04-26T17:54:42.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.929+0000 7f46a704d700 1 -- 172.21.15.31:0/1586102552 >> 172.21.15.31:0/1586102552 conn(0x7f46a00792d0 msgr2=0x7f46a0100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:42.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.929+0000 7f46a704d700 1 -- 172.21.15.31:0/1586102552 shutdown_connections 2024-04-26T17:54:42.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.929+0000 7f46a704d700 1 -- 172.21.15.31:0/1586102552 wait complete. 2024-04-26T17:54:43.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.800+0000 7f424363e700 1 -- 172.21.15.31:0/2725982140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f423c1094f0 msgr2=0x7f423c1098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:43.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.800+0000 7f424363e700 1 --2- 172.21.15.31:0/2725982140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f423c1094f0 0x7f423c1098d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4238009990 tx=0x7f4238009ca0 comp rx=0 tx=0).stop 2024-04-26T17:54:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.801+0000 7f424363e700 1 -- 172.21.15.31:0/2725982140 shutdown_connections 2024-04-26T17:54:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.801+0000 7f424363e700 1 --2- 172.21.15.31:0/2725982140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f423c103e90 0x7f423c1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.801+0000 7f424363e700 1 --2- 172.21.15.31:0/2725982140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f423c1094f0 0x7f423c1098d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.801+0000 7f424363e700 1 --2- 172.21.15.31:0/2725982140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f423c1034d0 0x7f423c103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.801+0000 7f424363e700 1 -- 172.21.15.31:0/2725982140 >> 172.21.15.31:0/2725982140 conn(0x7f423c07a4b0 msgr2=0x7f423c07a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.801+0000 7f424363e700 1 -- 172.21.15.31:0/2725982140 shutdown_connections 2024-04-26T17:54:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.801+0000 7f424363e700 1 -- 172.21.15.31:0/2725982140 wait complete. 2024-04-26T17:54:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.801+0000 7f424363e700 1 Processor -- start 2024-04-26T17:54:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.801+0000 7f424363e700 1 -- start start 2024-04-26T17:54:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f424363e700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f423c1034d0 0x7f423c19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f424363e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f423c103e90 0x7f423c19fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f424363e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f423c1094f0 0x7f423c1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f424363e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f423c1136a0 con 0x7f423c1094f0 2024-04-26T17:54:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f424363e700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f423c113520 con 0x7f423c1034d0 2024-04-26T17:54:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f424363e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f423c113820 con 0x7f423c103e90 2024-04-26T17:54:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f4241bdb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f423c1094f0 0x7f423c1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f4241bdb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f423c1094f0 0x7f423c1999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50916/0 (socket says 172.21.15.31:50916) 2024-04-26T17:54:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f4241bdb700 1 -- 172.21.15.31:0/3089020616 learned_addr learned my addr 172.21.15.31:0/3089020616 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f4240bd9700 1 --2- 172.21.15.31:0/3089020616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f423c103e90 0x7f423c19fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f42413da700 1 --2- 172.21.15.31:0/3089020616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f423c1034d0 0x7f423c19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f4241bdb700 1 -- 172.21.15.31:0/3089020616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f423c103e90 msgr2=0x7f423c19fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f4241bdb700 1 --2- 172.21.15.31:0/3089020616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f423c103e90 0x7f423c19fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f4241bdb700 1 -- 172.21.15.31:0/3089020616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f423c1034d0 msgr2=0x7f423c19f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f4241bdb700 1 --2- 172.21.15.31:0/3089020616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f423c1034d0 0x7f423c19f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f4241bdb700 1 -- 172.21.15.31:0/3089020616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f423c19a140 con 0x7f423c1094f0 2024-04-26T17:54:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f4241bdb700 1 --2- 172.21.15.31:0/3089020616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f423c1094f0 0x7f423c1999e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f423400b970 tx=0x7f423400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f42327fc700 1 -- 172.21.15.31:0/3089020616 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4234019070 con 0x7f423c1094f0 2024-04-26T17:54:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f42327fc700 1 -- 172.21.15.31:0/3089020616 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f423400cde0 con 0x7f423c1094f0 2024-04-26T17:54:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.802+0000 7f424363e700 1 -- 172.21.15.31:0/3089020616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f423c19a430 con 0x7f423c1094f0 2024-04-26T17:54:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.803+0000 7f42327fc700 1 -- 172.21.15.31:0/3089020616 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4234007370 con 0x7f423c1094f0 2024-04-26T17:54:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.803+0000 7f424363e700 1 -- 172.21.15.31:0/3089020616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f423c1a66a0 con 0x7f423c1094f0 2024-04-26T17:54:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.803+0000 7f424363e700 1 -- 172.21.15.31:0/3089020616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f423c063b30 con 0x7f423c1094f0 2024-04-26T17:54:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.805+0000 7f42327fc700 1 -- 172.21.15.31:0/3089020616 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f423401c070 con 0x7f423c1094f0 2024-04-26T17:54:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.805+0000 7f42327fc700 1 --2- 172.21.15.31:0/3089020616 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f42280a3cd0 0x7f42280a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.805+0000 7f42413da700 1 --2- 172.21.15.31:0/3089020616 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f42280a3cd0 0x7f42280a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.805+0000 7f42327fc700 1 -- 172.21.15.31:0/3089020616 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f42340d0350 con 0x7f423c1094f0 2024-04-26T17:54:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.806+0000 7f42327fc700 1 --2- 172.21.15.31:0/3089020616 >> [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903] conn(0x7f42280ad790 0x7f42280afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.806+0000 7f42327fc700 1 -- 172.21.15.31:0/3089020616 --> [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4234004860 con 0x7f42280ad790 2024-04-26T17:54:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.806+0000 7f42327fc700 1 -- 172.21.15.31:0/3089020616 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f42340d07a0 con 0x7f423c1094f0 2024-04-26T17:54:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.806+0000 7f42413da700 1 --2- 172.21.15.31:0/3089020616 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f42280a3cd0 0x7f42280a6190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4238006010 tx=0x7f423800b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.806+0000 7f4240bd9700 1 --2- 172.21.15.31:0/3089020616 >> [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903] conn(0x7f42280ad790 0x7f42280afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.806+0000 7f4240bd9700 1 --2- 172.21.15.31:0/3089020616 >> [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903] conn(0x7f42280ad790 0x7f42280afc30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.807+0000 7f42327fc700 1 -- 172.21.15.31:0/3089020616 <== osd.10 v2:172.21.15.97:6824/4097344903 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4234004860 con 0x7f42280ad790 2024-04-26T17:54:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.824+0000 7f424363e700 1 -- 172.21.15.31:0/3089020616 --> [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f423c063fb0 con 0x7f42280ad790 2024-04-26T17:54:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.825+0000 7f42327fc700 1 -- 172.21.15.31:0/3089020616 <== osd.10 v2:172.21.15.97:6824/4097344903 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f423c063fb0 con 0x7f42280ad790 2024-04-26T17:54:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.825+0000 7f424363e700 1 -- 172.21.15.31:0/3089020616 >> [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903] conn(0x7f42280ad790 msgr2=0x7f42280afc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.825+0000 7f424363e700 1 --2- 172.21.15.31:0/3089020616 >> [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903] conn(0x7f42280ad790 0x7f42280afc30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.825+0000 7f424363e700 1 -- 172.21.15.31:0/3089020616 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f42280a3cd0 msgr2=0x7f42280a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.825+0000 7f424363e700 1 --2- 172.21.15.31:0/3089020616 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f42280a3cd0 0x7f42280a6190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4238006010 tx=0x7f423800b540 comp rx=0 tx=0).stop 2024-04-26T17:54:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.825+0000 7f424363e700 1 -- 172.21.15.31:0/3089020616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f423c1094f0 msgr2=0x7f423c1999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.825+0000 7f424363e700 1 --2- 172.21.15.31:0/3089020616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f423c1094f0 0x7f423c1999e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f423400b970 tx=0x7f423400bc80 comp rx=0 tx=0).stop 2024-04-26T17:54:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.825+0000 7f4241bdb700 1 -- 172.21.15.31:0/3089020616 reap_dead start 2024-04-26T17:54:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.825+0000 7f424363e700 1 -- 172.21.15.31:0/3089020616 shutdown_connections 2024-04-26T17:54:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.825+0000 7f424363e700 1 -- 172.21.15.31:0/3089020616 >> 172.21.15.31:0/3089020616 conn(0x7f423c07a4b0 msgr2=0x7f423c102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.826+0000 7f424363e700 1 -- 172.21.15.31:0/3089020616 shutdown_connections 2024-04-26T17:54:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:42.826+0000 7f424363e700 1 -- 172.21.15.31:0/3089020616 wait complete. 2024-04-26T17:54:43.921 INFO:teuthology.orchestra.run.smithi031.stdout:309237645353 2024-04-26T17:54:43.922 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-26T17:54:44.227 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:44 smithi031 ceph-mon[29164]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:54:44.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:44 smithi097 ceph-mon[32419]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:54:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:44 smithi186 ceph-mon[32797]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:54:44.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.333+0000 7f9e77641700 1 -- 172.21.15.31:0/3515247131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e70103ea0 msgr2=0x7f9e701102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:44.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.333+0000 7f9e77641700 1 --2- 172.21.15.31:0/3515247131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e70103ea0 0x7f9e701102c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800b0a0 tx=0x7f9e6800b3b0 comp rx=0 tx=0).stop 2024-04-26T17:54:44.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.333+0000 7f9e77641700 1 -- 172.21.15.31:0/3515247131 shutdown_connections 2024-04-26T17:54:44.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.333+0000 7f9e77641700 1 --2- 172.21.15.31:0/3515247131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e70109480 0x7f9e70109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:44.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.333+0000 7f9e77641700 1 --2- 172.21.15.31:0/3515247131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e70103ea0 0x7f9e701102c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:44.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.333+0000 7f9e77641700 1 --2- 172.21.15.31:0/3515247131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e701034e0 0x7f9e70103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.333+0000 7f9e77641700 1 -- 172.21.15.31:0/3515247131 >> 172.21.15.31:0/3515247131 conn(0x7f9e7007a5e0 msgr2=0x7f9e7007a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.333+0000 7f9e77641700 1 -- 172.21.15.31:0/3515247131 shutdown_connections 2024-04-26T17:54:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.333+0000 7f9e77641700 1 -- 172.21.15.31:0/3515247131 wait complete. 2024-04-26T17:54:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e77641700 1 Processor -- start 2024-04-26T17:54:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e77641700 1 -- start start 2024-04-26T17:54:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e77641700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e701034e0 0x7f9e70075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e77641700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e70103ea0 0x7f9e70075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e77641700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e70109480 0x7f9e700761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e77641700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9e701135f0 con 0x7f9e701034e0 2024-04-26T17:54:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e77641700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f9e70113470 con 0x7f9e70103ea0 2024-04-26T17:54:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e77641700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9e70113770 con 0x7f9e70109480 2024-04-26T17:54:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e753dd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e701034e0 0x7f9e70075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e753dd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e701034e0 0x7f9e70075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50950/0 (socket says 172.21.15.31:50950) 2024-04-26T17:54:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e753dd700 1 -- 172.21.15.31:0/4229637839 learned_addr learned my addr 172.21.15.31:0/4229637839 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e74bdc700 1 --2- 172.21.15.31:0/4229637839 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e70103ea0 0x7f9e70075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e75bde700 1 --2- 172.21.15.31:0/4229637839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e70109480 0x7f9e700761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e753dd700 1 -- 172.21.15.31:0/4229637839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e70109480 msgr2=0x7f9e700761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e753dd700 1 --2- 172.21.15.31:0/4229637839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e70109480 0x7f9e700761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e753dd700 1 -- 172.21.15.31:0/4229637839 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e70103ea0 msgr2=0x7f9e70075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e753dd700 1 --2- 172.21.15.31:0/4229637839 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9e70103ea0 0x7f9e70075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.334+0000 7f9e753dd700 1 -- 172.21.15.31:0/4229637839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e701a8040 con 0x7f9e701034e0 2024-04-26T17:54:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.335+0000 7f9e753dd700 1 --2- 172.21.15.31:0/4229637839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e701034e0 0x7f9e70075720 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c00d950 tx=0x7f9e6c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.335+0000 7f9e667fc700 1 -- 172.21.15.31:0/4229637839 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c016070 con 0x7f9e701034e0 2024-04-26T17:54:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.335+0000 7f9e77641700 1 -- 172.21.15.31:0/4229637839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e701a8330 con 0x7f9e701034e0 2024-04-26T17:54:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.335+0000 7f9e667fc700 1 -- 172.21.15.31:0/4229637839 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e6c009df0 con 0x7f9e701034e0 2024-04-26T17:54:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.335+0000 7f9e77641700 1 -- 172.21.15.31:0/4229637839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e701a88c0 con 0x7f9e701034e0 2024-04-26T17:54:44.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.335+0000 7f9e667fc700 1 -- 172.21.15.31:0/4229637839 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c00f3c0 con 0x7f9e701034e0 2024-04-26T17:54:44.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.336+0000 7f9e77641700 1 -- 172.21.15.31:0/4229637839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9e70063b30 con 0x7f9e701034e0 2024-04-26T17:54:44.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.337+0000 7f9e667fc700 1 -- 172.21.15.31:0/4229637839 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9e6c019070 con 0x7f9e701034e0 2024-04-26T17:54:44.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.337+0000 7f9e667fc700 1 --2- 172.21.15.31:0/4229637839 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9e5c0a3cd0 0x7f9e5c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:44.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.338+0000 7f9e74bdc700 1 --2- 172.21.15.31:0/4229637839 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9e5c0a3cd0 0x7f9e5c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:44.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.338+0000 7f9e667fc700 1 -- 172.21.15.31:0/4229637839 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f9e6c0d0640 con 0x7f9e701034e0 2024-04-26T17:54:44.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.338+0000 7f9e667fc700 1 --2- 172.21.15.31:0/4229637839 >> [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075] conn(0x7f9e5c0ad790 0x7f9e5c0afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:44.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.338+0000 7f9e667fc700 1 -- 172.21.15.31:0/4229637839 --> [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9e6c014e80 con 0x7f9e5c0ad790 2024-04-26T17:54:44.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.338+0000 7f9e667fc700 1 -- 172.21.15.31:0/4229637839 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f9e6c0d0a90 con 0x7f9e701034e0 2024-04-26T17:54:44.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.338+0000 7f9e74bdc700 1 --2- 172.21.15.31:0/4229637839 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9e5c0a3cd0 0x7f9e5c0a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9e7007ac70 tx=0x7f9e6000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:44.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.338+0000 7f9e75bde700 1 --2- 172.21.15.31:0/4229637839 >> [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075] conn(0x7f9e5c0ad790 0x7f9e5c0afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:44.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.338+0000 7f9e75bde700 1 --2- 172.21.15.31:0/4229637839 >> [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075] conn(0x7f9e5c0ad790 0x7f9e5c0afc30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:44.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.339+0000 7f9e667fc700 1 -- 172.21.15.31:0/4229637839 <== osd.7 v2:172.21.15.97:6816/3940083075 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9e6c014e80 con 0x7f9e5c0ad790 2024-04-26T17:54:44.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.356+0000 7f9e77641700 1 -- 172.21.15.31:0/4229637839 --> [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9e70063f30 con 0x7f9e5c0ad790 2024-04-26T17:54:44.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.357+0000 7f9e667fc700 1 -- 172.21.15.31:0/4229637839 <== osd.7 v2:172.21.15.97:6816/3940083075 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9e70063f30 con 0x7f9e5c0ad790 2024-04-26T17:54:44.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.357+0000 7f9e77641700 1 -- 172.21.15.31:0/4229637839 >> [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075] conn(0x7f9e5c0ad790 msgr2=0x7f9e5c0afc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:44.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.357+0000 7f9e77641700 1 --2- 172.21.15.31:0/4229637839 >> [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075] conn(0x7f9e5c0ad790 0x7f9e5c0afc30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:44.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.357+0000 7f9e77641700 1 -- 172.21.15.31:0/4229637839 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9e5c0a3cd0 msgr2=0x7f9e5c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:44.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.357+0000 7f9e77641700 1 --2- 172.21.15.31:0/4229637839 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9e5c0a3cd0 0x7f9e5c0a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9e7007ac70 tx=0x7f9e6000b410 comp rx=0 tx=0).stop 2024-04-26T17:54:44.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.357+0000 7f9e77641700 1 -- 172.21.15.31:0/4229637839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e701034e0 msgr2=0x7f9e70075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:44.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.357+0000 7f9e77641700 1 --2- 172.21.15.31:0/4229637839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e701034e0 0x7f9e70075720 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c00d950 tx=0x7f9e6c00dc60 comp rx=0 tx=0).stop 2024-04-26T17:54:44.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.357+0000 7f9e75bde700 1 -- 172.21.15.31:0/4229637839 reap_dead start 2024-04-26T17:54:44.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.357+0000 7f9e77641700 1 -- 172.21.15.31:0/4229637839 shutdown_connections 2024-04-26T17:54:44.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.357+0000 7f9e77641700 1 -- 172.21.15.31:0/4229637839 >> 172.21.15.31:0/4229637839 conn(0x7f9e7007a5e0 msgr2=0x7f9e701015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:44.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.357+0000 7f9e77641700 1 -- 172.21.15.31:0/4229637839 shutdown_connections 2024-04-26T17:54:44.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:44.357+0000 7f9e77641700 1 -- 172.21.15.31:0/4229637839 wait complete. 2024-04-26T17:54:45.558 INFO:teuthology.orchestra.run.smithi031.stdout:219043332152 2024-04-26T17:54:45.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-26T17:54:45.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.948+0000 7f756e2a2700 1 -- 172.21.15.31:0/1915794625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75681094c0 msgr2=0x7f75681098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:45.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.948+0000 7f756e2a2700 1 --2- 172.21.15.31:0/1915794625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75681094c0 0x7f75681098a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f755c009990 tx=0x7f755c009ca0 comp rx=0 tx=0).stop 2024-04-26T17:54:45.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.948+0000 7f756e2a2700 1 -- 172.21.15.31:0/1915794625 shutdown_connections 2024-04-26T17:54:45.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.948+0000 7f756e2a2700 1 --2- 172.21.15.31:0/1915794625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7568103e30 0x7f7568110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:45.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.948+0000 7f756e2a2700 1 --2- 172.21.15.31:0/1915794625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75681094c0 0x7f75681098a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:45.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.948+0000 7f756e2a2700 1 --2- 172.21.15.31:0/1915794625 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7568103470 0x7f75681038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:45.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.948+0000 7f756e2a2700 1 -- 172.21.15.31:0/1915794625 >> 172.21.15.31:0/1915794625 conn(0x7f756807a580 msgr2=0x7f756807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:45.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.948+0000 7f756e2a2700 1 -- 172.21.15.31:0/1915794625 shutdown_connections 2024-04-26T17:54:45.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.948+0000 7f756e2a2700 1 -- 172.21.15.31:0/1915794625 wait complete. 2024-04-26T17:54:45.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.949+0000 7f756e2a2700 1 Processor -- start 2024-04-26T17:54:45.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.949+0000 7f756e2a2700 1 -- start start 2024-04-26T17:54:45.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.949+0000 7f756e2a2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7568103470 0x7f7568075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:45.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.949+0000 7f756e2a2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7568103e30 0x7f7568075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.949+0000 7f756e2a2700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75681094c0 0x7f75680761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.949+0000 7f756e2a2700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7568113640 con 0x7f7568103e30 2024-04-26T17:54:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.949+0000 7f756e2a2700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f75681134c0 con 0x7f75681094c0 2024-04-26T17:54:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.949+0000 7f756e2a2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75681137c0 con 0x7f7568103470 2024-04-26T17:54:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.949+0000 7f75677fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7568103e30 0x7f7568075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.949+0000 7f75677fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7568103e30 0x7f7568075c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41028/0 (socket says 172.21.15.31:41028) 2024-04-26T17:54:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.949+0000 7f75677fe700 1 -- 172.21.15.31:0/1317973602 learned_addr learned my addr 172.21.15.31:0/1317973602 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.949+0000 7f7567fff700 1 --2- 172.21.15.31:0/1317973602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7568103470 0x7f7568075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.949+0000 7f756c83f700 1 --2- 172.21.15.31:0/1317973602 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75681094c0 0x7f75680761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:45.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.950+0000 7f75677fe700 1 -- 172.21.15.31:0/1317973602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7568103470 msgr2=0x7f7568075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:45.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.950+0000 7f75677fe700 1 --2- 172.21.15.31:0/1317973602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7568103470 0x7f7568075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:45.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.950+0000 7f75677fe700 1 -- 172.21.15.31:0/1317973602 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75681094c0 msgr2=0x7f75680761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:45.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.950+0000 7f75677fe700 1 --2- 172.21.15.31:0/1317973602 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75681094c0 0x7f75680761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:45.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.950+0000 7f75677fe700 1 -- 172.21.15.31:0/1317973602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75681a8080 con 0x7f7568103e30 2024-04-26T17:54:45.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.950+0000 7f75677fe700 1 --2- 172.21.15.31:0/1317973602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7568103e30 0x7f7568075c60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f755800c9b0 tx=0x7f755800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:45.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.950+0000 7f75657fa700 1 -- 172.21.15.31:0/1317973602 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7558007b00 con 0x7f7568103e30 2024-04-26T17:54:45.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.950+0000 7f75657fa700 1 -- 172.21.15.31:0/1317973602 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7558007ca0 con 0x7f7568103e30 2024-04-26T17:54:45.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.951+0000 7f75657fa700 1 -- 172.21.15.31:0/1317973602 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7558018730 con 0x7f7568103e30 2024-04-26T17:54:45.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.951+0000 7f756e2a2700 1 -- 172.21.15.31:0/1317973602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75681a8370 con 0x7f7568103e30 2024-04-26T17:54:45.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.951+0000 7f756e2a2700 1 -- 172.21.15.31:0/1317973602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75681a8900 con 0x7f7568103e30 2024-04-26T17:54:45.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.951+0000 7f756e2a2700 1 -- 172.21.15.31:0/1317973602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7568063b30 con 0x7f7568103e30 2024-04-26T17:54:45.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.953+0000 7f75657fa700 1 -- 172.21.15.31:0/1317973602 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f755800f450 con 0x7f7568103e30 2024-04-26T17:54:45.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.954+0000 7f75657fa700 1 --2- 172.21.15.31:0/1317973602 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f75500a3d50 0x7f75500a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:45.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.954+0000 7f7567fff700 1 --2- 172.21.15.31:0/1317973602 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f75500a3d50 0x7f75500a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:45.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.954+0000 7f75657fa700 1 -- 172.21.15.31:0/1317973602 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f75580d0730 con 0x7f7568103e30 2024-04-26T17:54:45.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.954+0000 7f75657fa700 1 --2- 172.21.15.31:0/1317973602 >> [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352] conn(0x7f75500ad810 0x7f75500afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:45.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.954+0000 7f75657fa700 1 -- 172.21.15.31:0/1317973602 --> [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7558022e80 con 0x7f75500ad810 2024-04-26T17:54:45.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.954+0000 7f75657fa700 1 -- 172.21.15.31:0/1317973602 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f75580d0b80 con 0x7f7568103e30 2024-04-26T17:54:45.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.954+0000 7f7567fff700 1 --2- 172.21.15.31:0/1317973602 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f75500a3d50 0x7f75500a6210 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f755c00b5c0 tx=0x7f755c005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:45.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.954+0000 7f756c83f700 1 --2- 172.21.15.31:0/1317973602 >> [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352] conn(0x7f75500ad810 0x7f75500afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:45.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.955+0000 7f756c83f700 1 --2- 172.21.15.31:0/1317973602 >> [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352] conn(0x7f75500ad810 0x7f75500afcb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:45.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.956+0000 7f75657fa700 1 -- 172.21.15.31:0/1317973602 <== osd.11 v2:172.21.15.186:6824/1063973352 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7558022e80 con 0x7f75500ad810 2024-04-26T17:54:45.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.974+0000 7f756e2a2700 1 -- 172.21.15.31:0/1317973602 --> [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7568063f30 con 0x7f75500ad810 2024-04-26T17:54:45.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.974+0000 7f75657fa700 1 -- 172.21.15.31:0/1317973602 <== osd.11 v2:172.21.15.186:6824/1063973352 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7568063f30 con 0x7f75500ad810 2024-04-26T17:54:45.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.974+0000 7f756e2a2700 1 -- 172.21.15.31:0/1317973602 >> [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352] conn(0x7f75500ad810 msgr2=0x7f75500afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:45.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.974+0000 7f756e2a2700 1 --2- 172.21.15.31:0/1317973602 >> [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352] conn(0x7f75500ad810 0x7f75500afcb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:45.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.975+0000 7f756e2a2700 1 -- 172.21.15.31:0/1317973602 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f75500a3d50 msgr2=0x7f75500a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:45.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.975+0000 7f756e2a2700 1 --2- 172.21.15.31:0/1317973602 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f75500a3d50 0x7f75500a6210 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f755c00b5c0 tx=0x7f755c005e80 comp rx=0 tx=0).stop 2024-04-26T17:54:45.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.975+0000 7f756e2a2700 1 -- 172.21.15.31:0/1317973602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7568103e30 msgr2=0x7f7568075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:45.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.975+0000 7f756e2a2700 1 --2- 172.21.15.31:0/1317973602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7568103e30 0x7f7568075c60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f755800c9b0 tx=0x7f755800cd70 comp rx=0 tx=0).stop 2024-04-26T17:54:45.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.975+0000 7f756c83f700 1 -- 172.21.15.31:0/1317973602 reap_dead start 2024-04-26T17:54:45.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.975+0000 7f756e2a2700 1 -- 172.21.15.31:0/1317973602 shutdown_connections 2024-04-26T17:54:45.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.975+0000 7f756e2a2700 1 -- 172.21.15.31:0/1317973602 >> 172.21.15.31:0/1317973602 conn(0x7f756807a580 msgr2=0x7f75681021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:45.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.975+0000 7f756e2a2700 1 -- 172.21.15.31:0/1317973602 shutdown_connections 2024-04-26T17:54:45.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:45.975+0000 7f756e2a2700 1 -- 172.21.15.31:0/1317973602 wait complete. 2024-04-26T17:54:46.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:46 smithi097 ceph-mon[32419]: pgmap v348: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T17:54:46.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:46 smithi186 ceph-mon[32797]: pgmap v348: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T17:54:46.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:46 smithi031 ceph-mon[29164]: pgmap v348: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T17:54:46.697 INFO:teuthology.orchestra.run.smithi031.stdout:287762808877 2024-04-26T17:54:46.697 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-26T17:54:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.883+0000 7f6a3d564700 1 -- 172.21.15.31:0/18319173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a38109490 msgr2=0x7f6a38109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.883+0000 7f6a3d564700 1 --2- 172.21.15.31:0/18319173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a38109490 0x7f6a38109870 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6a28009990 tx=0x7f6a28009ca0 comp rx=0 tx=0).stop 2024-04-26T17:54:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.884+0000 7f6a3d564700 1 -- 172.21.15.31:0/18319173 shutdown_connections 2024-04-26T17:54:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.884+0000 7f6a3d564700 1 --2- 172.21.15.31:0/18319173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a3806c360 0x7f6a381102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.884+0000 7f6a3d564700 1 --2- 172.21.15.31:0/18319173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a38109490 0x7f6a38109870 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.884+0000 7f6a3d564700 1 --2- 172.21.15.31:0/18319173 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a3806b9c0 0x7f6a3806be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.884+0000 7f6a3d564700 1 -- 172.21.15.31:0/18319173 >> 172.21.15.31:0/18319173 conn(0x7f6a38079290 msgr2=0x7f6a380796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.884+0000 7f6a3d564700 1 -- 172.21.15.31:0/18319173 shutdown_connections 2024-04-26T17:54:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.884+0000 7f6a3d564700 1 -- 172.21.15.31:0/18319173 wait complete. 2024-04-26T17:54:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.885+0000 7f6a3d564700 1 Processor -- start 2024-04-26T17:54:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.885+0000 7f6a3d564700 1 -- start start 2024-04-26T17:54:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.885+0000 7f6a3d564700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a3806b9c0 0x7f6a3819bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.885+0000 7f6a3d564700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a3806c360 0x7f6a3819c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.885+0000 7f6a3d564700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a38109490 0x7f6a381a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.885+0000 7f6a3d564700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6a38115860 con 0x7f6a3806c360 2024-04-26T17:54:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.885+0000 7f6a3d564700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6a381156e0 con 0x7f6a38109490 2024-04-26T17:54:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.885+0000 7f6a3d564700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6a381159e0 con 0x7f6a3806b9c0 2024-04-26T17:54:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.885+0000 7f6a367fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a3806c360 0x7f6a3819c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.885+0000 7f6a367fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a3806c360 0x7f6a3819c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41050/0 (socket says 172.21.15.31:41050) 2024-04-26T17:54:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.885+0000 7f6a367fc700 1 -- 172.21.15.31:0/82757671 learned_addr learned my addr 172.21.15.31:0/82757671 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.885+0000 7f6a36ffd700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a3806b9c0 0x7f6a3819bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.886+0000 7f6a367fc700 1 -- 172.21.15.31:0/82757671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a3806b9c0 msgr2=0x7f6a3819bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.886+0000 7f6a377fe700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a38109490 0x7f6a381a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.886+0000 7f6a367fc700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a3806b9c0 0x7f6a3819bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.886+0000 7f6a367fc700 1 -- 172.21.15.31:0/82757671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a38109490 msgr2=0x7f6a381a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.886+0000 7f6a367fc700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a38109490 0x7f6a381a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.886+0000 7f6a367fc700 1 -- 172.21.15.31:0/82757671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a381a0d60 con 0x7f6a3806c360 2024-04-26T17:54:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.886+0000 7f6a367fc700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a3806c360 0x7f6a3819c290 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00e9c0 tx=0x7f6a2c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.886+0000 7f6a377fe700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a38109490 0x7f6a381a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:54:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.886+0000 7f6a1ffff700 1 -- 172.21.15.31:0/82757671 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c00cbd0 con 0x7f6a3806c360 2024-04-26T17:54:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.886+0000 7f6a3d564700 1 -- 172.21.15.31:0/82757671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a381a1050 con 0x7f6a3806c360 2024-04-26T17:54:46.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.886+0000 7f6a3d564700 1 -- 172.21.15.31:0/82757671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a381a88d0 con 0x7f6a3806c360 2024-04-26T17:54:46.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.887+0000 7f6a1ffff700 1 -- 172.21.15.31:0/82757671 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a2c00cd70 con 0x7f6a3806c360 2024-04-26T17:54:46.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.887+0000 7f6a1ffff700 1 -- 172.21.15.31:0/82757671 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c005610 con 0x7f6a3806c360 2024-04-26T17:54:46.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.887+0000 7f6a3d564700 1 -- 172.21.15.31:0/82757671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6a24001030 con 0x7f6a3806c360 2024-04-26T17:54:46.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.890+0000 7f6a1ffff700 1 -- 172.21.15.31:0/82757671 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6a2c004750 con 0x7f6a3806c360 2024-04-26T17:54:46.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.890+0000 7f6a1ffff700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6a200a3d50 0x7f6a200a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:46.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.890+0000 7f6a1ffff700 1 -- 172.21.15.31:0/82757671 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f6a2c0d0d30 con 0x7f6a3806c360 2024-04-26T17:54:46.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.890+0000 7f6a36ffd700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6a200a3d50 0x7f6a200a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:46.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.891+0000 7f6a1ffff700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395] conn(0x7f6a200ad810 0x7f6a200afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:46.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.891+0000 7f6a1ffff700 1 -- 172.21.15.31:0/82757671 --> [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a2c01ee50 con 0x7f6a200ad810 2024-04-26T17:54:46.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.891+0000 7f6a1ffff700 1 -- 172.21.15.31:0/82757671 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f6a2c0d1170 con 0x7f6a3806c360 2024-04-26T17:54:46.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.891+0000 7f6a377fe700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395] conn(0x7f6a200ad810 0x7f6a200afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:46.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.891+0000 7f6a36ffd700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6a200a3d50 0x7f6a200a6210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6a2800b2e0 tx=0x7f6a28005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:46.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.891+0000 7f6a377fe700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395] conn(0x7f6a200ad810 0x7f6a200afcb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:46.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.892+0000 7f6a1ffff700 1 -- 172.21.15.31:0/82757671 <== osd.6 v2:172.21.15.31:6818/2955759395 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a2c01ee50 con 0x7f6a200ad810 2024-04-26T17:54:46.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.910+0000 7f6a3d564700 1 -- 172.21.15.31:0/82757671 --> [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6a24002e60 con 0x7f6a200ad810 2024-04-26T17:54:46.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.910+0000 7f6a1ffff700 1 -- 172.21.15.31:0/82757671 <== osd.6 v2:172.21.15.31:6818/2955759395 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6a24002e60 con 0x7f6a200ad810 2024-04-26T17:54:46.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.911+0000 7f6a3d564700 1 -- 172.21.15.31:0/82757671 >> [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395] conn(0x7f6a200ad810 msgr2=0x7f6a200afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:46.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.911+0000 7f6a3d564700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395] conn(0x7f6a200ad810 0x7f6a200afcb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:46.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.911+0000 7f6a3d564700 1 -- 172.21.15.31:0/82757671 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6a200a3d50 msgr2=0x7f6a200a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:46.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.911+0000 7f6a3d564700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6a200a3d50 0x7f6a200a6210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6a2800b2e0 tx=0x7f6a28005fd0 comp rx=0 tx=0).stop 2024-04-26T17:54:46.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.911+0000 7f6a3d564700 1 -- 172.21.15.31:0/82757671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a3806c360 msgr2=0x7f6a3819c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.911+0000 7f6a3d564700 1 --2- 172.21.15.31:0/82757671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a3806c360 0x7f6a3819c290 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00e9c0 tx=0x7f6a2c00ecd0 comp rx=0 tx=0).stop 2024-04-26T17:54:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.911+0000 7f6a377fe700 1 -- 172.21.15.31:0/82757671 reap_dead start 2024-04-26T17:54:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.911+0000 7f6a3d564700 1 -- 172.21.15.31:0/82757671 shutdown_connections 2024-04-26T17:54:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.911+0000 7f6a3d564700 1 -- 172.21.15.31:0/82757671 >> 172.21.15.31:0/82757671 conn(0x7f6a38079290 msgr2=0x7f6a381004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.911+0000 7f6a3d564700 1 -- 172.21.15.31:0/82757671 shutdown_connections 2024-04-26T17:54:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:46.911+0000 7f6a3d564700 1 -- 172.21.15.31:0/82757671 wait complete. 2024-04-26T17:54:47.557 INFO:teuthology.orchestra.run.smithi031.stdout:343597383716 2024-04-26T17:54:47.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-26T17:54:47.773 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:48.225 INFO:teuthology.orchestra.run.smithi031.stdout:197568495678 2024-04-26T17:54:48.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-26T17:54:48.255 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:48.260 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:48.261 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:48.261 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:48.300 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:48.314 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:48.327 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:48.368 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:54:48.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:48 smithi097 ceph-mon[32419]: pgmap v349: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T17:54:48.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:48 smithi031 ceph-mon[29164]: pgmap v349: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T17:54:48.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:48 smithi186 ceph-mon[32797]: pgmap v349: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T17:54:50.173 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:50 smithi031 ceph-mon[29164]: pgmap v350: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T17:54:50.173 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:50 smithi031 ceph-mon[29164]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-26T17:54:50.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:50 smithi097 ceph-mon[32419]: pgmap v350: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T17:54:50.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:50 smithi097 ceph-mon[32419]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-26T17:54:50.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:50 smithi186 ceph-mon[32797]: pgmap v350: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T17:54:50.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:50 smithi186 ceph-mon[32797]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-26T17:54:52.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:52 smithi097 ceph-mon[32419]: pgmap v351: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-26T17:54:52.444 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:52 smithi031 ceph-mon[29164]: pgmap v351: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-26T17:54:52.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:52 smithi186 ceph-mon[32797]: pgmap v351: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-26T17:54:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.362+0000 7f680ebb4700 1 -- 172.21.15.31:0/3110043267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6808103e60 msgr2=0x7f68081102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.362+0000 7f680ebb4700 1 --2- 172.21.15.31:0/3110043267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6808103e60 0x7f68081102b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f680000b0a0 tx=0x7f680000b3b0 comp rx=0 tx=0).stop 2024-04-26T17:54:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.362+0000 7f680ebb4700 1 -- 172.21.15.31:0/3110043267 shutdown_connections 2024-04-26T17:54:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.362+0000 7f680ebb4700 1 --2- 172.21.15.31:0/3110043267 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68081094e0 0x7f68081098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.362+0000 7f680ebb4700 1 --2- 172.21.15.31:0/3110043267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6808103e60 0x7f68081102b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.362+0000 7f680ebb4700 1 --2- 172.21.15.31:0/3110043267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68081034a0 0x7f6808103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.362+0000 7f680ebb4700 1 -- 172.21.15.31:0/3110043267 >> 172.21.15.31:0/3110043267 conn(0x7f680807a590 msgr2=0x7f680807a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.362+0000 7f680ebb4700 1 -- 172.21.15.31:0/3110043267 shutdown_connections 2024-04-26T17:54:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.362+0000 7f680ebb4700 1 -- 172.21.15.31:0/3110043267 wait complete. 2024-04-26T17:54:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.362+0000 7f680ebb4700 1 Processor -- start 2024-04-26T17:54:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.362+0000 7f680ebb4700 1 -- start start 2024-04-26T17:54:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680ebb4700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68081034a0 0x7f680819f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680ebb4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6808103e60 0x7f680819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680ebb4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68081094e0 0x7f6808199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680ebb4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f68081135e0 con 0x7f68081094e0 2024-04-26T17:54:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680ebb4700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6808113460 con 0x7f68081034a0 2024-04-26T17:54:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680ebb4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6808113760 con 0x7f6808103e60 2024-04-26T17:54:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680d151700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68081094e0 0x7f6808199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680c950700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68081034a0 0x7f680819f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680d151700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68081094e0 0x7f6808199900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41074/0 (socket says 172.21.15.31:41074) 2024-04-26T17:54:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680d151700 1 -- 172.21.15.31:0/3248695010 learned_addr learned my addr 172.21.15.31:0/3248695010 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f6807fff700 1 --2- 172.21.15.31:0/3248695010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6808103e60 0x7f680819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680d151700 1 -- 172.21.15.31:0/3248695010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6808103e60 msgr2=0x7f680819fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680d151700 1 --2- 172.21.15.31:0/3248695010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6808103e60 0x7f680819fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680d151700 1 -- 172.21.15.31:0/3248695010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68081034a0 msgr2=0x7f680819f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680d151700 1 --2- 172.21.15.31:0/3248695010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68081034a0 0x7f680819f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680d151700 1 -- 172.21.15.31:0/3248695010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f680819a060 con 0x7f68081094e0 2024-04-26T17:54:53.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f680d151700 1 --2- 172.21.15.31:0/3248695010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68081094e0 0x7f6808199900 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f68000062a0 tx=0x7f6800004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:53.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.363+0000 7f6805ffb700 1 -- 172.21.15.31:0/3248695010 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6800021070 con 0x7f68081094e0 2024-04-26T17:54:53.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.364+0000 7f680ebb4700 1 -- 172.21.15.31:0/3248695010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f680819a2f0 con 0x7f68081094e0 2024-04-26T17:54:53.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.364+0000 7f6805ffb700 1 -- 172.21.15.31:0/3248695010 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6800007e30 con 0x7f68081094e0 2024-04-26T17:54:53.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.364+0000 7f6805ffb700 1 -- 172.21.15.31:0/3248695010 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6800004780 con 0x7f68081094e0 2024-04-26T17:54:53.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.364+0000 7f680ebb4700 1 -- 172.21.15.31:0/3248695010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68081a6610 con 0x7f68081094e0 2024-04-26T17:54:53.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.365+0000 7f680ebb4700 1 -- 172.21.15.31:0/3248695010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f680806a110 con 0x7f68081094e0 2024-04-26T17:54:53.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.366+0000 7f6805ffb700 1 -- 172.21.15.31:0/3248695010 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6800007980 con 0x7f68081094e0 2024-04-26T17:54:53.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.366+0000 7f6805ffb700 1 --2- 172.21.15.31:0/3248695010 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f67f80a3c40 0x7f67f80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:53.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.366+0000 7f6805ffb700 1 -- 172.21.15.31:0/3248695010 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f680000f020 con 0x7f68081094e0 2024-04-26T17:54:53.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.366+0000 7f680c950700 1 --2- 172.21.15.31:0/3248695010 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f67f80a3c40 0x7f67f80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:53.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.367+0000 7f680c950700 1 --2- 172.21.15.31:0/3248695010 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f67f80a3c40 0x7f67f80a6100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f680819aa40 tx=0x7f67fc0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:53.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.371+0000 7f6805ffb700 1 -- 172.21.15.31:0/3248695010 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f680002a020 con 0x7f68081094e0 2024-04-26T17:54:53.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.594+0000 7f680ebb4700 1 -- 172.21.15.31:0/3248695010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f680819ae30 con 0x7f68081094e0 2024-04-26T17:54:53.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.594+0000 7f6805ffb700 1 -- 172.21.15.31:0/3248695010 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f680009e310 con 0x7f68081094e0 2024-04-26T17:54:53.595 INFO:teuthology.orchestra.run.smithi031.stdout:141733920841 2024-04-26T17:54:53.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.596+0000 7f680ebb4700 1 -- 172.21.15.31:0/3248695010 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f67f80a3c40 msgr2=0x7f67f80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:53.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.596+0000 7f680ebb4700 1 --2- 172.21.15.31:0/3248695010 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f67f80a3c40 0x7f67f80a6100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f680819aa40 tx=0x7f67fc0073d0 comp rx=0 tx=0).stop 2024-04-26T17:54:53.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.596+0000 7f680ebb4700 1 -- 172.21.15.31:0/3248695010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68081094e0 msgr2=0x7f6808199900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:53.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.596+0000 7f680ebb4700 1 --2- 172.21.15.31:0/3248695010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68081094e0 0x7f6808199900 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f68000062a0 tx=0x7f6800004c80 comp rx=0 tx=0).stop 2024-04-26T17:54:53.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.596+0000 7f680ebb4700 1 -- 172.21.15.31:0/3248695010 shutdown_connections 2024-04-26T17:54:53.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.596+0000 7f680ebb4700 1 --2- 172.21.15.31:0/3248695010 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f67f80a3c40 0x7f67f80a6100 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:53.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.596+0000 7f680ebb4700 1 --2- 172.21.15.31:0/3248695010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f68081034a0 0x7f680819f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:53.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.596+0000 7f680ebb4700 1 --2- 172.21.15.31:0/3248695010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68081094e0 0x7f6808199900 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:53.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.596+0000 7f680ebb4700 1 --2- 172.21.15.31:0/3248695010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6808103e60 0x7f680819fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:53.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.596+0000 7f680ebb4700 1 -- 172.21.15.31:0/3248695010 >> 172.21.15.31:0/3248695010 conn(0x7f680807a590 msgr2=0x7f6808102260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:53.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.596+0000 7f680ebb4700 1 -- 172.21.15.31:0/3248695010 shutdown_connections 2024-04-26T17:54:53.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:53.596+0000 7f680ebb4700 1 -- 172.21.15.31:0/3248695010 wait complete. 2024-04-26T17:54:54.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:54 smithi097 ceph-mon[32419]: pgmap v352: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T17:54:54.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:54 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3248695010' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T17:54:54.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:54 smithi186 ceph-mon[32797]: pgmap v352: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T17:54:54.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:54 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3248695010' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T17:54:54.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:54 smithi031 ceph-mon[29164]: pgmap v352: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T17:54:54.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:54 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3248695010' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T17:54:54.596 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920837 got 141733920841 for osd.4 2024-04-26T17:54:54.596 DEBUG:teuthology.parallel:result is None 2024-04-26T17:54:56.215 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:56 smithi031 ceph-mon[29164]: pgmap v353: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T17:54:56.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:56 smithi097 ceph-mon[32419]: pgmap v353: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T17:54:56.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:56 smithi186 ceph-mon[32797]: pgmap v353: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T17:54:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.771+0000 7feba4863700 1 -- 172.21.15.31:0/3674125451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba01094e0 msgr2=0x7feba01098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.771+0000 7feba4863700 1 --2- 172.21.15.31:0/3674125451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba01094e0 0x7feba01098c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7feb90009990 tx=0x7feb90009ca0 comp rx=0 tx=0).stop 2024-04-26T17:54:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.771+0000 7feba4863700 1 -- 172.21.15.31:0/3674125451 shutdown_connections 2024-04-26T17:54:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.771+0000 7feba4863700 1 --2- 172.21.15.31:0/3674125451 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feba0103e60 0x7feba01102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.771+0000 7feba4863700 1 --2- 172.21.15.31:0/3674125451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba01094e0 0x7feba01098c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.771+0000 7feba4863700 1 --2- 172.21.15.31:0/3674125451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feba01034a0 0x7feba0103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.771+0000 7feba4863700 1 -- 172.21.15.31:0/3674125451 >> 172.21.15.31:0/3674125451 conn(0x7feba007a590 msgr2=0x7feba007a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.771+0000 7feba4863700 1 -- 172.21.15.31:0/3674125451 shutdown_connections 2024-04-26T17:54:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.771+0000 7feba4863700 1 -- 172.21.15.31:0/3674125451 wait complete. 2024-04-26T17:54:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.772+0000 7feba4863700 1 Processor -- start 2024-04-26T17:54:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.772+0000 7feba4863700 1 -- start start 2024-04-26T17:54:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.772+0000 7feba4863700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feba01034a0 0x7feba0075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.772+0000 7feba4863700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feba0103e60 0x7feba0075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.772+0000 7feba4863700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba01094e0 0x7feba00761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.772+0000 7feba4863700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feba01136e0 con 0x7feba01094e0 2024-04-26T17:54:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.772+0000 7feba4863700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7feba0113560 con 0x7feba0103e60 2024-04-26T17:54:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.772+0000 7feba4863700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feba0113860 con 0x7feba01034a0 2024-04-26T17:54:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.772+0000 7feb9ed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba01094e0 0x7feba00761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.772+0000 7feb9ed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba01094e0 0x7feba00761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52664/0 (socket says 172.21.15.31:52664) 2024-04-26T17:54:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.772+0000 7feb9ed9d700 1 -- 172.21.15.31:0/4077189934 learned_addr learned my addr 172.21.15.31:0/4077189934 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.772+0000 7feb9dd9b700 1 --2- 172.21.15.31:0/4077189934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feba0103e60 0x7feba0075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.772+0000 7feb9e59c700 1 --2- 172.21.15.31:0/4077189934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feba01034a0 0x7feba0075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.773+0000 7feb9ed9d700 1 -- 172.21.15.31:0/4077189934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feba01034a0 msgr2=0x7feba0075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.773+0000 7feb9ed9d700 1 --2- 172.21.15.31:0/4077189934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feba01034a0 0x7feba0075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.773+0000 7feb9ed9d700 1 -- 172.21.15.31:0/4077189934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feba0103e60 msgr2=0x7feba0075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.773+0000 7feb9ed9d700 1 --2- 172.21.15.31:0/4077189934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feba0103e60 0x7feba0075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.773+0000 7feb9ed9d700 1 -- 172.21.15.31:0/4077189934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba01a8260 con 0x7feba01094e0 2024-04-26T17:54:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.773+0000 7feb9ed9d700 1 --2- 172.21.15.31:0/4077189934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba01094e0 0x7feba00761a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feb9800b970 tx=0x7feb9800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.773+0000 7feb8f7fe700 1 -- 172.21.15.31:0/4077189934 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb98019070 con 0x7feba01094e0 2024-04-26T17:54:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.773+0000 7feb8f7fe700 1 -- 172.21.15.31:0/4077189934 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb9800cde0 con 0x7feba01094e0 2024-04-26T17:54:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.773+0000 7feba4863700 1 -- 172.21.15.31:0/4077189934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba01a8550 con 0x7feba01094e0 2024-04-26T17:54:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.773+0000 7feb8f7fe700 1 -- 172.21.15.31:0/4077189934 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb98007370 con 0x7feba01094e0 2024-04-26T17:54:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.773+0000 7feba4863700 1 -- 172.21.15.31:0/4077189934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba01a8a40 con 0x7feba01094e0 2024-04-26T17:54:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.774+0000 7feba4863700 1 -- 172.21.15.31:0/4077189934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feba006a110 con 0x7feba01094e0 2024-04-26T17:54:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.775+0000 7feb8f7fe700 1 -- 172.21.15.31:0/4077189934 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7feb9801c070 con 0x7feba01094e0 2024-04-26T17:54:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.776+0000 7feb8f7fe700 1 --2- 172.21.15.31:0/4077189934 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb880a3c40 0x7feb880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.776+0000 7feb9e59c700 1 --2- 172.21.15.31:0/4077189934 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb880a3c40 0x7feb880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.776+0000 7feb8f7fe700 1 -- 172.21.15.31:0/4077189934 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7feb980cf960 con 0x7feba01094e0 2024-04-26T17:54:57.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.776+0000 7feb9e59c700 1 --2- 172.21.15.31:0/4077189934 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb880a3c40 0x7feb880a6100 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7feb90006010 tx=0x7feb9001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.780+0000 7feb8f7fe700 1 -- 172.21.15.31:0/4077189934 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feb9809c310 con 0x7feba01094e0 2024-04-26T17:54:57.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.993+0000 7feba4863700 1 -- 172.21.15.31:0/4077189934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7feba002e490 con 0x7feba01094e0 2024-04-26T17:54:57.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.994+0000 7feb8f7fe700 1 -- 172.21.15.31:0/4077189934 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7feb9809bc10 con 0x7feba01094e0 2024-04-26T17:54:57.994 INFO:teuthology.orchestra.run.smithi031.stdout:219043332154 2024-04-26T17:54:57.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.996+0000 7feba4863700 1 -- 172.21.15.31:0/4077189934 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb880a3c40 msgr2=0x7feb880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:57.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.996+0000 7feba4863700 1 --2- 172.21.15.31:0/4077189934 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb880a3c40 0x7feb880a6100 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7feb90006010 tx=0x7feb9001a040 comp rx=0 tx=0).stop 2024-04-26T17:54:57.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.996+0000 7feba4863700 1 -- 172.21.15.31:0/4077189934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba01094e0 msgr2=0x7feba00761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:57.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.996+0000 7feba4863700 1 --2- 172.21.15.31:0/4077189934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba01094e0 0x7feba00761a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feb9800b970 tx=0x7feb9800bc80 comp rx=0 tx=0).stop 2024-04-26T17:54:57.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.996+0000 7feba4863700 1 -- 172.21.15.31:0/4077189934 shutdown_connections 2024-04-26T17:54:57.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.996+0000 7feba4863700 1 --2- 172.21.15.31:0/4077189934 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb880a3c40 0x7feb880a6100 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:57.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.996+0000 7feba4863700 1 --2- 172.21.15.31:0/4077189934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba01094e0 0x7feba00761a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:57.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.996+0000 7feba4863700 1 --2- 172.21.15.31:0/4077189934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feba01034a0 0x7feba0075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:57.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.996+0000 7feba4863700 1 --2- 172.21.15.31:0/4077189934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feba0103e60 0x7feba0075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:57.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.996+0000 7feba4863700 1 -- 172.21.15.31:0/4077189934 >> 172.21.15.31:0/4077189934 conn(0x7feba007a590 msgr2=0x7feba0102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:57.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.996+0000 7feba4863700 1 -- 172.21.15.31:0/4077189934 shutdown_connections 2024-04-26T17:54:57.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:57.996+0000 7feba4863700 1 -- 172.21.15.31:0/4077189934 wait complete. 2024-04-26T17:54:58.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:58 smithi031 ceph-mon[29164]: pgmap v354: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T17:54:58.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:58 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/4077189934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T17:54:58.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:58 smithi097 ceph-mon[32419]: pgmap v354: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T17:54:58.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:58 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/4077189934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T17:54:58.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:58 smithi186 ceph-mon[32797]: pgmap v354: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T17:54:58.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:58 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/4077189934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T17:54:58.652 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332152 got 219043332154 for osd.7 2024-04-26T17:54:58.652 DEBUG:teuthology.parallel:result is None 2024-04-26T17:54:58.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.829+0000 7f229464c700 1 -- 172.21.15.31:0/4221371019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c06c510 msgr2=0x7f228c1102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:58.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.829+0000 7f229464c700 1 --2- 172.21.15.31:0/4221371019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c06c510 0x7f228c1102b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f228400b0a0 tx=0x7f228400b3b0 comp rx=0 tx=0).stop 2024-04-26T17:54:58.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.829+0000 7f229464c700 1 -- 172.21.15.31:0/4221371019 shutdown_connections 2024-04-26T17:54:58.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.829+0000 7f229464c700 1 --2- 172.21.15.31:0/4221371019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f228c1094e0 0x7f228c1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:58.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.829+0000 7f229464c700 1 --2- 172.21.15.31:0/4221371019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c06c510 0x7f228c1102b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:58.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.829+0000 7f229464c700 1 --2- 172.21.15.31:0/4221371019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f228c06bb50 0x7f228c06bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:58.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.829+0000 7f229464c700 1 -- 172.21.15.31:0/4221371019 >> 172.21.15.31:0/4221371019 conn(0x7f228c078190 msgr2=0x7f228c0785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:58.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.829+0000 7f229464c700 1 -- 172.21.15.31:0/4221371019 shutdown_connections 2024-04-26T17:54:58.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.829+0000 7f229464c700 1 -- 172.21.15.31:0/4221371019 wait complete. 2024-04-26T17:54:58.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f229464c700 1 Processor -- start 2024-04-26T17:54:58.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f229464c700 1 -- start start 2024-04-26T17:54:58.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f229464c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f228c06bb50 0x7f228c19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:58.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f229464c700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f228c06c510 0x7f228c19fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:58.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f229464c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c1094e0 0x7f228c1997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:58.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f229464c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f228c113610 con 0x7f228c1094e0 2024-04-26T17:54:58.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f229464c700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f228c113490 con 0x7f228c06c510 2024-04-26T17:54:58.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f229464c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f228c113790 con 0x7f228c06bb50 2024-04-26T17:54:58.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f2292be9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c1094e0 0x7f228c1997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:58.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f2292be9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c1094e0 0x7f228c1997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52680/0 (socket says 172.21.15.31:52680) 2024-04-26T17:54:58.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f2292be9700 1 -- 172.21.15.31:0/3445878437 learned_addr learned my addr 172.21.15.31:0/3445878437 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:54:58.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f2291be7700 1 --2- 172.21.15.31:0/3445878437 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f228c06c510 0x7f228c19fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:58.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f22923e8700 1 --2- 172.21.15.31:0/3445878437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f228c06bb50 0x7f228c19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f2292be9700 1 -- 172.21.15.31:0/3445878437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f228c06bb50 msgr2=0x7f228c19f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f2292be9700 1 --2- 172.21.15.31:0/3445878437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f228c06bb50 0x7f228c19f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f2292be9700 1 -- 172.21.15.31:0/3445878437 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f228c06c510 msgr2=0x7f228c19fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f2292be9700 1 --2- 172.21.15.31:0/3445878437 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f228c06c510 0x7f228c19fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f2292be9700 1 -- 172.21.15.31:0/3445878437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228c199fb0 con 0x7f228c1094e0 2024-04-26T17:54:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.830+0000 7f2292be9700 1 --2- 172.21.15.31:0/3445878437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c1094e0 0x7f228c1997d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f22840062a0 tx=0x7f2284004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.831+0000 7f22837fe700 1 -- 172.21.15.31:0/3445878437 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2284021070 con 0x7f228c1094e0 2024-04-26T17:54:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.831+0000 7f22837fe700 1 -- 172.21.15.31:0/3445878437 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2284007c80 con 0x7f228c1094e0 2024-04-26T17:54:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.831+0000 7f229464c700 1 -- 172.21.15.31:0/3445878437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f228c19a240 con 0x7f228c1094e0 2024-04-26T17:54:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.831+0000 7f22837fe700 1 -- 172.21.15.31:0/3445878437 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2284005280 con 0x7f228c1094e0 2024-04-26T17:54:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.831+0000 7f229464c700 1 -- 172.21.15.31:0/3445878437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f228c1a6460 con 0x7f228c1094e0 2024-04-26T17:54:58.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.832+0000 7f229464c700 1 -- 172.21.15.31:0/3445878437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f228c104b80 con 0x7f228c1094e0 2024-04-26T17:54:58.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.833+0000 7f22837fe700 1 -- 172.21.15.31:0/3445878437 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2284024070 con 0x7f228c1094e0 2024-04-26T17:54:58.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.833+0000 7f22837fe700 1 --2- 172.21.15.31:0/3445878437 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f22780a3c40 0x7f22780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:54:58.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.834+0000 7f22923e8700 1 --2- 172.21.15.31:0/3445878437 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f22780a3c40 0x7f22780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:54:58.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.834+0000 7f22837fe700 1 -- 172.21.15.31:0/3445878437 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f228400f020 con 0x7f228c1094e0 2024-04-26T17:54:58.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.834+0000 7f22923e8700 1 --2- 172.21.15.31:0/3445878437 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f22780a3c40 0x7f22780a6100 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2288005fd0 tx=0x7f228800af20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:54:58.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:58.837+0000 7f22837fe700 1 -- 172.21.15.31:0/3445878437 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f228402b020 con 0x7f228c1094e0 2024-04-26T17:54:59.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:59.048+0000 7f229464c700 1 -- 172.21.15.31:0/3445878437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f228c1a6690 con 0x7f228c1094e0 2024-04-26T17:54:59.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:59.049+0000 7f22837fe700 1 -- 172.21.15.31:0/3445878437 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f228409e930 con 0x7f228c1094e0 2024-04-26T17:54:59.049 INFO:teuthology.orchestra.run.smithi031.stdout:163208757317 2024-04-26T17:54:59.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:59.051+0000 7f229464c700 1 -- 172.21.15.31:0/3445878437 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f22780a3c40 msgr2=0x7f22780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:59.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:59.051+0000 7f229464c700 1 --2- 172.21.15.31:0/3445878437 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f22780a3c40 0x7f22780a6100 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2288005fd0 tx=0x7f228800af20 comp rx=0 tx=0).stop 2024-04-26T17:54:59.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:59.051+0000 7f229464c700 1 -- 172.21.15.31:0/3445878437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c1094e0 msgr2=0x7f228c1997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:54:59.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:59.051+0000 7f229464c700 1 --2- 172.21.15.31:0/3445878437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c1094e0 0x7f228c1997d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f22840062a0 tx=0x7f2284004dc0 comp rx=0 tx=0).stop 2024-04-26T17:54:59.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:59.051+0000 7f229464c700 1 -- 172.21.15.31:0/3445878437 shutdown_connections 2024-04-26T17:54:59.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:59.051+0000 7f229464c700 1 --2- 172.21.15.31:0/3445878437 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f22780a3c40 0x7f22780a6100 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:59.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:59.051+0000 7f229464c700 1 --2- 172.21.15.31:0/3445878437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f228c1094e0 0x7f228c1997d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:59.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:59.051+0000 7f229464c700 1 --2- 172.21.15.31:0/3445878437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f228c06bb50 0x7f228c19f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:59.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:59.051+0000 7f229464c700 1 --2- 172.21.15.31:0/3445878437 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f228c06c510 0x7f228c19fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:54:59.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:59.051+0000 7f229464c700 1 -- 172.21.15.31:0/3445878437 >> 172.21.15.31:0/3445878437 conn(0x7f228c078190 msgr2=0x7f228c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:54:59.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:59.051+0000 7f229464c700 1 -- 172.21.15.31:0/3445878437 shutdown_connections 2024-04-26T17:54:59.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:54:59.051+0000 7f229464c700 1 -- 172.21.15.31:0/3445878437 wait complete. 2024-04-26T17:54:59.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:54:59 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3445878437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T17:54:59.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:54:59 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3445878437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T17:54:59.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:54:59 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3445878437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T17:55:00.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:00 smithi031 ceph-mon[29164]: pgmap v355: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T17:55:00.418 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:00 smithi186 ceph-mon[32797]: pgmap v355: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T17:55:00.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:00 smithi097 ceph-mon[32419]: pgmap v355: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T17:55:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.245+0000 7ffbca182700 1 -- 172.21.15.31:0/1325056506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc406b9c0 msgr2=0x7ffbc406be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.245+0000 7ffbca182700 1 --2- 172.21.15.31:0/1325056506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc406b9c0 0x7ffbc406be20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009990 tx=0x7ffbb4009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.245+0000 7ffbca182700 1 -- 172.21.15.31:0/1325056506 shutdown_connections 2024-04-26T17:55:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.245+0000 7ffbca182700 1 --2- 172.21.15.31:0/1325056506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffbc406c360 0x7ffbc41102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.245+0000 7ffbca182700 1 --2- 172.21.15.31:0/1325056506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbc4109490 0x7ffbc4109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.245+0000 7ffbca182700 1 --2- 172.21.15.31:0/1325056506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc406b9c0 0x7ffbc406be20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.245+0000 7ffbca182700 1 -- 172.21.15.31:0/1325056506 >> 172.21.15.31:0/1325056506 conn(0x7ffbc4079290 msgr2=0x7ffbc40796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.245+0000 7ffbca182700 1 -- 172.21.15.31:0/1325056506 shutdown_connections 2024-04-26T17:55:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.245+0000 7ffbca182700 1 -- 172.21.15.31:0/1325056506 wait complete. 2024-04-26T17:55:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.246+0000 7ffbca182700 1 Processor -- start 2024-04-26T17:55:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.246+0000 7ffbca182700 1 -- start start 2024-04-26T17:55:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbca182700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbc406b9c0 0x7ffbc419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbca182700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc406c360 0x7ffbc419fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbca182700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffbc4109490 0x7ffbc41999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbca182700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc4113620 con 0x7ffbc406c360 2024-04-26T17:55:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbca182700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc41134a0 con 0x7ffbc4109490 2024-04-26T17:55:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbca182700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc41137a0 con 0x7ffbc406b9c0 2024-04-26T17:55:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbc2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc406c360 0x7ffbc419fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbc2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc406c360 0x7ffbc419fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52692/0 (socket says 172.21.15.31:52692) 2024-04-26T17:55:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbc2ffd700 1 -- 172.21.15.31:0/1259932429 learned_addr learned my addr 172.21.15.31:0/1259932429 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbc3fff700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffbc4109490 0x7ffbc41999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbc37fe700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbc406b9c0 0x7ffbc419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbc2ffd700 1 -- 172.21.15.31:0/1259932429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbc406b9c0 msgr2=0x7ffbc419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbc2ffd700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbc406b9c0 0x7ffbc419f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbc2ffd700 1 -- 172.21.15.31:0/1259932429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffbc4109490 msgr2=0x7ffbc41999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbc2ffd700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffbc4109490 0x7ffbc41999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbc2ffd700 1 -- 172.21.15.31:0/1259932429 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc419a120 con 0x7ffbc406c360 2024-04-26T17:55:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbc3fff700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffbc4109490 0x7ffbc41999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:55:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.247+0000 7ffbc37fe700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbc406b9c0 0x7ffbc419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:55:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.248+0000 7ffbc2ffd700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc406c360 0x7ffbc419fcf0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40057f0 tx=0x7ffbb400b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.248+0000 7ffbc0ff9700 1 -- 172.21.15.31:0/1259932429 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4027070 con 0x7ffbc406c360 2024-04-26T17:55:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.248+0000 7ffbc0ff9700 1 -- 172.21.15.31:0/1259932429 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb400fbb0 con 0x7ffbc406c360 2024-04-26T17:55:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.248+0000 7ffbca182700 1 -- 172.21.15.31:0/1259932429 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbc419a3b0 con 0x7ffbc406c360 2024-04-26T17:55:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.248+0000 7ffbc0ff9700 1 -- 172.21.15.31:0/1259932429 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4017650 con 0x7ffbc406c360 2024-04-26T17:55:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.248+0000 7ffbca182700 1 -- 172.21.15.31:0/1259932429 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbc41a6680 con 0x7ffbc406c360 2024-04-26T17:55:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.249+0000 7ffbca182700 1 -- 172.21.15.31:0/1259932429 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbc4104b40 con 0x7ffbc406c360 2024-04-26T17:55:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.251+0000 7ffbc0ff9700 1 -- 172.21.15.31:0/1259932429 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ffbb402a070 con 0x7ffbc406c360 2024-04-26T17:55:01.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.251+0000 7ffbc0ff9700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ffbac0a3c40 0x7ffbac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:01.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.251+0000 7ffbc37fe700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ffbac0a3c40 0x7ffbac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:01.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.251+0000 7ffbc0ff9700 1 -- 172.21.15.31:0/1259932429 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7ffbb40d15d0 con 0x7ffbc406c360 2024-04-26T17:55:01.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.252+0000 7ffbc37fe700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ffbac0a3c40 0x7ffbac0a6100 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4102280 tx=0x7ffbb8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:01.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.255+0000 7ffbc0ff9700 1 -- 172.21.15.31:0/1259932429 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffbb4031080 con 0x7ffbc406c360 2024-04-26T17:55:01.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.467+0000 7ffbca182700 1 -- 172.21.15.31:0/1259932429 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7ffbc410ba00 con 0x7ffbc406c360 2024-04-26T17:55:01.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.467+0000 7ffbc0ff9700 1 -- 172.21.15.31:0/1259932429 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ffbb4013070 con 0x7ffbc406c360 2024-04-26T17:55:01.467 INFO:teuthology.orchestra.run.smithi031.stdout:77309411413 2024-04-26T17:55:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.469+0000 7ffbca182700 1 -- 172.21.15.31:0/1259932429 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ffbac0a3c40 msgr2=0x7ffbac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.469+0000 7ffbca182700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ffbac0a3c40 0x7ffbac0a6100 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4102280 tx=0x7ffbb8008040 comp rx=0 tx=0).stop 2024-04-26T17:55:01.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.469+0000 7ffbca182700 1 -- 172.21.15.31:0/1259932429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc406c360 msgr2=0x7ffbc419fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.469+0000 7ffbca182700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc406c360 0x7ffbc419fcf0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40057f0 tx=0x7ffbb400b6c0 comp rx=0 tx=0).stop 2024-04-26T17:55:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.469+0000 7ffbca182700 1 -- 172.21.15.31:0/1259932429 shutdown_connections 2024-04-26T17:55:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.469+0000 7ffbca182700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7ffbac0a3c40 0x7ffbac0a6100 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.469+0000 7ffbca182700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffbc4109490 0x7ffbc41999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.469+0000 7ffbca182700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbc406b9c0 0x7ffbc419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.469+0000 7ffbca182700 1 --2- 172.21.15.31:0/1259932429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc406c360 0x7ffbc419fcf0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.469+0000 7ffbca182700 1 -- 172.21.15.31:0/1259932429 >> 172.21.15.31:0/1259932429 conn(0x7ffbc4079290 msgr2=0x7ffbc4100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.470+0000 7ffbca182700 1 -- 172.21.15.31:0/1259932429 shutdown_connections 2024-04-26T17:55:01.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:01.470+0000 7ffbca182700 1 -- 172.21.15.31:0/1259932429 wait complete. 2024-04-26T17:55:02.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:02 smithi031 ceph-mon[29164]: pgmap v356: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T17:55:02.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:02 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1259932429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T17:55:02.395 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757313 got 163208757317 for osd.5 2024-04-26T17:55:02.395 DEBUG:teuthology.parallel:result is None 2024-04-26T17:55:02.436 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:02 smithi097 ceph-mon[32419]: pgmap v356: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T17:55:02.436 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:02 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1259932429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T17:55:02.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:02 smithi186 ceph-mon[32797]: pgmap v356: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T17:55:02.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:02 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1259932429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T17:55:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.601+0000 7fb375850700 1 -- 172.21.15.31:0/538670843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb37006b130 msgr2=0x7fb37006b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.601+0000 7fb375850700 1 --2- 172.21.15.31:0/538670843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb37006b130 0x7fb37006b590 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb364009990 tx=0x7fb364009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.601+0000 7fb375850700 1 -- 172.21.15.31:0/538670843 shutdown_connections 2024-04-26T17:55:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.601+0000 7fb375850700 1 --2- 172.21.15.31:0/538670843 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3701094a0 0x7fb370109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.601+0000 7fb375850700 1 --2- 172.21.15.31:0/538670843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb37006bad0 0x7fb3701102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.601+0000 7fb375850700 1 --2- 172.21.15.31:0/538670843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb37006b130 0x7fb37006b590 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.601+0000 7fb375850700 1 -- 172.21.15.31:0/538670843 >> 172.21.15.31:0/538670843 conn(0x7fb3700792d0 msgr2=0x7fb3700796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.601+0000 7fb375850700 1 -- 172.21.15.31:0/538670843 shutdown_connections 2024-04-26T17:55:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.601+0000 7fb375850700 1 -- 172.21.15.31:0/538670843 wait complete. 2024-04-26T17:55:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.602+0000 7fb375850700 1 Processor -- start 2024-04-26T17:55:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.602+0000 7fb375850700 1 -- start start 2024-04-26T17:55:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.602+0000 7fb375850700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb37006b130 0x7fb3701023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.602+0000 7fb375850700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb37006bad0 0x7fb370102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.602+0000 7fb375850700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3701094a0 0x7fb370102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb375850700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb3701135d0 con 0x7fb3701094a0 2024-04-26T17:55:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb375850700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb370113450 con 0x7fb37006b130 2024-04-26T17:55:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb375850700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb370113750 con 0x7fb37006bad0 2024-04-26T17:55:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb36f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3701094a0 0x7fb370102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb36f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3701094a0 0x7fb370102e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52714/0 (socket says 172.21.15.31:52714) 2024-04-26T17:55:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb36f7fe700 1 -- 172.21.15.31:0/2895781261 learned_addr learned my addr 172.21.15.31:0/2895781261 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb36effd700 1 --2- 172.21.15.31:0/2895781261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb37006b130 0x7fb3701023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb36e7fc700 1 --2- 172.21.15.31:0/2895781261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb37006bad0 0x7fb370102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb36f7fe700 1 -- 172.21.15.31:0/2895781261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb37006bad0 msgr2=0x7fb370102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb36f7fe700 1 --2- 172.21.15.31:0/2895781261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb37006bad0 0x7fb370102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb36f7fe700 1 -- 172.21.15.31:0/2895781261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb37006b130 msgr2=0x7fb3701023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb36f7fe700 1 --2- 172.21.15.31:0/2895781261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb37006b130 0x7fb3701023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb36f7fe700 1 -- 172.21.15.31:0/2895781261 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3701a7fd0 con 0x7fb3701094a0 2024-04-26T17:55:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb36f7fe700 1 --2- 172.21.15.31:0/2895781261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3701094a0 0x7fb370102e70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb368007ee0 tx=0x7fb36800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb37484e700 1 -- 172.21.15.31:0/2895781261 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36800dbd0 con 0x7fb3701094a0 2024-04-26T17:55:02.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb37484e700 1 -- 172.21.15.31:0/2895781261 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb36800dd70 con 0x7fb3701094a0 2024-04-26T17:55:02.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb375850700 1 -- 172.21.15.31:0/2895781261 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3701a82c0 con 0x7fb3701094a0 2024-04-26T17:55:02.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb37484e700 1 -- 172.21.15.31:0/2895781261 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb368005990 con 0x7fb3701094a0 2024-04-26T17:55:02.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.603+0000 7fb375850700 1 -- 172.21.15.31:0/2895781261 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3701a8800 con 0x7fb3701094a0 2024-04-26T17:55:02.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.604+0000 7fb375850700 1 -- 172.21.15.31:0/2895781261 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb37006a110 con 0x7fb3701094a0 2024-04-26T17:55:02.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.605+0000 7fb37484e700 1 -- 172.21.15.31:0/2895781261 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb36801a020 con 0x7fb3701094a0 2024-04-26T17:55:02.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.606+0000 7fb37484e700 1 --2- 172.21.15.31:0/2895781261 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb3580a3c40 0x7fb3580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:02.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.606+0000 7fb36effd700 1 --2- 172.21.15.31:0/2895781261 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb3580a3c40 0x7fb3580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:02.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.606+0000 7fb37484e700 1 -- 172.21.15.31:0/2895781261 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fb3680cfc10 con 0x7fb3701094a0 2024-04-26T17:55:02.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.607+0000 7fb36effd700 1 --2- 172.21.15.31:0/2895781261 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb3580a3c40 0x7fb3580a6100 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb360005fd0 tx=0x7fb36000af20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:02.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.611+0000 7fb37484e700 1 -- 172.21.15.31:0/2895781261 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb368010070 con 0x7fb3701094a0 2024-04-26T17:55:02.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.824+0000 7fb375850700 1 -- 172.21.15.31:0/2895781261 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fb37002e490 con 0x7fb3701094a0 2024-04-26T17:55:02.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.824+0000 7fb37484e700 1 -- 172.21.15.31:0/2895781261 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fb36809c620 con 0x7fb3701094a0 2024-04-26T17:55:02.825 INFO:teuthology.orchestra.run.smithi031.stdout:309237645356 2024-04-26T17:55:02.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.827+0000 7fb375850700 1 -- 172.21.15.31:0/2895781261 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb3580a3c40 msgr2=0x7fb3580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:02.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.827+0000 7fb375850700 1 --2- 172.21.15.31:0/2895781261 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb3580a3c40 0x7fb3580a6100 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb360005fd0 tx=0x7fb36000af20 comp rx=0 tx=0).stop 2024-04-26T17:55:02.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.827+0000 7fb375850700 1 -- 172.21.15.31:0/2895781261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3701094a0 msgr2=0x7fb370102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:02.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.827+0000 7fb375850700 1 --2- 172.21.15.31:0/2895781261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3701094a0 0x7fb370102e70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb368007ee0 tx=0x7fb36800d3b0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.827+0000 7fb375850700 1 -- 172.21.15.31:0/2895781261 shutdown_connections 2024-04-26T17:55:02.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.827+0000 7fb375850700 1 --2- 172.21.15.31:0/2895781261 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb3580a3c40 0x7fb3580a6100 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.827+0000 7fb375850700 1 --2- 172.21.15.31:0/2895781261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb37006b130 0x7fb3701023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.827+0000 7fb375850700 1 --2- 172.21.15.31:0/2895781261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3701094a0 0x7fb370102e70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.827+0000 7fb375850700 1 --2- 172.21.15.31:0/2895781261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb37006bad0 0x7fb370102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.827+0000 7fb375850700 1 -- 172.21.15.31:0/2895781261 >> 172.21.15.31:0/2895781261 conn(0x7fb3700792d0 msgr2=0x7fb370100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:02.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.827+0000 7fb375850700 1 -- 172.21.15.31:0/2895781261 shutdown_connections 2024-04-26T17:55:02.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.827+0000 7fb375850700 1 -- 172.21.15.31:0/2895781261 wait complete. 2024-04-26T17:55:02.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.936+0000 7f25b70fa700 1 -- 172.21.15.31:0/3285409516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b0101e50 msgr2=0x7f25b01022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:02.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.936+0000 7f25b70fa700 1 --2- 172.21.15.31:0/3285409516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b0101e50 0x7f25b01022d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f259c009990 tx=0x7f259c009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.936+0000 7f25b70fa700 1 -- 172.21.15.31:0/3285409516 shutdown_connections 2024-04-26T17:55:02.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.936+0000 7f25b70fa700 1 --2- 172.21.15.31:0/3285409516 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25b0109600 0x7f25b01099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.936+0000 7f25b70fa700 1 --2- 172.21.15.31:0/3285409516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25b0102810 0x7f25b0110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.936+0000 7f25b70fa700 1 --2- 172.21.15.31:0/3285409516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b0101e50 0x7f25b01022d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.936+0000 7f25b70fa700 1 -- 172.21.15.31:0/3285409516 >> 172.21.15.31:0/3285409516 conn(0x7f25b0077590 msgr2=0x7f25b00779a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.936+0000 7f25b70fa700 1 -- 172.21.15.31:0/3285409516 shutdown_connections 2024-04-26T17:55:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.936+0000 7f25b70fa700 1 -- 172.21.15.31:0/3285409516 wait complete. 2024-04-26T17:55:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25b70fa700 1 Processor -- start 2024-04-26T17:55:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25b70fa700 1 -- start start 2024-04-26T17:55:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25b70fa700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25b0101e50 0x7f25b019bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25b70fa700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b0102810 0x7f25b019c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25b70fa700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25b0109600 0x7f25b01a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25b70fa700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f25b01135d0 con 0x7f25b0102810 2024-04-26T17:55:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25b70fa700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f25b0113450 con 0x7f25b0101e50 2024-04-26T17:55:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25b70fa700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f25b0113750 con 0x7f25b0109600 2024-04-26T17:55:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25affff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b0102810 0x7f25b019c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25affff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b0102810 0x7f25b019c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52732/0 (socket says 172.21.15.31:52732) 2024-04-26T17:55:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25affff700 1 -- 172.21.15.31:0/3844259913 learned_addr learned my addr 172.21.15.31:0/3844259913 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25b4e96700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25b0101e50 0x7f25b019bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25b5697700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25b0109600 0x7f25b01a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25affff700 1 -- 172.21.15.31:0/3844259913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25b0109600 msgr2=0x7f25b01a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25affff700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25b0109600 0x7f25b01a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25affff700 1 -- 172.21.15.31:0/3844259913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25b0101e50 msgr2=0x7f25b019bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25affff700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25b0101e50 0x7f25b019bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25affff700 1 -- 172.21.15.31:0/3844259913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b01a0d60 con 0x7f25b0102810 2024-04-26T17:55:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25b5697700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25b0109600 0x7f25b01a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:55:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.937+0000 7f25affff700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b0102810 0x7f25b019c2e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f259c000c00 tx=0x7f259c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.938+0000 7f25adffb700 1 -- 172.21.15.31:0/3844259913 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c027070 con 0x7f25b0102810 2024-04-26T17:55:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.938+0000 7f25adffb700 1 -- 172.21.15.31:0/3844259913 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f259c0040e0 con 0x7f25b0102810 2024-04-26T17:55:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.938+0000 7f25b70fa700 1 -- 172.21.15.31:0/3844259913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b01a0ff0 con 0x7f25b0102810 2024-04-26T17:55:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.938+0000 7f25adffb700 1 -- 172.21.15.31:0/3844259913 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c00f460 con 0x7f25b0102810 2024-04-26T17:55:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.938+0000 7f25b70fa700 1 -- 172.21.15.31:0/3844259913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b01a88d0 con 0x7f25b0102810 2024-04-26T17:55:02.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.939+0000 7f25b70fa700 1 -- 172.21.15.31:0/3844259913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25b006a110 con 0x7f25b0102810 2024-04-26T17:55:02.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.940+0000 7f25adffb700 1 -- 172.21.15.31:0/3844259913 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f259c02a070 con 0x7f25b0102810 2024-04-26T17:55:02.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.940+0000 7f25adffb700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f25a00a3c40 0x7f25a00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:02.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.941+0000 7f25b4e96700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f25a00a3c40 0x7f25a00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:02.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.941+0000 7f25adffb700 1 -- 172.21.15.31:0/3844259913 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f259c0d1420 con 0x7f25b0102810 2024-04-26T17:55:02.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.941+0000 7f25b4e96700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f25a00a3c40 0x7f25a00a6100 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f25a4005950 tx=0x7f25a400a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:02.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:02.945+0000 7f25adffb700 1 -- 172.21.15.31:0/3844259913 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f259c05eff0 con 0x7f25b0102810 2024-04-26T17:55:03.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:03.158+0000 7f25b70fa700 1 -- 172.21.15.31:0/3844259913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f25b002e490 con 0x7f25b0102810 2024-04-26T17:55:03.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:03.158+0000 7f25adffb700 1 -- 172.21.15.31:0/3844259913 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f259c013070 con 0x7f25b0102810 2024-04-26T17:55:03.159 INFO:teuthology.orchestra.run.smithi031.stdout:253403070518 2024-04-26T17:55:03.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:03.160+0000 7f25b70fa700 1 -- 172.21.15.31:0/3844259913 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f25a00a3c40 msgr2=0x7f25a00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:03.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:03.160+0000 7f25b70fa700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f25a00a3c40 0x7f25a00a6100 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f25a4005950 tx=0x7f25a400a300 comp rx=0 tx=0).stop 2024-04-26T17:55:03.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:03.160+0000 7f25b70fa700 1 -- 172.21.15.31:0/3844259913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b0102810 msgr2=0x7f25b019c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:03.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:03.160+0000 7f25b70fa700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b0102810 0x7f25b019c2e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f259c000c00 tx=0x7f259c0047b0 comp rx=0 tx=0).stop 2024-04-26T17:55:03.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:03.161+0000 7f25b70fa700 1 -- 172.21.15.31:0/3844259913 shutdown_connections 2024-04-26T17:55:03.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:03.161+0000 7f25b70fa700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f25a00a3c40 0x7f25a00a6100 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:03.161+0000 7f25b70fa700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25b0101e50 0x7f25b019bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:03.161+0000 7f25b70fa700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25b0109600 0x7f25b01a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:03.161+0000 7f25b70fa700 1 --2- 172.21.15.31:0/3844259913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b0102810 0x7f25b019c2e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:03.161+0000 7f25b70fa700 1 -- 172.21.15.31:0/3844259913 >> 172.21.15.31:0/3844259913 conn(0x7f25b0077590 msgr2=0x7f25b010eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:03.161+0000 7f25b70fa700 1 -- 172.21.15.31:0/3844259913 shutdown_connections 2024-04-26T17:55:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:03.161+0000 7f25b70fa700 1 -- 172.21.15.31:0/3844259913 wait complete. 2024-04-26T17:55:03.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:03 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2895781261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T17:55:03.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:03 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2895781261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T17:55:03.567 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:03.567 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:03.605 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645353 got 309237645356 for osd.10 2024-04-26T17:55:03.605 DEBUG:teuthology.parallel:result is None 2024-04-26T17:55:03.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:03 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2895781261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T17:55:04.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:04 smithi186 ceph-mon[32797]: pgmap v357: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T17:55:04.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:04 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3844259913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T17:55:04.542 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:04 smithi031 ceph-mon[29164]: pgmap v357: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T17:55:04.543 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:04 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3844259913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T17:55:04.570 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411408 got 77309411413 for osd.2 2024-04-26T17:55:04.570 DEBUG:teuthology.parallel:result is None 2024-04-26T17:55:04.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:04 smithi097 ceph-mon[32419]: pgmap v357: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T17:55:04.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:04 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3844259913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T17:55:05.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.137+0000 7fe1178f2700 1 -- 172.21.15.31:0/1117475948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1101034a0 msgr2=0x7fe110103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:05.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.137+0000 7fe1178f2700 1 --2- 172.21.15.31:0/1117475948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1101034a0 0x7fe110103920 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe100009990 tx=0x7fe100009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:05.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.137+0000 7fe1178f2700 1 -- 172.21.15.31:0/1117475948 shutdown_connections 2024-04-26T17:55:05.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.137+0000 7fe1178f2700 1 --2- 172.21.15.31:0/1117475948 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1101094e0 0x7fe1101098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:05.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.137+0000 7fe1178f2700 1 --2- 172.21.15.31:0/1117475948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe110103e60 0x7fe1101102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:05.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.137+0000 7fe1178f2700 1 --2- 172.21.15.31:0/1117475948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1101034a0 0x7fe110103920 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:05.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.137+0000 7fe1178f2700 1 -- 172.21.15.31:0/1117475948 >> 172.21.15.31:0/1117475948 conn(0x7fe11007a590 msgr2=0x7fe11007a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:05.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.138+0000 7fe1178f2700 1 -- 172.21.15.31:0/1117475948 shutdown_connections 2024-04-26T17:55:05.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.138+0000 7fe1178f2700 1 -- 172.21.15.31:0/1117475948 wait complete. 2024-04-26T17:55:05.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.138+0000 7fe1178f2700 1 Processor -- start 2024-04-26T17:55:05.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.138+0000 7fe1178f2700 1 -- start start 2024-04-26T17:55:05.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe1178f2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1101034a0 0x7fe110199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:05.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe1178f2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe110103e60 0x7fe11019a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:05.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe1178f2700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1101094e0 0x7fe11019e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:05.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe1178f2700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101135e0 con 0x7fe110103e60 2024-04-26T17:55:05.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe1178f2700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe110113460 con 0x7fe1101094e0 2024-04-26T17:55:05.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe1178f2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe110113760 con 0x7fe1101034a0 2024-04-26T17:55:05.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe114e8d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe110103e60 0x7fe11019a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:05.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe114e8d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe110103e60 0x7fe11019a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53086/0 (socket says 172.21.15.31:53086) 2024-04-26T17:55:05.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe114e8d700 1 -- 172.21.15.31:0/858264109 learned_addr learned my addr 172.21.15.31:0/858264109 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:05.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe115e8f700 1 --2- 172.21.15.31:0/858264109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1101094e0 0x7fe11019e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:05.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe114e8d700 1 -- 172.21.15.31:0/858264109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1101034a0 msgr2=0x7fe110199c10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T17:55:05.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe114e8d700 1 --2- 172.21.15.31:0/858264109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1101034a0 0x7fe110199c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:05.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe114e8d700 1 -- 172.21.15.31:0/858264109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1101094e0 msgr2=0x7fe11019e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:05.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe114e8d700 1 --2- 172.21.15.31:0/858264109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1101094e0 0x7fe11019e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:05.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe114e8d700 1 -- 172.21.15.31:0/858264109 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11019ec10 con 0x7fe110103e60 2024-04-26T17:55:05.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.139+0000 7fe114e8d700 1 --2- 172.21.15.31:0/858264109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe110103e60 0x7fe11019a150 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe1000057c0 tx=0x7fe1000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:05.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.140+0000 7fe1067fc700 1 -- 172.21.15.31:0/858264109 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100027070 con 0x7fe110103e60 2024-04-26T17:55:05.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.140+0000 7fe1067fc700 1 -- 172.21.15.31:0/858264109 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe100003f20 con 0x7fe110103e60 2024-04-26T17:55:05.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.140+0000 7fe1178f2700 1 -- 172.21.15.31:0/858264109 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11019eea0 con 0x7fe110103e60 2024-04-26T17:55:05.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.140+0000 7fe1067fc700 1 -- 172.21.15.31:0/858264109 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100022c30 con 0x7fe110103e60 2024-04-26T17:55:05.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.140+0000 7fe1178f2700 1 -- 172.21.15.31:0/858264109 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1101a67c0 con 0x7fe110103e60 2024-04-26T17:55:05.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.141+0000 7fe1178f2700 1 -- 172.21.15.31:0/858264109 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe11006a110 con 0x7fe110103e60 2024-04-26T17:55:05.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.142+0000 7fe1067fc700 1 -- 172.21.15.31:0/858264109 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe10002a070 con 0x7fe110103e60 2024-04-26T17:55:05.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.142+0000 7fe1067fc700 1 --2- 172.21.15.31:0/858264109 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe0fc0a3c40 0x7fe0fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:05.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.142+0000 7fe11568e700 1 --2- 172.21.15.31:0/858264109 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe0fc0a3c40 0x7fe0fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:05.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.142+0000 7fe1067fc700 1 -- 172.21.15.31:0/858264109 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fe1000d1320 con 0x7fe110103e60 2024-04-26T17:55:05.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.143+0000 7fe11568e700 1 --2- 172.21.15.31:0/858264109 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe0fc0a3c40 0x7fe0fc0a6100 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00a8f0 tx=0x7fe10c005b90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:05.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.147+0000 7fe1067fc700 1 -- 172.21.15.31:0/858264109 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe10005eff0 con 0x7fe110103e60 2024-04-26T17:55:05.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.364+0000 7fe1178f2700 1 -- 172.21.15.31:0/858264109 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fe11002e490 con 0x7fe110103e60 2024-04-26T17:55:05.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.364+0000 7fe1067fc700 1 -- 172.21.15.31:0/858264109 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe100013070 con 0x7fe110103e60 2024-04-26T17:55:05.365 INFO:teuthology.orchestra.run.smithi031.stdout:107374182479 2024-04-26T17:55:05.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.367+0000 7fe1178f2700 1 -- 172.21.15.31:0/858264109 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe0fc0a3c40 msgr2=0x7fe0fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:05.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.367+0000 7fe1178f2700 1 --2- 172.21.15.31:0/858264109 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe0fc0a3c40 0x7fe0fc0a6100 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00a8f0 tx=0x7fe10c005b90 comp rx=0 tx=0).stop 2024-04-26T17:55:05.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.367+0000 7fe1178f2700 1 -- 172.21.15.31:0/858264109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe110103e60 msgr2=0x7fe11019a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:05.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.367+0000 7fe1178f2700 1 --2- 172.21.15.31:0/858264109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe110103e60 0x7fe11019a150 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe1000057c0 tx=0x7fe1000057f0 comp rx=0 tx=0).stop 2024-04-26T17:55:05.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.367+0000 7fe1178f2700 1 -- 172.21.15.31:0/858264109 shutdown_connections 2024-04-26T17:55:05.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.367+0000 7fe1178f2700 1 --2- 172.21.15.31:0/858264109 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe0fc0a3c40 0x7fe0fc0a6100 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:05.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.367+0000 7fe1178f2700 1 --2- 172.21.15.31:0/858264109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1101094e0 0x7fe11019e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:05.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.367+0000 7fe1178f2700 1 --2- 172.21.15.31:0/858264109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1101034a0 0x7fe110199c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:05.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.367+0000 7fe1178f2700 1 --2- 172.21.15.31:0/858264109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe110103e60 0x7fe11019a150 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:05.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.367+0000 7fe1178f2700 1 -- 172.21.15.31:0/858264109 >> 172.21.15.31:0/858264109 conn(0x7fe11007a590 msgr2=0x7fe1101021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:05.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:05 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:55:05.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.367+0000 7fe1178f2700 1 -- 172.21.15.31:0/858264109 shutdown_connections 2024-04-26T17:55:05.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:05.367+0000 7fe1178f2700 1 -- 172.21.15.31:0/858264109 wait complete. 2024-04-26T17:55:05.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:05 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:55:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:05 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:55:06.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.019+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/1766681552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2e41094e0 msgr2=0x7fe2e41098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:06.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.019+0000 7fe2ea1ee700 1 --2- 172.21.15.31:0/1766681552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2e41094e0 0x7fe2e41098c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400b0a0 tx=0x7fe2d400b3b0 comp rx=0 tx=0).stop 2024-04-26T17:55:06.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.019+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/1766681552 shutdown_connections 2024-04-26T17:55:06.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.019+0000 7fe2ea1ee700 1 --2- 172.21.15.31:0/1766681552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2e4103e60 0x7fe2e41102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:06.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.019+0000 7fe2ea1ee700 1 --2- 172.21.15.31:0/1766681552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2e41094e0 0x7fe2e41098c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:06.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.019+0000 7fe2ea1ee700 1 --2- 172.21.15.31:0/1766681552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2e41034a0 0x7fe2e4103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:06.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.019+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/1766681552 >> 172.21.15.31:0/1766681552 conn(0x7fe2e407a590 msgr2=0x7fe2e407a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:06.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.019+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/1766681552 shutdown_connections 2024-04-26T17:55:06.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.019+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/1766681552 wait complete. 2024-04-26T17:55:06.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.019+0000 7fe2ea1ee700 1 Processor -- start 2024-04-26T17:55:06.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2ea1ee700 1 -- start start 2024-04-26T17:55:06.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2ea1ee700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2e41034a0 0x7fe2e419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:06.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2ea1ee700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2e4103e60 0x7fe2e419fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:06.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2ea1ee700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2e41094e0 0x7fe2e4199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:06.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2ea1ee700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e4113610 con 0x7fe2e4103e60 2024-04-26T17:55:06.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2ea1ee700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e4113490 con 0x7fe2e41094e0 2024-04-26T17:55:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2ea1ee700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e4113790 con 0x7fe2e41034a0 2024-04-26T17:55:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2e2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2e4103e60 0x7fe2e419fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2e2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2e4103e60 0x7fe2e419fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53094/0 (socket says 172.21.15.31:53094) 2024-04-26T17:55:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2e2ffd700 1 -- 172.21.15.31:0/7135682 learned_addr learned my addr 172.21.15.31:0/7135682 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2e37fe700 1 --2- 172.21.15.31:0/7135682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2e41034a0 0x7fe2e419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2e3fff700 1 --2- 172.21.15.31:0/7135682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2e41094e0 0x7fe2e4199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2e2ffd700 1 -- 172.21.15.31:0/7135682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2e41034a0 msgr2=0x7fe2e419f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2e2ffd700 1 --2- 172.21.15.31:0/7135682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2e41034a0 0x7fe2e419f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2e2ffd700 1 -- 172.21.15.31:0/7135682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2e41094e0 msgr2=0x7fe2e4199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2e2ffd700 1 --2- 172.21.15.31:0/7135682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2e41094e0 0x7fe2e4199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.020+0000 7fe2e2ffd700 1 -- 172.21.15.31:0/7135682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e419a090 con 0x7fe2e4103e60 2024-04-26T17:55:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.021+0000 7fe2e2ffd700 1 --2- 172.21.15.31:0/7135682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2e4103e60 0x7fe2e419fcb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00d950 tx=0x7fe2cc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.021+0000 7fe2e0ff9700 1 -- 172.21.15.31:0/7135682 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc016070 con 0x7fe2e4103e60 2024-04-26T17:55:06.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.021+0000 7fe2e0ff9700 1 -- 172.21.15.31:0/7135682 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2cc00fd40 con 0x7fe2e4103e60 2024-04-26T17:55:06.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.021+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/7135682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2e419a380 con 0x7fe2e4103e60 2024-04-26T17:55:06.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.021+0000 7fe2e0ff9700 1 -- 172.21.15.31:0/7135682 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc00e3b0 con 0x7fe2e4103e60 2024-04-26T17:55:06.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.021+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/7135682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2e41a6640 con 0x7fe2e4103e60 2024-04-26T17:55:06.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.022+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/7135682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2e410b950 con 0x7fe2e4103e60 2024-04-26T17:55:06.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.023+0000 7fe2e0ff9700 1 -- 172.21.15.31:0/7135682 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe2cc019070 con 0x7fe2e4103e60 2024-04-26T17:55:06.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.024+0000 7fe2e0ff9700 1 --2- 172.21.15.31:0/7135682 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe2d00a3c40 0x7fe2d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:06.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.024+0000 7fe2e37fe700 1 --2- 172.21.15.31:0/7135682 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe2d00a3c40 0x7fe2d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:06.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.024+0000 7fe2e0ff9700 1 -- 172.21.15.31:0/7135682 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fe2cc0d0200 con 0x7fe2e4103e60 2024-04-26T17:55:06.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.024+0000 7fe2e37fe700 1 --2- 172.21.15.31:0/7135682 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe2d00a3c40 0x7fe2d00a6100 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009e60 tx=0x7fe2d400b7b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:06.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.028+0000 7fe2e0ff9700 1 -- 172.21.15.31:0/7135682 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe2cc09cc10 con 0x7fe2e4103e60 2024-04-26T17:55:06.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.244+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/7135682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fe2e419ae60 con 0x7fe2e4103e60 2024-04-26T17:55:06.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:06 smithi031 ceph-mon[29164]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:06.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:06 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/858264109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T17:55:06.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:06 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:06.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:06 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:06.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:06 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:06.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:06 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:06.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.244+0000 7fe2e0ff9700 1 -- 172.21.15.31:0/7135682 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe2cc09c510 con 0x7fe2e4103e60 2024-04-26T17:55:06.245 INFO:teuthology.orchestra.run.smithi031.stdout:287762808881 2024-04-26T17:55:06.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.247+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/7135682 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe2d00a3c40 msgr2=0x7fe2d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:06.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.247+0000 7fe2ea1ee700 1 --2- 172.21.15.31:0/7135682 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe2d00a3c40 0x7fe2d00a6100 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009e60 tx=0x7fe2d400b7b0 comp rx=0 tx=0).stop 2024-04-26T17:55:06.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.247+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/7135682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2e4103e60 msgr2=0x7fe2e419fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:06.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.247+0000 7fe2ea1ee700 1 --2- 172.21.15.31:0/7135682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2e4103e60 0x7fe2e419fcb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00d950 tx=0x7fe2cc00dc60 comp rx=0 tx=0).stop 2024-04-26T17:55:06.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.247+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/7135682 shutdown_connections 2024-04-26T17:55:06.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.247+0000 7fe2ea1ee700 1 --2- 172.21.15.31:0/7135682 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fe2d00a3c40 0x7fe2d00a6100 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:06.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.247+0000 7fe2ea1ee700 1 --2- 172.21.15.31:0/7135682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2e41094e0 0x7fe2e4199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:06.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.247+0000 7fe2ea1ee700 1 --2- 172.21.15.31:0/7135682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2e41034a0 0x7fe2e419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:06.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.247+0000 7fe2ea1ee700 1 --2- 172.21.15.31:0/7135682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2e4103e60 0x7fe2e419fcb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:06.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.247+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/7135682 >> 172.21.15.31:0/7135682 conn(0x7fe2e407a590 msgr2=0x7fe2e41022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:06.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.247+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/7135682 shutdown_connections 2024-04-26T17:55:06.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:06.247+0000 7fe2ea1ee700 1 -- 172.21.15.31:0/7135682 wait complete. 2024-04-26T17:55:06.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:06 smithi186 ceph-mon[32797]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:06.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:06 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/858264109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T17:55:06.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:06 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:06.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:06 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:06.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:06 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:06.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:06 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:06.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:06 smithi097 ceph-mon[32419]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:06.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:06 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/858264109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T17:55:06.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:06 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:06.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:06 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:06.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:06 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:06.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:06 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:07.326 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182474 got 107374182479 for osd.3 2024-04-26T17:55:07.327 DEBUG:teuthology.parallel:result is None 2024-04-26T17:55:07.441 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:07 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/7135682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T17:55:07.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:07 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/7135682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T17:55:07.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:07 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/7135682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T17:55:07.784 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808877 got 287762808881 for osd.9 2024-04-26T17:55:07.785 DEBUG:teuthology.parallel:result is None 2024-04-26T17:55:08.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:08 smithi031 ceph-mon[29164]: pgmap v359: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:08.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:08 smithi186 ceph-mon[32797]: pgmap v359: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:08.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:08 smithi097 ceph-mon[32419]: pgmap v359: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:08.709 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070512 got 253403070518 for osd.8 2024-04-26T17:55:08.709 DEBUG:teuthology.parallel:result is None 2024-04-26T17:55:09.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.153+0000 7fb4dcb88700 1 -- 172.21.15.31:0/130605088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4d8108390 msgr2=0x7fb4d8108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:09.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.153+0000 7fb4dcb88700 1 --2- 172.21.15.31:0/130605088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4d8108390 0x7fb4d8108770 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8009990 tx=0x7fb4c8009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:09.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.153+0000 7fb4dcb88700 1 -- 172.21.15.31:0/130605088 shutdown_connections 2024-04-26T17:55:09.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.153+0000 7fb4dcb88700 1 --2- 172.21.15.31:0/130605088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4d8102ce0 0x7fb4d810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:09.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.153+0000 7fb4dcb88700 1 --2- 172.21.15.31:0/130605088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4d8108390 0x7fb4d8108770 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:09.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.153+0000 7fb4dcb88700 1 --2- 172.21.15.31:0/130605088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4d8102320 0x7fb4d81027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:09.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.153+0000 7fb4dcb88700 1 -- 172.21.15.31:0/130605088 >> 172.21.15.31:0/130605088 conn(0x7fb4d80792d0 msgr2=0x7fb4d80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:09.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.153+0000 7fb4dcb88700 1 -- 172.21.15.31:0/130605088 shutdown_connections 2024-04-26T17:55:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.153+0000 7fb4dcb88700 1 -- 172.21.15.31:0/130605088 wait complete. 2024-04-26T17:55:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.154+0000 7fb4dcb88700 1 Processor -- start 2024-04-26T17:55:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.154+0000 7fb4dcb88700 1 -- start start 2024-04-26T17:55:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.154+0000 7fb4dcb88700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4d8102320 0x7fb4d8104000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.154+0000 7fb4dcb88700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4d8102ce0 0x7fb4d8104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.154+0000 7fb4dcb88700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4d8108390 0x7fb4d8104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.154+0000 7fb4dcb88700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d81124b0 con 0x7fb4d8108390 2024-04-26T17:55:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.154+0000 7fb4dcb88700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d8112330 con 0x7fb4d8102320 2024-04-26T17:55:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.154+0000 7fb4dcb88700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d8112630 con 0x7fb4d8102ce0 2024-04-26T17:55:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4d6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4d8108390 0x7fb4d8104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4d6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4d8108390 0x7fb4d8104a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53116/0 (socket says 172.21.15.31:53116) 2024-04-26T17:55:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4d6d9d700 1 -- 172.21.15.31:0/2190596877 learned_addr learned my addr 172.21.15.31:0/2190596877 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:09.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4d5d9b700 1 --2- 172.21.15.31:0/2190596877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4d8102ce0 0x7fb4d8104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:09.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4d659c700 1 --2- 172.21.15.31:0/2190596877 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4d8102320 0x7fb4d8104000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:09.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4d6d9d700 1 -- 172.21.15.31:0/2190596877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4d8102ce0 msgr2=0x7fb4d8104540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:09.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4d6d9d700 1 --2- 172.21.15.31:0/2190596877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4d8102ce0 0x7fb4d8104540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:09.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4d6d9d700 1 -- 172.21.15.31:0/2190596877 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4d8102320 msgr2=0x7fb4d8104000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:09.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4d6d9d700 1 --2- 172.21.15.31:0/2190596877 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4d8102320 0x7fb4d8104000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:09.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4d6d9d700 1 -- 172.21.15.31:0/2190596877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d81a7f60 con 0x7fb4d8108390 2024-04-26T17:55:09.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4d6d9d700 1 --2- 172.21.15.31:0/2190596877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4d8108390 0x7fb4d8104a80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0007d70 tx=0x7fb4d000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:09.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4c77fe700 1 -- 172.21.15.31:0/2190596877 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0016020 con 0x7fb4d8108390 2024-04-26T17:55:09.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4dcb88700 1 -- 172.21.15.31:0/2190596877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4d81a8250 con 0x7fb4d8108390 2024-04-26T17:55:09.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4dcb88700 1 -- 172.21.15.31:0/2190596877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4d81a8790 con 0x7fb4d8108390 2024-04-26T17:55:09.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4c77fe700 1 -- 172.21.15.31:0/2190596877 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4d000ad40 con 0x7fb4d8108390 2024-04-26T17:55:09.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.155+0000 7fb4c77fe700 1 -- 172.21.15.31:0/2190596877 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d00115e0 con 0x7fb4d8108390 2024-04-26T17:55:09.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.156+0000 7fb4dcb88700 1 -- 172.21.15.31:0/2190596877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4d806a110 con 0x7fb4d8108390 2024-04-26T17:55:09.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.158+0000 7fb4c77fe700 1 -- 172.21.15.31:0/2190596877 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb4d0019020 con 0x7fb4d8108390 2024-04-26T17:55:09.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.158+0000 7fb4c77fe700 1 --2- 172.21.15.31:0/2190596877 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb4c00a3cc0 0x7fb4c00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:09.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.158+0000 7fb4d659c700 1 --2- 172.21.15.31:0/2190596877 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb4c00a3cc0 0x7fb4c00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:09.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.158+0000 7fb4c77fe700 1 -- 172.21.15.31:0/2190596877 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fb4d00d0810 con 0x7fb4d8108390 2024-04-26T17:55:09.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.159+0000 7fb4d659c700 1 --2- 172.21.15.31:0/2190596877 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb4c00a3cc0 0x7fb4c00a6180 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800b5c0 tx=0x7fb4c8005a90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:09.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.167+0000 7fb4c77fe700 1 -- 172.21.15.31:0/2190596877 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb4d0020020 con 0x7fb4d8108390 2024-04-26T17:55:09.178 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.397+0000 7fb4dcb88700 1 -- 172.21.15.31:0/2190596877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fb4d81a8aa0 con 0x7fb4d8108390 2024-04-26T17:55:09.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.397+0000 7fb4c77fe700 1 -- 172.21.15.31:0/2190596877 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb4d009d220 con 0x7fb4d8108390 2024-04-26T17:55:09.398 INFO:teuthology.orchestra.run.smithi031.stdout:34359738464 2024-04-26T17:55:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.399+0000 7fb4dcb88700 1 -- 172.21.15.31:0/2190596877 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb4c00a3cc0 msgr2=0x7fb4c00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.399+0000 7fb4dcb88700 1 --2- 172.21.15.31:0/2190596877 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb4c00a3cc0 0x7fb4c00a6180 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800b5c0 tx=0x7fb4c8005a90 comp rx=0 tx=0).stop 2024-04-26T17:55:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.399+0000 7fb4dcb88700 1 -- 172.21.15.31:0/2190596877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4d8108390 msgr2=0x7fb4d8104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.399+0000 7fb4dcb88700 1 --2- 172.21.15.31:0/2190596877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4d8108390 0x7fb4d8104a80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0007d70 tx=0x7fb4d000a340 comp rx=0 tx=0).stop 2024-04-26T17:55:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.399+0000 7fb4dcb88700 1 -- 172.21.15.31:0/2190596877 shutdown_connections 2024-04-26T17:55:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.399+0000 7fb4dcb88700 1 --2- 172.21.15.31:0/2190596877 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb4c00a3cc0 0x7fb4c00a6180 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.399+0000 7fb4dcb88700 1 --2- 172.21.15.31:0/2190596877 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4d8102320 0x7fb4d8104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.399+0000 7fb4dcb88700 1 --2- 172.21.15.31:0/2190596877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4d8108390 0x7fb4d8104a80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.399+0000 7fb4dcb88700 1 --2- 172.21.15.31:0/2190596877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4d8102ce0 0x7fb4d8104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.399+0000 7fb4dcb88700 1 -- 172.21.15.31:0/2190596877 >> 172.21.15.31:0/2190596877 conn(0x7fb4d80792d0 msgr2=0x7fb4d81015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.399+0000 7fb4dcb88700 1 -- 172.21.15.31:0/2190596877 shutdown_connections 2024-04-26T17:55:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:09.399+0000 7fb4dcb88700 1 -- 172.21.15.31:0/2190596877 wait complete. 2024-04-26T17:55:10.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:10 smithi031 ceph-mon[29164]: pgmap v360: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:10.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:10 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2190596877' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T17:55:10.347 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738457 got 34359738464 for osd.0 2024-04-26T17:55:10.348 DEBUG:teuthology.parallel:result is None 2024-04-26T17:55:10.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:10 smithi186 ceph-mon[32797]: pgmap v360: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:10.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:10 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2190596877' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T17:55:10.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:10 smithi097 ceph-mon[32419]: pgmap v360: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:10.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:10 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2190596877' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T17:55:11.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:11 smithi186 ceph-mon[32797]: pgmap v361: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:11.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:11 smithi031 ceph-mon[29164]: pgmap v361: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:11.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:11 smithi097 ceph-mon[32419]: pgmap v361: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:12.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.393+0000 7f96ed464700 1 -- 172.21.15.31:0/945418464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96e806bad0 msgr2=0x7f96e81102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:12.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.393+0000 7f96ed464700 1 --2- 172.21.15.31:0/945418464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96e806bad0 0x7f96e81102e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f96e000b0a0 tx=0x7f96e000b3b0 comp rx=0 tx=0).stop 2024-04-26T17:55:12.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.393+0000 7f96ed464700 1 -- 172.21.15.31:0/945418464 shutdown_connections 2024-04-26T17:55:12.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.393+0000 7f96ed464700 1 --2- 172.21.15.31:0/945418464 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96e81094a0 0x7f96e8109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:12.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.393+0000 7f96ed464700 1 --2- 172.21.15.31:0/945418464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96e806bad0 0x7f96e81102e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:12.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.393+0000 7f96ed464700 1 --2- 172.21.15.31:0/945418464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96e806b130 0x7f96e806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:12.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.393+0000 7f96ed464700 1 -- 172.21.15.31:0/945418464 >> 172.21.15.31:0/945418464 conn(0x7f96e80792d0 msgr2=0x7f96e80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:12.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.393+0000 7f96ed464700 1 -- 172.21.15.31:0/945418464 shutdown_connections 2024-04-26T17:55:12.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.393+0000 7f96ed464700 1 -- 172.21.15.31:0/945418464 wait complete. 2024-04-26T17:55:12.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.394+0000 7f96ed464700 1 Processor -- start 2024-04-26T17:55:12.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.394+0000 7f96ed464700 1 -- start start 2024-04-26T17:55:12.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.394+0000 7f96ed464700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96e806b130 0x7f96e81023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:12.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.394+0000 7f96ed464700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96e806bad0 0x7f96e8102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:12.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.394+0000 7f96ed464700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96e81094a0 0x7f96e8102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:12.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.394+0000 7f96ed464700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f96e81135c0 con 0x7f96e806b130 2024-04-26T17:55:12.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.394+0000 7f96ed464700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f96e8113440 con 0x7f96e806bad0 2024-04-26T17:55:12.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.394+0000 7f96ed464700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f96e8113740 con 0x7f96e81094a0 2024-04-26T17:55:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.394+0000 7f96e6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96e806b130 0x7f96e81023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.394+0000 7f96e6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96e806b130 0x7f96e81023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53130/0 (socket says 172.21.15.31:53130) 2024-04-26T17:55:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.394+0000 7f96e6ffd700 1 -- 172.21.15.31:0/1167802039 learned_addr learned my addr 172.21.15.31:0/1167802039 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.394+0000 7f96e67fc700 1 --2- 172.21.15.31:0/1167802039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96e806bad0 0x7f96e8102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.394+0000 7f96e77fe700 1 --2- 172.21.15.31:0/1167802039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96e81094a0 0x7f96e8102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.395+0000 7f96e6ffd700 1 -- 172.21.15.31:0/1167802039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96e81094a0 msgr2=0x7f96e8102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.395+0000 7f96e6ffd700 1 --2- 172.21.15.31:0/1167802039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96e81094a0 0x7f96e8102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.395+0000 7f96e6ffd700 1 -- 172.21.15.31:0/1167802039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96e806bad0 msgr2=0x7f96e8102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.395+0000 7f96e6ffd700 1 --2- 172.21.15.31:0/1167802039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96e806bad0 0x7f96e8102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.395+0000 7f96e6ffd700 1 -- 172.21.15.31:0/1167802039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e81a7fa0 con 0x7f96e806b130 2024-04-26T17:55:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.395+0000 7f96e6ffd700 1 --2- 172.21.15.31:0/1167802039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96e806b130 0x7f96e81023f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f96d800d980 tx=0x7f96d800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.395+0000 7f96cffff700 1 -- 172.21.15.31:0/1167802039 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d8017070 con 0x7f96e806b130 2024-04-26T17:55:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.395+0000 7f96ed464700 1 -- 172.21.15.31:0/1167802039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e81a8290 con 0x7f96e806b130 2024-04-26T17:55:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.395+0000 7f96cffff700 1 -- 172.21.15.31:0/1167802039 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96d8005020 con 0x7f96e806b130 2024-04-26T17:55:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.395+0000 7f96cffff700 1 -- 172.21.15.31:0/1167802039 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d800fa10 con 0x7f96e806b130 2024-04-26T17:55:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.395+0000 7f96ed464700 1 -- 172.21.15.31:0/1167802039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e81a8780 con 0x7f96e806b130 2024-04-26T17:55:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.396+0000 7f96ed464700 1 -- 172.21.15.31:0/1167802039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96e806a110 con 0x7f96e806b130 2024-04-26T17:55:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.398+0000 7f96cffff700 1 -- 172.21.15.31:0/1167802039 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f96d8004a30 con 0x7f96e806b130 2024-04-26T17:55:12.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.398+0000 7f96cffff700 1 --2- 172.21.15.31:0/1167802039 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f96d00a3c40 0x7f96d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:12.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.399+0000 7f96e67fc700 1 --2- 172.21.15.31:0/1167802039 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f96d00a3c40 0x7f96d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:12.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.399+0000 7f96cffff700 1 -- 172.21.15.31:0/1167802039 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f96d80d0620 con 0x7f96e806b130 2024-04-26T17:55:12.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.399+0000 7f96e67fc700 1 --2- 172.21.15.31:0/1167802039 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f96d00a3c40 0x7f96d00a6100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f96e8079860 tx=0x7f96dc00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:12.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.402+0000 7f96cffff700 1 -- 172.21.15.31:0/1167802039 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f96d8020020 con 0x7f96e806b130 2024-04-26T17:55:12.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.621+0000 7f96ed464700 1 -- 172.21.15.31:0/1167802039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f96e802e490 con 0x7f96e806b130 2024-04-26T17:55:12.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.621+0000 7f96cffff700 1 -- 172.21.15.31:0/1167802039 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f96d809cfd0 con 0x7f96e806b130 2024-04-26T17:55:12.622 INFO:teuthology.orchestra.run.smithi031.stdout:55834574940 2024-04-26T17:55:12.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.623+0000 7f96ed464700 1 -- 172.21.15.31:0/1167802039 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f96d00a3c40 msgr2=0x7f96d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:12.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.623+0000 7f96ed464700 1 --2- 172.21.15.31:0/1167802039 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f96d00a3c40 0x7f96d00a6100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f96e8079860 tx=0x7f96dc00a380 comp rx=0 tx=0).stop 2024-04-26T17:55:12.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.623+0000 7f96ed464700 1 -- 172.21.15.31:0/1167802039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96e806b130 msgr2=0x7f96e81023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:12.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.623+0000 7f96ed464700 1 --2- 172.21.15.31:0/1167802039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96e806b130 0x7f96e81023f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f96d800d980 tx=0x7f96d800dc90 comp rx=0 tx=0).stop 2024-04-26T17:55:12.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.623+0000 7f96ed464700 1 -- 172.21.15.31:0/1167802039 shutdown_connections 2024-04-26T17:55:12.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.623+0000 7f96ed464700 1 --2- 172.21.15.31:0/1167802039 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f96d00a3c40 0x7f96d00a6100 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:12.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.623+0000 7f96ed464700 1 --2- 172.21.15.31:0/1167802039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96e81094a0 0x7f96e8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:12.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.623+0000 7f96ed464700 1 --2- 172.21.15.31:0/1167802039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96e806b130 0x7f96e81023f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:12.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.623+0000 7f96ed464700 1 --2- 172.21.15.31:0/1167802039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96e806bad0 0x7f96e8102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:12.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.623+0000 7f96ed464700 1 -- 172.21.15.31:0/1167802039 >> 172.21.15.31:0/1167802039 conn(0x7f96e80792d0 msgr2=0x7f96e8100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:12.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.623+0000 7f96ed464700 1 -- 172.21.15.31:0/1167802039 shutdown_connections 2024-04-26T17:55:12.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:12.623+0000 7f96ed464700 1 -- 172.21.15.31:0/1167802039 wait complete. 2024-04-26T17:55:13.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.031+0000 7fac90507700 1 -- 172.21.15.31:0/3161148699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac88102780 msgr2=0x7fac881101f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.031+0000 7fac90507700 1 --2- 172.21.15.31:0/3161148699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac88102780 0x7fac881101f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fac8000b0a0 tx=0x7fac8000b3b0 comp rx=0 tx=0).stop 2024-04-26T17:55:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.031+0000 7fac90507700 1 -- 172.21.15.31:0/3161148699 shutdown_connections 2024-04-26T17:55:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.031+0000 7fac90507700 1 --2- 172.21.15.31:0/3161148699 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac881084e0 0x7fac881088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.031+0000 7fac90507700 1 --2- 172.21.15.31:0/3161148699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac88102780 0x7fac881101f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.031+0000 7fac90507700 1 --2- 172.21.15.31:0/3161148699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac88101dc0 0x7fac88102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.031+0000 7fac90507700 1 -- 172.21.15.31:0/3161148699 >> 172.21.15.31:0/3161148699 conn(0x7fac88077640 msgr2=0x7fac88077a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.031+0000 7fac90507700 1 -- 172.21.15.31:0/3161148699 shutdown_connections 2024-04-26T17:55:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.031+0000 7fac90507700 1 -- 172.21.15.31:0/3161148699 wait complete. 2024-04-26T17:55:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.031+0000 7fac90507700 1 Processor -- start 2024-04-26T17:55:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.031+0000 7fac90507700 1 -- start start 2024-04-26T17:55:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac90507700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac88101dc0 0x7fac8819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:13.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac90507700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac88102780 0x7fac8819fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:13.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac90507700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac881084e0 0x7fac88199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:13.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac90507700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fac88113670 con 0x7fac88101dc0 2024-04-26T17:55:13.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac90507700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fac881134f0 con 0x7fac881084e0 2024-04-26T17:55:13.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac90507700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fac881137f0 con 0x7fac88102780 2024-04-26T17:55:13.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac8e2a3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac88101dc0 0x7fac8819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:13.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac8e2a3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac88101dc0 0x7fac8819f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53150/0 (socket says 172.21.15.31:53150) 2024-04-26T17:55:13.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac8e2a3700 1 -- 172.21.15.31:0/412067374 learned_addr learned my addr 172.21.15.31:0/412067374 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:13.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac8eaa4700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac881084e0 0x7fac88199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:13.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac8daa2700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac88102780 0x7fac8819fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:13.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac8e2a3700 1 -- 172.21.15.31:0/412067374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac88102780 msgr2=0x7fac8819fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:13.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac8e2a3700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac88102780 0x7fac8819fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:13.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac8e2a3700 1 -- 172.21.15.31:0/412067374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac881084e0 msgr2=0x7fac88199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:13.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac8e2a3700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac881084e0 0x7fac88199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:13.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac8e2a3700 1 -- 172.21.15.31:0/412067374 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac8819a170 con 0x7fac88101dc0 2024-04-26T17:55:13.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac8daa2700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac88102780 0x7fac8819fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:55:13.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac8e2a3700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac88101dc0 0x7fac8819f850 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fac8400d980 tx=0x7fac8400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:13.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.032+0000 7fac7b7fe700 1 -- 172.21.15.31:0/412067374 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac84017030 con 0x7fac88101dc0 2024-04-26T17:55:13.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.033+0000 7fac7b7fe700 1 -- 172.21.15.31:0/412067374 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac840044d0 con 0x7fac88101dc0 2024-04-26T17:55:13.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.033+0000 7fac90507700 1 -- 172.21.15.31:0/412067374 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac8819a460 con 0x7fac88101dc0 2024-04-26T17:55:13.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.033+0000 7fac7b7fe700 1 -- 172.21.15.31:0/412067374 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac840039e0 con 0x7fac88101dc0 2024-04-26T17:55:13.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.033+0000 7fac90507700 1 -- 172.21.15.31:0/412067374 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac881a6720 con 0x7fac88101dc0 2024-04-26T17:55:13.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.034+0000 7fac90507700 1 -- 172.21.15.31:0/412067374 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac8806a110 con 0x7fac88101dc0 2024-04-26T17:55:13.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.035+0000 7fac7b7fe700 1 -- 172.21.15.31:0/412067374 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fac8401a070 con 0x7fac88101dc0 2024-04-26T17:55:13.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.035+0000 7fac7b7fe700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac740a3c40 0x7fac740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:13.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.035+0000 7fac8daa2700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac740a3c40 0x7fac740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:13.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.035+0000 7fac7b7fe700 1 -- 172.21.15.31:0/412067374 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fac840cfcf0 con 0x7fac88101dc0 2024-04-26T17:55:13.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.036+0000 7fac8daa2700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac740a3c40 0x7fac740a6100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fac88077cd0 tx=0x7fac7c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:13.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.040+0000 7fac7b7fe700 1 -- 172.21.15.31:0/412067374 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac840d4070 con 0x7fac88101dc0 2024-04-26T17:55:13.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.252+0000 7fac90507700 1 -- 172.21.15.31:0/412067374 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fac8819af40 con 0x7fac88101dc0 2024-04-26T17:55:13.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.252+0000 7fac7b7fe700 1 -- 172.21.15.31:0/412067374 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fac8409c700 con 0x7fac88101dc0 2024-04-26T17:55:13.253 INFO:teuthology.orchestra.run.smithi031.stdout:343597383722 2024-04-26T17:55:13.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.254+0000 7fac90507700 1 -- 172.21.15.31:0/412067374 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac740a3c40 msgr2=0x7fac740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:13.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.254+0000 7fac90507700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac740a3c40 0x7fac740a6100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fac88077cd0 tx=0x7fac7c008040 comp rx=0 tx=0).stop 2024-04-26T17:55:13.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.254+0000 7fac90507700 1 -- 172.21.15.31:0/412067374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac88101dc0 msgr2=0x7fac8819f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:13.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.254+0000 7fac90507700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac88101dc0 0x7fac8819f850 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fac8400d980 tx=0x7fac8400dc90 comp rx=0 tx=0).stop 2024-04-26T17:55:13.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.254+0000 7fac90507700 1 -- 172.21.15.31:0/412067374 shutdown_connections 2024-04-26T17:55:13.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.254+0000 7fac90507700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fac740a3c40 0x7fac740a6100 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:13.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.254+0000 7fac90507700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac881084e0 0x7fac88199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:13.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.254+0000 7fac90507700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac88101dc0 0x7fac8819f850 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:13.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.254+0000 7fac90507700 1 --2- 172.21.15.31:0/412067374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac88102780 0x7fac8819fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:13.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.254+0000 7fac90507700 1 -- 172.21.15.31:0/412067374 >> 172.21.15.31:0/412067374 conn(0x7fac88077640 msgr2=0x7fac88100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:13.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.255+0000 7fac90507700 1 -- 172.21.15.31:0/412067374 shutdown_connections 2024-04-26T17:55:13.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:13.255+0000 7fac90507700 1 -- 172.21.15.31:0/412067374 wait complete. 2024-04-26T17:55:13.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:13 smithi031 ceph-mon[29164]: pgmap v362: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:13.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:13 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1167802039' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T17:55:13.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:13 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/412067374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T17:55:13.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:13 smithi186 ceph-mon[32797]: pgmap v362: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:13.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:13 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1167802039' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T17:55:13.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:13 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/412067374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T17:55:13.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:13 smithi097 ceph-mon[32419]: pgmap v362: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:13.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:13 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1167802039' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T17:55:13.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:13 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/412067374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T17:55:14.136 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574933 got 55834574940 for osd.1 2024-04-26T17:55:14.137 DEBUG:teuthology.parallel:result is None 2024-04-26T17:55:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.754+0000 7f8c81de5700 1 -- 172.21.15.31:0/293793290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c7c06c510 msgr2=0x7f8c7c1102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.754+0000 7f8c81de5700 1 --2- 172.21.15.31:0/293793290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c7c06c510 0x7f8c7c1102b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400b0a0 tx=0x7f8c7400b3b0 comp rx=0 tx=0).stop 2024-04-26T17:55:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.754+0000 7f8c81de5700 1 -- 172.21.15.31:0/293793290 shutdown_connections 2024-04-26T17:55:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.754+0000 7f8c81de5700 1 --2- 172.21.15.31:0/293793290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c7c06c510 0x7f8c7c1102b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.754+0000 7f8c81de5700 1 --2- 172.21.15.31:0/293793290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c7c1094e0 0x7f8c7c1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.754+0000 7f8c81de5700 1 --2- 172.21.15.31:0/293793290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c7c06bb50 0x7f8c7c06bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.754+0000 7f8c81de5700 1 -- 172.21.15.31:0/293793290 >> 172.21.15.31:0/293793290 conn(0x7f8c7c078190 msgr2=0x7f8c7c0785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.755+0000 7f8c81de5700 1 -- 172.21.15.31:0/293793290 shutdown_connections 2024-04-26T17:55:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.755+0000 7f8c81de5700 1 -- 172.21.15.31:0/293793290 wait complete. 2024-04-26T17:55:14.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.755+0000 7f8c81de5700 1 Processor -- start 2024-04-26T17:55:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.755+0000 7f8c81de5700 1 -- start start 2024-04-26T17:55:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c81de5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c7c06bb50 0x7f8c7c1a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c81de5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c7c06c510 0x7f8c7c1a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c81de5700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c7c1094e0 0x7f8c7c19bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c81de5700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c113610 con 0x7f8c7c06c510 2024-04-26T17:55:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c81de5700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c113490 con 0x7f8c7c1094e0 2024-04-26T17:55:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c81de5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c113790 con 0x7f8c7c06bb50 2024-04-26T17:55:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c7affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c7c06c510 0x7f8c7c1a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c7affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c7c06c510 0x7f8c7c1a1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39762/0 (socket says 172.21.15.31:39762) 2024-04-26T17:55:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c7affd700 1 -- 172.21.15.31:0/2084593140 learned_addr learned my addr 172.21.15.31:0/2084593140 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c7bfff700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c7c1094e0 0x7f8c7c19bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c7b7fe700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c7c06bb50 0x7f8c7c1a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:14.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c7affd700 1 -- 172.21.15.31:0/2084593140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c7c06bb50 msgr2=0x7f8c7c1a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c7affd700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c7c06bb50 0x7f8c7c1a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c7affd700 1 -- 172.21.15.31:0/2084593140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c7c1094e0 msgr2=0x7f8c7c19bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c7affd700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c7c1094e0 0x7f8c7c19bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c7affd700 1 -- 172.21.15.31:0/2084593140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7c19c240 con 0x7f8c7c06c510 2024-04-26T17:55:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c7b7fe700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c7c06bb50 0x7f8c7c1a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:55:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.756+0000 7f8c7affd700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c7c06c510 0x7f8c7c1a1e60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000d980 tx=0x7f8c7000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.757+0000 7f8c78ff9700 1 -- 172.21.15.31:0/2084593140 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c70016070 con 0x7f8c7c06c510 2024-04-26T17:55:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.757+0000 7f8c78ff9700 1 -- 172.21.15.31:0/2084593140 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c70004830 con 0x7f8c7c06c510 2024-04-26T17:55:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.757+0000 7f8c81de5700 1 -- 172.21.15.31:0/2084593140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c7c19c530 con 0x7f8c7c06c510 2024-04-26T17:55:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.757+0000 7f8c78ff9700 1 -- 172.21.15.31:0/2084593140 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c700054c0 con 0x7f8c7c06c510 2024-04-26T17:55:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.757+0000 7f8c81de5700 1 -- 172.21.15.31:0/2084593140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c7c1a87f0 con 0x7f8c7c06c510 2024-04-26T17:55:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.758+0000 7f8c81de5700 1 -- 172.21.15.31:0/2084593140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c7c104b80 con 0x7f8c7c06c510 2024-04-26T17:55:14.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.759+0000 7f8c78ff9700 1 -- 172.21.15.31:0/2084593140 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8c70004220 con 0x7f8c7c06c510 2024-04-26T17:55:14.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.760+0000 7f8c78ff9700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8c640a3c40 0x7f8c640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:14.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.760+0000 7f8c7b7fe700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8c640a3c40 0x7f8c640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:14.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.760+0000 7f8c78ff9700 1 -- 172.21.15.31:0/2084593140 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f8c700d0770 con 0x7f8c7c06c510 2024-04-26T17:55:14.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.760+0000 7f8c7b7fe700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8c640a3c40 0x7f8c640a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c005fd0 tx=0x7f8c6c00a5e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:14.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.764+0000 7f8c78ff9700 1 -- 172.21.15.31:0/2084593140 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c7001f020 con 0x7f8c7c06c510 2024-04-26T17:55:14.892 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383716 got 343597383722 for osd.11 2024-04-26T17:55:14.892 DEBUG:teuthology.parallel:result is None 2024-04-26T17:55:14.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.980+0000 7f8c81de5700 1 -- 172.21.15.31:0/2084593140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f8c7c1a8a20 con 0x7f8c7c06c510 2024-04-26T17:55:14.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.981+0000 7f8c78ff9700 1 -- 172.21.15.31:0/2084593140 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8c7009d120 con 0x7f8c7c06c510 2024-04-26T17:55:14.982 INFO:teuthology.orchestra.run.smithi031.stdout:197568495684 2024-04-26T17:55:14.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.983+0000 7f8c81de5700 1 -- 172.21.15.31:0/2084593140 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8c640a3c40 msgr2=0x7f8c640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:14.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.983+0000 7f8c81de5700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8c640a3c40 0x7f8c640a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c005fd0 tx=0x7f8c6c00a5e0 comp rx=0 tx=0).stop 2024-04-26T17:55:14.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.983+0000 7f8c81de5700 1 -- 172.21.15.31:0/2084593140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c7c06c510 msgr2=0x7f8c7c1a1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:14.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.983+0000 7f8c81de5700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c7c06c510 0x7f8c7c1a1e60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000d980 tx=0x7f8c7000dc90 comp rx=0 tx=0).stop 2024-04-26T17:55:14.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.983+0000 7f8c81de5700 1 -- 172.21.15.31:0/2084593140 shutdown_connections 2024-04-26T17:55:14.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.983+0000 7f8c81de5700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f8c640a3c40 0x7f8c640a6100 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:14.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.983+0000 7f8c81de5700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c7c1094e0 0x7f8c7c19bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:14.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.983+0000 7f8c81de5700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c7c06bb50 0x7f8c7c1a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:14.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.983+0000 7f8c81de5700 1 --2- 172.21.15.31:0/2084593140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c7c06c510 0x7f8c7c1a1e60 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:14.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.983+0000 7f8c81de5700 1 -- 172.21.15.31:0/2084593140 >> 172.21.15.31:0/2084593140 conn(0x7f8c7c078190 msgr2=0x7f8c7c10e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:14.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.983+0000 7f8c81de5700 1 -- 172.21.15.31:0/2084593140 shutdown_connections 2024-04-26T17:55:14.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:14.984+0000 7f8c81de5700 1 -- 172.21.15.31:0/2084593140 wait complete. 2024-04-26T17:55:15.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:15 smithi031 ceph-mon[29164]: pgmap v363: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:15.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:15 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2084593140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T17:55:15.642 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495678 got 197568495684 for osd.6 2024-04-26T17:55:15.643 DEBUG:teuthology.parallel:result is None 2024-04-26T17:55:15.643 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T17:55:15.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T17:55:15.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:15 smithi186 ceph-mon[32797]: pgmap v363: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:15.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:15 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2084593140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T17:55:15.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:15 smithi097 ceph-mon[32419]: pgmap v363: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:15.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:15 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2084593140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T17:55:15.909 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:17.046 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:17.047 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:17.047 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:55:17.047 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:55:17.047 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:17.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:16 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:17.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:16 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:17.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:16 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:55:17.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:16 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:55:17.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:16 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:17.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:16 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:17.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:16 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:17.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:16 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:55:17.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:16 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:55:17.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:16 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:55:17.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.579+0000 7fdf860e6700 1 -- 172.21.15.31:0/2620474018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf80103e30 msgr2=0x7fdf80110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:17.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.579+0000 7fdf860e6700 1 --2- 172.21.15.31:0/2620474018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf80103e30 0x7fdf80110290 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000b0a0 tx=0x7fdf7000b3b0 comp rx=0 tx=0).stop 2024-04-26T17:55:17.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.579+0000 7fdf860e6700 1 -- 172.21.15.31:0/2620474018 shutdown_connections 2024-04-26T17:55:17.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.579+0000 7fdf860e6700 1 --2- 172.21.15.31:0/2620474018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf80103e30 0x7fdf80110290 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:17.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.579+0000 7fdf860e6700 1 --2- 172.21.15.31:0/2620474018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf801094c0 0x7fdf801098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:17.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.579+0000 7fdf860e6700 1 --2- 172.21.15.31:0/2620474018 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf80103470 0x7fdf801038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.579+0000 7fdf860e6700 1 -- 172.21.15.31:0/2620474018 >> 172.21.15.31:0/2620474018 conn(0x7fdf8007a580 msgr2=0x7fdf8007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.579+0000 7fdf860e6700 1 -- 172.21.15.31:0/2620474018 shutdown_connections 2024-04-26T17:55:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.579+0000 7fdf860e6700 1 -- 172.21.15.31:0/2620474018 wait complete. 2024-04-26T17:55:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.580+0000 7fdf860e6700 1 Processor -- start 2024-04-26T17:55:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.580+0000 7fdf860e6700 1 -- start start 2024-04-26T17:55:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.580+0000 7fdf860e6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf80103470 0x7fdf80075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.580+0000 7fdf860e6700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf80103e30 0x7fdf80075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.580+0000 7fdf860e6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf801094c0 0x7fdf800761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.580+0000 7fdf860e6700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdf801135c0 con 0x7fdf801094c0 2024-04-26T17:55:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.580+0000 7fdf860e6700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fdf80113440 con 0x7fdf80103e30 2024-04-26T17:55:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.580+0000 7fdf860e6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdf80113740 con 0x7fdf80103470 2024-04-26T17:55:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.580+0000 7fdf7ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf801094c0 0x7fdf800761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.580+0000 7fdf7ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf801094c0 0x7fdf800761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39774/0 (socket says 172.21.15.31:39774) 2024-04-26T17:55:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.580+0000 7fdf7ffff700 1 -- 172.21.15.31:0/206435013 learned_addr learned my addr 172.21.15.31:0/206435013 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.581+0000 7fdf7effd700 1 --2- 172.21.15.31:0/206435013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf80103e30 0x7fdf80075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.581+0000 7fdf7ffff700 1 -- 172.21.15.31:0/206435013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf80103470 msgr2=0x7fdf80075720 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T17:55:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.581+0000 7fdf7ffff700 1 --2- 172.21.15.31:0/206435013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf80103470 0x7fdf80075720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.581+0000 7fdf7ffff700 1 -- 172.21.15.31:0/206435013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf80103e30 msgr2=0x7fdf80075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.581+0000 7fdf7ffff700 1 --2- 172.21.15.31:0/206435013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf80103e30 0x7fdf80075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.581+0000 7fdf7ffff700 1 -- 172.21.15.31:0/206435013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf801a7fc0 con 0x7fdf801094c0 2024-04-26T17:55:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.581+0000 7fdf7effd700 1 --2- 172.21.15.31:0/206435013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf80103e30 0x7fdf80075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:55:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.581+0000 7fdf7ffff700 1 --2- 172.21.15.31:0/206435013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf801094c0 0x7fdf800761a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000b830 tx=0x7fdf70004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.581+0000 7fdf7cff9700 1 -- 172.21.15.31:0/206435013 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70021070 con 0x7fdf801094c0 2024-04-26T17:55:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.581+0000 7fdf7cff9700 1 -- 172.21.15.31:0/206435013 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf70003c00 con 0x7fdf801094c0 2024-04-26T17:55:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.581+0000 7fdf860e6700 1 -- 172.21.15.31:0/206435013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf801a8250 con 0x7fdf801094c0 2024-04-26T17:55:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.581+0000 7fdf7cff9700 1 -- 172.21.15.31:0/206435013 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70004810 con 0x7fdf801094c0 2024-04-26T17:55:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.581+0000 7fdf860e6700 1 -- 172.21.15.31:0/206435013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf801a8700 con 0x7fdf801094c0 2024-04-26T17:55:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.582+0000 7fdf860e6700 1 -- 172.21.15.31:0/206435013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf8006a110 con 0x7fdf801094c0 2024-04-26T17:55:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.583+0000 7fdf7cff9700 1 -- 172.21.15.31:0/206435013 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdf70007a00 con 0x7fdf801094c0 2024-04-26T17:55:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.584+0000 7fdf7cff9700 1 --2- 172.21.15.31:0/206435013 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fdf6c0a3c40 0x7fdf6c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.584+0000 7fdf7f7fe700 1 --2- 172.21.15.31:0/206435013 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fdf6c0a3c40 0x7fdf6c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.584+0000 7fdf7cff9700 1 -- 172.21.15.31:0/206435013 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fdf7000f020 con 0x7fdf801094c0 2024-04-26T17:55:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.584+0000 7fdf7f7fe700 1 --2- 172.21.15.31:0/206435013 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fdf6c0a3c40 0x7fdf6c0a6100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdf78005fd0 tx=0x7fdf78005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:17.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.588+0000 7fdf7cff9700 1 -- 172.21.15.31:0/206435013 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf7002a020 con 0x7fdf801094c0 2024-04-26T17:55:17.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.797+0000 7fdf860e6700 1 -- 172.21.15.31:0/206435013 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdf80063860 con 0x7fdf6c0a3c40 2024-04-26T17:55:17.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.807+0000 7fdf7cff9700 1 -- 172.21.15.31:0/206435013 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259002 (secure 0 0 0) 0x7fdf80063860 con 0x7fdf6c0a3c40 2024-04-26T17:55:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:55:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.809+0000 7fdf860e6700 1 -- 172.21.15.31:0/206435013 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fdf6c0a3c40 msgr2=0x7fdf6c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.809+0000 7fdf860e6700 1 --2- 172.21.15.31:0/206435013 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fdf6c0a3c40 0x7fdf6c0a6100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdf78005fd0 tx=0x7fdf78005f60 comp rx=0 tx=0).stop 2024-04-26T17:55:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.809+0000 7fdf860e6700 1 -- 172.21.15.31:0/206435013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf801094c0 msgr2=0x7fdf800761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.809+0000 7fdf860e6700 1 --2- 172.21.15.31:0/206435013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf801094c0 0x7fdf800761a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000b830 tx=0x7fdf70004d90 comp rx=0 tx=0).stop 2024-04-26T17:55:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.810+0000 7fdf860e6700 1 -- 172.21.15.31:0/206435013 shutdown_connections 2024-04-26T17:55:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.810+0000 7fdf860e6700 1 --2- 172.21.15.31:0/206435013 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fdf6c0a3c40 0x7fdf6c0a6100 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.810+0000 7fdf860e6700 1 --2- 172.21.15.31:0/206435013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf801094c0 0x7fdf800761a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.810+0000 7fdf860e6700 1 --2- 172.21.15.31:0/206435013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf80103470 0x7fdf80075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.810+0000 7fdf860e6700 1 --2- 172.21.15.31:0/206435013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf80103e30 0x7fdf80075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.810+0000 7fdf860e6700 1 -- 172.21.15.31:0/206435013 >> 172.21.15.31:0/206435013 conn(0x7fdf8007a580 msgr2=0x7fdf801020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.810+0000 7fdf860e6700 1 -- 172.21.15.31:0/206435013 shutdown_connections 2024-04-26T17:55:17.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:17.810+0000 7fdf860e6700 1 -- 172.21.15.31:0/206435013 wait complete. 2024-04-26T17:55:17.814 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-26T17:55:18.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:17 smithi031 ceph-mon[29164]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:18.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:17 smithi097 ceph-mon[32419]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:18.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:17 smithi186 ceph-mon[32797]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:18.478 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-04-26T17:55:16.340157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657328,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":838,"num_read_kb":542,"num_write":355,"num_write_kb":3058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361164,"kb_used_data":12012,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580660,"statfs":{"total":1151940427776,"available":1151570595840,"internally_reserved":0,"allocated":12300288,"data_stored":7722624,"data_compressed":106863,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002911"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.349263+0000","last_change":"2024-04-26T17:53:03.616592+0000","last_active":"2024-04-26T17:54:48.349263+0000","last_peered":"2024-04-26T17:54:48.349263+0000","last_clean":"2024-04-26T17:54:48.349263+0000","last_became_active":"2024-04-26T17:53:03.616255+0000","last_became_peered":"2024-04-26T17:53:03.616255+0000","last_unstale":"2024-04-26T17:54:48.349263+0000","last_undegraded":"2024-04-26T17:54:48.349263+0000","last_fullsized":"2024-04-26T17:54:48.349263+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:43:31.183679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.380447+0000","last_change":"2024-04-26T17:53:03.615402+0000","last_active":"2024-04-26T17:54:48.380447+0000","last_peered":"2024-04-26T17:54:48.380447+0000","last_clean":"2024-04-26T17:54:48.380447+0000","last_became_active":"2024-04-26T17:53:03.615048+0000","last_became_peered":"2024-04-26T17:53:03.615048+0000","last_unstale":"2024-04-26T17:54:48.380447+0000","last_undegraded":"2024-04-26T17:54:48.380447+0000","last_fullsized":"2024-04-26T17:54:48.380447+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:05:12.454770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.386996+0000","last_change":"2024-04-26T17:53:03.622625+0000","last_active":"2024-04-26T17:54:48.386996+0000","last_peered":"2024-04-26T17:54:48.386996+0000","last_clean":"2024-04-26T17:54:48.386996+0000","last_became_active":"2024-04-26T17:53:03.622305+0000","last_became_peered":"2024-04-26T17:53:03.622305+0000","last_unstale":"2024-04-26T17:54:48.386996+0000","last_undegraded":"2024-04-26T17:54:48.386996+0000","last_fullsized":"2024-04-26T17:54:48.386996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:06.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.340614+0000","last_change":"2024-04-26T17:53:03.617650+0000","last_active":"2024-04-26T17:54:48.340614+0000","last_peered":"2024-04-26T17:54:48.340614+0000","last_clean":"2024-04-26T17:54:48.340614+0000","last_became_active":"2024-04-26T17:53:03.617220+0000","last_became_peered":"2024-04-26T17:53:03.617220+0000","last_unstale":"2024-04-26T17:54:48.340614+0000","last_undegraded":"2024-04-26T17:54:48.340614+0000","last_fullsized":"2024-04-26T17:54:48.340614+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:31.765577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.344350+0000","last_change":"2024-04-26T17:53:03.615429+0000","last_active":"2024-04-26T17:54:48.344350+0000","last_peered":"2024-04-26T17:54:48.344350+0000","last_clean":"2024-04-26T17:54:48.344350+0000","last_became_active":"2024-04-26T17:53:03.615079+0000","last_became_peered":"2024-04-26T17:53:03.615079+0000","last_unstale":"2024-04-26T17:54:48.344350+0000","last_undegraded":"2024-04-26T17:54:48.344350+0000","last_fullsized":"2024-04-26T17:54:48.344350+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:20.994016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.355757+0000","last_change":"2024-04-26T17:53:03.624619+0000","last_active":"2024-04-26T17:54:48.355757+0000","last_peered":"2024-04-26T17:54:48.355757+0000","last_clean":"2024-04-26T17:54:48.355757+0000","last_became_active":"2024-04-26T17:53:03.624251+0000","last_became_peered":"2024-04-26T17:53:03.624251+0000","last_unstale":"2024-04-26T17:54:48.355757+0000","last_undegraded":"2024-04-26T17:54:48.355757+0000","last_fullsized":"2024-04-26T17:54:48.355757+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:50:17.535235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.352676+0000","last_change":"2024-04-26T17:53:03.619325+0000","last_active":"2024-04-26T17:54:48.352676+0000","last_peered":"2024-04-26T17:54:48.352676+0000","last_clean":"2024-04-26T17:54:48.352676+0000","last_became_active":"2024-04-26T17:53:03.618989+0000","last_became_peered":"2024-04-26T17:53:03.618989+0000","last_unstale":"2024-04-26T17:54:48.352676+0000","last_undegraded":"2024-04-26T17:54:48.352676+0000","last_fullsized":"2024-04-26T17:54:48.352676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:10.195742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.396614+0000","last_change":"2024-04-26T17:53:03.609346+0000","last_active":"2024-04-26T17:54:48.396614+0000","last_peered":"2024-04-26T17:54:48.396614+0000","last_clean":"2024-04-26T17:54:48.396614+0000","last_became_active":"2024-04-26T17:53:03.609021+0000","last_became_peered":"2024-04-26T17:53:03.609021+0000","last_unstale":"2024-04-26T17:54:48.396614+0000","last_undegraded":"2024-04-26T17:54:48.396614+0000","last_fullsized":"2024-04-26T17:54:48.396614+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:44.328599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.355179+0000","last_change":"2024-04-26T17:53:03.618573+0000","last_active":"2024-04-26T17:54:48.355179+0000","last_peered":"2024-04-26T17:54:48.355179+0000","last_clean":"2024-04-26T17:54:48.355179+0000","last_became_active":"2024-04-26T17:53:03.618223+0000","last_became_peered":"2024-04-26T17:53:03.618223+0000","last_unstale":"2024-04-26T17:54:48.355179+0000","last_undegraded":"2024-04-26T17:54:48.355179+0000","last_fullsized":"2024-04-26T17:54:48.355179+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:16.749205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.346295+0000","last_change":"2024-04-26T17:53:03.621530+0000","last_active":"2024-04-26T17:54:48.346295+0000","last_peered":"2024-04-26T17:54:48.346295+0000","last_clean":"2024-04-26T17:54:48.346295+0000","last_became_active":"2024-04-26T17:53:03.621174+0000","last_became_peered":"2024-04-26T17:53:03.621174+0000","last_unstale":"2024-04-26T17:54:48.346295+0000","last_undegraded":"2024-04-26T17:54:48.346295+0000","last_fullsized":"2024-04-26T17:54:48.346295+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:31.388232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.361254+0000","last_change":"2024-04-26T17:53:03.617682+0000","last_active":"2024-04-26T17:54:48.361254+0000","last_peered":"2024-04-26T17:54:48.361254+0000","last_clean":"2024-04-26T17:54:48.361254+0000","last_became_active":"2024-04-26T17:53:03.617348+0000","last_became_peered":"2024-04-26T17:53:03.617348+0000","last_unstale":"2024-04-26T17:54:48.361254+0000","last_undegraded":"2024-04-26T17:54:48.361254+0000","last_fullsized":"2024-04-26T17:54:48.361254+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:21.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.333063+0000","last_change":"2024-04-26T17:53:03.617955+0000","last_active":"2024-04-26T17:54:48.333063+0000","last_peered":"2024-04-26T17:54:48.333063+0000","last_clean":"2024-04-26T17:54:48.333063+0000","last_became_active":"2024-04-26T17:53:03.616765+0000","last_became_peered":"2024-04-26T17:53:03.616765+0000","last_unstale":"2024-04-26T17:54:48.333063+0000","last_undegraded":"2024-04-26T17:54:48.333063+0000","last_fullsized":"2024-04-26T17:54:48.333063+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:59.292277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.358178+0000","last_change":"2024-04-26T17:53:49.767807+0000","last_active":"2024-04-26T17:54:48.358178+0000","last_peered":"2024-04-26T17:54:48.358178+0000","last_clean":"2024-04-26T17:54:48.358178+0000","last_became_active":"2024-04-26T17:53:49.767404+0000","last_became_peered":"2024-04-26T17:53:49.767404+0000","last_unstale":"2024-04-26T17:54:48.358178+0000","last_undegraded":"2024-04-26T17:54:48.358178+0000","last_fullsized":"2024-04-26T17:54:48.358178+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:20:45.482580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.386973+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:54:48.386973+0000","last_peered":"2024-04-26T17:54:48.386973+0000","last_clean":"2024-04-26T17:54:48.386973+0000","last_became_active":"2024-04-26T17:53:03.616623+0000","last_became_peered":"2024-04-26T17:53:03.616623+0000","last_unstale":"2024-04-26T17:54:48.386973+0000","last_undegraded":"2024-04-26T17:54:48.386973+0000","last_fullsized":"2024-04-26T17:54:48.386973+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:04.231549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.387143+0000","last_change":"2024-04-26T17:53:03.622147+0000","last_active":"2024-04-26T17:54:48.387143+0000","last_peered":"2024-04-26T17:54:48.387143+0000","last_clean":"2024-04-26T17:54:48.387143+0000","last_became_active":"2024-04-26T17:53:03.621542+0000","last_became_peered":"2024-04-26T17:53:03.621542+0000","last_unstale":"2024-04-26T17:54:48.387143+0000","last_undegraded":"2024-04-26T17:54:48.387143+0000","last_fullsized":"2024-04-26T17:54:48.387143+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:01.746954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.385788+0000","last_change":"2024-04-26T17:53:03.624433+0000","last_active":"2024-04-26T17:54:48.385788+0000","last_peered":"2024-04-26T17:54:48.385788+0000","last_clean":"2024-04-26T17:54:48.385788+0000","last_became_active":"2024-04-26T17:53:03.624135+0000","last_became_peered":"2024-04-26T17:53:03.624135+0000","last_unstale":"2024-04-26T17:54:48.385788+0000","last_undegraded":"2024-04-26T17:54:48.385788+0000","last_fullsized":"2024-04-26T17:54:48.385788+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:32:20.521917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.348971+0000","last_change":"2024-04-26T17:53:03.623607+0000","last_active":"2024-04-26T17:54:48.348971+0000","last_peered":"2024-04-26T17:54:48.348971+0000","last_clean":"2024-04-26T17:54:48.348971+0000","last_became_active":"2024-04-26T17:53:03.621783+0000","last_became_peered":"2024-04-26T17:53:03.621783+0000","last_unstale":"2024-04-26T17:54:48.348971+0000","last_undegraded":"2024-04-26T17:54:48.348971+0000","last_fullsized":"2024-04-26T17:54:48.348971+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:38:59.872868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.359683+0000","last_change":"2024-04-26T17:53:03.614974+0000","last_active":"2024-04-26T17:54:48.359683+0000","last_peered":"2024-04-26T17:54:48.359683+0000","last_clean":"2024-04-26T17:54:48.359683+0000","last_became_active":"2024-04-26T17:53:03.614619+0000","last_became_peered":"2024-04-26T17:53:03.614619+0000","last_unstale":"2024-04-26T17:54:48.359683+0000","last_undegraded":"2024-04-26T17:54:48.359683+0000","last_fullsized":"2024-04-26T17:54:48.359683+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:50.729531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.373272+0000","last_change":"2024-04-26T17:53:03.611140+0000","last_active":"2024-04-26T17:54:48.373272+0000","last_peered":"2024-04-26T17:54:48.373272+0000","last_clean":"2024-04-26T17:54:48.373272+0000","last_became_active":"2024-04-26T17:53:03.610549+0000","last_became_peered":"2024-04-26T17:53:03.610549+0000","last_unstale":"2024-04-26T17:54:48.373272+0000","last_undegraded":"2024-04-26T17:54:48.373272+0000","last_fullsized":"2024-04-26T17:54:48.373272+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:46:17.178598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.761876+0000","last_change":"2024-04-26T17:53:00.585971+0000","last_active":"2024-04-26T17:53:49.761876+0000","last_peered":"2024-04-26T17:53:49.761876+0000","last_clean":"2024-04-26T17:53:49.761876+0000","last_became_active":"2024-04-26T17:53:00.585648+0000","last_became_peered":"2024-04-26T17:53:00.585648+0000","last_unstale":"2024-04-26T17:53:49.761876+0000","last_undegraded":"2024-04-26T17:53:49.761876+0000","last_fullsized":"2024-04-26T17:53:49.761876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:00:24.082773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.358423+0000","last_change":"2024-04-26T17:53:49.770579+0000","last_active":"2024-04-26T17:54:48.358423+0000","last_peered":"2024-04-26T17:54:48.358423+0000","last_clean":"2024-04-26T17:54:48.358423+0000","last_became_active":"2024-04-26T17:53:49.770250+0000","last_became_peered":"2024-04-26T17:53:49.770250+0000","last_unstale":"2024-04-26T17:54:48.358423+0000","last_undegraded":"2024-04-26T17:54:48.358423+0000","last_fullsized":"2024-04-26T17:54:48.358423+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:35.758790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.766078+0000","last_change":"2024-04-26T17:53:01.145438+0000","last_active":"2024-04-26T17:53:49.766078+0000","last_peered":"2024-04-26T17:53:49.766078+0000","last_clean":"2024-04-26T17:53:49.766078+0000","last_became_active":"2024-04-26T17:53:01.145030+0000","last_became_peered":"2024-04-26T17:53:01.145030+0000","last_unstale":"2024-04-26T17:53:49.766078+0000","last_undegraded":"2024-04-26T17:53:49.766078+0000","last_fullsized":"2024-04-26T17:53:49.766078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:15.650247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.387694+0000","last_change":"2024-04-26T17:53:03.622647+0000","last_active":"2024-04-26T17:54:48.387694+0000","last_peered":"2024-04-26T17:54:48.387694+0000","last_clean":"2024-04-26T17:54:48.387694+0000","last_became_active":"2024-04-26T17:53:03.622298+0000","last_became_peered":"2024-04-26T17:53:03.622298+0000","last_unstale":"2024-04-26T17:54:48.387694+0000","last_undegraded":"2024-04-26T17:54:48.387694+0000","last_fullsized":"2024-04-26T17:54:48.387694+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:37.230539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.775803+0000","last_change":"2024-04-26T17:53:49.775405+0000","last_active":"2024-04-26T17:53:49.775803+0000","last_peered":"2024-04-26T17:53:49.775803+0000","last_clean":"2024-04-26T17:53:49.775803+0000","last_became_active":"2024-04-26T17:53:49.770536+0000","last_became_peered":"2024-04-26T17:53:49.770536+0000","last_unstale":"2024-04-26T17:53:49.775803+0000","last_undegraded":"2024-04-26T17:53:49.775803+0000","last_fullsized":"2024-04-26T17:53:49.775803+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:34.643452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.362560+0000","last_change":"2024-04-26T17:53:03.617462+0000","last_active":"2024-04-26T17:54:48.362560+0000","last_peered":"2024-04-26T17:54:48.362560+0000","last_clean":"2024-04-26T17:54:48.362560+0000","last_became_active":"2024-04-26T17:53:03.617115+0000","last_became_peered":"2024-04-26T17:53:03.617115+0000","last_unstale":"2024-04-26T17:54:48.362560+0000","last_undegraded":"2024-04-26T17:54:48.362560+0000","last_fullsized":"2024-04-26T17:54:48.362560+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:11.406146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767810+0000","last_change":"2024-04-26T17:53:00.583378+0000","last_active":"2024-04-26T17:53:49.767810+0000","last_peered":"2024-04-26T17:53:49.767810+0000","last_clean":"2024-04-26T17:53:49.767810+0000","last_became_active":"2024-04-26T17:53:00.583020+0000","last_became_peered":"2024-04-26T17:53:00.583020+0000","last_unstale":"2024-04-26T17:53:49.767810+0000","last_undegraded":"2024-04-26T17:53:49.767810+0000","last_fullsized":"2024-04-26T17:53:49.767810+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:24.089996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.356230+0000","last_change":"2024-04-26T17:53:03.609557+0000","last_active":"2024-04-26T17:54:48.356230+0000","last_peered":"2024-04-26T17:54:48.356230+0000","last_clean":"2024-04-26T17:54:48.356230+0000","last_became_active":"2024-04-26T17:53:03.609028+0000","last_became_peered":"2024-04-26T17:53:03.609028+0000","last_unstale":"2024-04-26T17:54:48.356230+0000","last_undegraded":"2024-04-26T17:54:48.356230+0000","last_fullsized":"2024-04-26T17:54:48.356230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:39:22.860877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":150,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:55:13.864356+0000","last_change":"2024-04-26T17:53:00.586683+0000","last_active":"2024-04-26T17:55:13.864356+0000","last_peered":"2024-04-26T17:55:13.864356+0000","last_clean":"2024-04-26T17:55:13.864356+0000","last_became_active":"2024-04-26T17:53:00.586272+0000","last_became_peered":"2024-04-26T17:53:00.586272+0000","last_unstale":"2024-04-26T17:55:13.864356+0000","last_undegraded":"2024-04-26T17:55:13.864356+0000","last_fullsized":"2024-04-26T17:55:13.864356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:20:29.068607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.352410+0000","last_change":"2024-04-26T17:53:03.611073+0000","last_active":"2024-04-26T17:54:48.352410+0000","last_peered":"2024-04-26T17:54:48.352410+0000","last_clean":"2024-04-26T17:54:48.352410+0000","last_became_active":"2024-04-26T17:53:03.610701+0000","last_became_peered":"2024-04-26T17:53:03.610701+0000","last_unstale":"2024-04-26T17:54:48.352410+0000","last_undegraded":"2024-04-26T17:54:48.352410+0000","last_fullsized":"2024-04-26T17:54:48.352410+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:01:00.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764638+0000","last_change":"2024-04-26T17:53:00.586082+0000","last_active":"2024-04-26T17:53:49.764638+0000","last_peered":"2024-04-26T17:53:49.764638+0000","last_clean":"2024-04-26T17:53:49.764638+0000","last_became_active":"2024-04-26T17:53:00.585678+0000","last_became_peered":"2024-04-26T17:53:00.585678+0000","last_unstale":"2024-04-26T17:53:49.764638+0000","last_undegraded":"2024-04-26T17:53:49.764638+0000","last_fullsized":"2024-04-26T17:53:49.764638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:55:25.523846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.356370+0000","last_change":"2024-04-26T17:53:03.600230+0000","last_active":"2024-04-26T17:54:48.356370+0000","last_peered":"2024-04-26T17:54:48.356370+0000","last_clean":"2024-04-26T17:54:48.356370+0000","last_became_active":"2024-04-26T17:53:03.599713+0000","last_became_peered":"2024-04-26T17:53:03.599713+0000","last_unstale":"2024-04-26T17:54:48.356370+0000","last_undegraded":"2024-04-26T17:54:48.356370+0000","last_fullsized":"2024-04-26T17:54:48.356370+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:45.822561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760523+0000","last_change":"2024-04-26T17:53:00.584588+0000","last_active":"2024-04-26T17:53:49.760523+0000","last_peered":"2024-04-26T17:53:49.760523+0000","last_clean":"2024-04-26T17:53:49.760523+0000","last_became_active":"2024-04-26T17:53:00.584284+0000","last_became_peered":"2024-04-26T17:53:00.584284+0000","last_unstale":"2024-04-26T17:53:49.760523+0000","last_undegraded":"2024-04-26T17:53:49.760523+0000","last_fullsized":"2024-04-26T17:53:49.760523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:08.428904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.393973+0000","last_change":"2024-04-26T17:53:03.615237+0000","last_active":"2024-04-26T17:54:48.393973+0000","last_peered":"2024-04-26T17:54:48.393973+0000","last_clean":"2024-04-26T17:54:48.393973+0000","last_became_active":"2024-04-26T17:53:03.614704+0000","last_became_peered":"2024-04-26T17:53:03.614704+0000","last_unstale":"2024-04-26T17:54:48.393973+0000","last_undegraded":"2024-04-26T17:54:48.393973+0000","last_fullsized":"2024-04-26T17:54:48.393973+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:27.251839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758907+0000","last_change":"2024-04-26T17:53:00.589287+0000","last_active":"2024-04-26T17:53:49.758907+0000","last_peered":"2024-04-26T17:53:49.758907+0000","last_clean":"2024-04-26T17:53:49.758907+0000","last_became_active":"2024-04-26T17:53:00.588119+0000","last_became_peered":"2024-04-26T17:53:00.588119+0000","last_unstale":"2024-04-26T17:53:49.758907+0000","last_undegraded":"2024-04-26T17:53:49.758907+0000","last_fullsized":"2024-04-26T17:53:49.758907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:25.389777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.392842+0000","last_change":"2024-04-26T17:53:03.608061+0000","last_active":"2024-04-26T17:54:48.392842+0000","last_peered":"2024-04-26T17:54:48.392842+0000","last_clean":"2024-04-26T17:54:48.392842+0000","last_became_active":"2024-04-26T17:53:03.607624+0000","last_became_peered":"2024-04-26T17:53:03.607624+0000","last_unstale":"2024-04-26T17:54:48.392842+0000","last_undegraded":"2024-04-26T17:54:48.392842+0000","last_fullsized":"2024-04-26T17:54:48.392842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:15.111011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":143,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:55:13.863837+0000","last_change":"2024-04-26T17:53:00.588621+0000","last_active":"2024-04-26T17:55:13.863837+0000","last_peered":"2024-04-26T17:55:13.863837+0000","last_clean":"2024-04-26T17:55:13.863837+0000","last_became_active":"2024-04-26T17:53:00.587330+0000","last_became_peered":"2024-04-26T17:53:00.587330+0000","last_unstale":"2024-04-26T17:55:13.863837+0000","last_undegraded":"2024-04-26T17:55:13.863837+0000","last_fullsized":"2024-04-26T17:55:13.863837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:01:12.178610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.363628+0000","last_change":"2024-04-26T17:53:03.615723+0000","last_active":"2024-04-26T17:54:48.363628+0000","last_peered":"2024-04-26T17:54:48.363628+0000","last_clean":"2024-04-26T17:54:48.363628+0000","last_became_active":"2024-04-26T17:53:03.614931+0000","last_became_peered":"2024-04-26T17:53:03.614931+0000","last_unstale":"2024-04-26T17:54:48.363628+0000","last_undegraded":"2024-04-26T17:54:48.363628+0000","last_fullsized":"2024-04-26T17:54:48.363628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:09.663876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767693+0000","last_change":"2024-04-26T17:53:00.589232+0000","last_active":"2024-04-26T17:53:49.767693+0000","last_peered":"2024-04-26T17:53:49.767693+0000","last_clean":"2024-04-26T17:53:49.767693+0000","last_became_active":"2024-04-26T17:53:00.588863+0000","last_became_peered":"2024-04-26T17:53:00.588863+0000","last_unstale":"2024-04-26T17:53:49.767693+0000","last_undegraded":"2024-04-26T17:53:49.767693+0000","last_fullsized":"2024-04-26T17:53:49.767693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:14.360782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.386612+0000","last_change":"2024-04-26T17:53:03.624923+0000","last_active":"2024-04-26T17:54:48.386612+0000","last_peered":"2024-04-26T17:54:48.386612+0000","last_clean":"2024-04-26T17:54:48.386612+0000","last_became_active":"2024-04-26T17:53:03.624620+0000","last_became_peered":"2024-04-26T17:53:03.624620+0000","last_unstale":"2024-04-26T17:54:48.386612+0000","last_undegraded":"2024-04-26T17:54:48.386612+0000","last_fullsized":"2024-04-26T17:54:48.386612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:50:51.259563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759526+0000","last_change":"2024-04-26T17:53:01.146950+0000","last_active":"2024-04-26T17:53:49.759526+0000","last_peered":"2024-04-26T17:53:49.759526+0000","last_clean":"2024-04-26T17:53:49.759526+0000","last_became_active":"2024-04-26T17:53:01.146591+0000","last_became_peered":"2024-04-26T17:53:01.146591+0000","last_unstale":"2024-04-26T17:53:49.759526+0000","last_undegraded":"2024-04-26T17:53:49.759526+0000","last_fullsized":"2024-04-26T17:53:49.759526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:17.756660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.361614+0000","last_change":"2024-04-26T17:53:03.626944+0000","last_active":"2024-04-26T17:54:48.361614+0000","last_peered":"2024-04-26T17:54:48.361614+0000","last_clean":"2024-04-26T17:54:48.361614+0000","last_became_active":"2024-04-26T17:53:03.626594+0000","last_became_peered":"2024-04-26T17:53:03.626594+0000","last_unstale":"2024-04-26T17:54:48.361614+0000","last_undegraded":"2024-04-26T17:54:48.361614+0000","last_fullsized":"2024-04-26T17:54:48.361614+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:52:42.997251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'33","reported_seq":295,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:55:14.864787+0000","last_change":"2024-04-26T17:53:00.588353+0000","last_active":"2024-04-26T17:55:14.864787+0000","last_peered":"2024-04-26T17:55:14.864787+0000","last_clean":"2024-04-26T17:55:14.864787+0000","last_became_active":"2024-04-26T17:53:00.586351+0000","last_became_peered":"2024-04-26T17:53:00.586351+0000","last_unstale":"2024-04-26T17:55:14.864787+0000","last_undegraded":"2024-04-26T17:55:14.864787+0000","last_fullsized":"2024-04-26T17:55:14.864787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:41.660525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62584,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":236,"num_read_kb":179,"num_write":34,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.348671+0000","last_change":"2024-04-26T17:53:03.612052+0000","last_active":"2024-04-26T17:54:48.348671+0000","last_peered":"2024-04-26T17:54:48.348671+0000","last_clean":"2024-04-26T17:54:48.348671+0000","last_became_active":"2024-04-26T17:53:03.611733+0000","last_became_peered":"2024-04-26T17:53:03.611733+0000","last_unstale":"2024-04-26T17:54:48.348671+0000","last_undegraded":"2024-04-26T17:54:48.348671+0000","last_fullsized":"2024-04-26T17:54:48.348671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:05:23.958317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769171+0000","last_change":"2024-04-26T17:53:00.588284+0000","last_active":"2024-04-26T17:53:49.769171+0000","last_peered":"2024-04-26T17:53:49.769171+0000","last_clean":"2024-04-26T17:53:49.769171+0000","last_became_active":"2024-04-26T17:53:00.587942+0000","last_became_peered":"2024-04-26T17:53:00.587942+0000","last_unstale":"2024-04-26T17:53:49.769171+0000","last_undegraded":"2024-04-26T17:53:49.769171+0000","last_fullsized":"2024-04-26T17:53:49.769171+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:43.782971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.374913+0000","last_change":"2024-04-26T17:53:03.613092+0000","last_active":"2024-04-26T17:54:48.374913+0000","last_peered":"2024-04-26T17:54:48.374913+0000","last_clean":"2024-04-26T17:54:48.374913+0000","last_became_active":"2024-04-26T17:53:03.612773+0000","last_became_peered":"2024-04-26T17:53:03.612773+0000","last_unstale":"2024-04-26T17:54:48.374913+0000","last_undegraded":"2024-04-26T17:54:48.374913+0000","last_fullsized":"2024-04-26T17:54:48.374913+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:41:21.340375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759054+0000","last_change":"2024-04-26T17:53:00.588777+0000","last_active":"2024-04-26T17:53:49.759054+0000","last_peered":"2024-04-26T17:53:49.759054+0000","last_clean":"2024-04-26T17:53:49.759054+0000","last_became_active":"2024-04-26T17:53:00.588380+0000","last_became_peered":"2024-04-26T17:53:00.588380+0000","last_unstale":"2024-04-26T17:53:49.759054+0000","last_undegraded":"2024-04-26T17:53:49.759054+0000","last_fullsized":"2024-04-26T17:53:49.759054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:44.587381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.357814+0000","last_change":"2024-04-26T17:53:03.620344+0000","last_active":"2024-04-26T17:54:48.357814+0000","last_peered":"2024-04-26T17:54:48.357814+0000","last_clean":"2024-04-26T17:54:48.357814+0000","last_became_active":"2024-04-26T17:53:03.619944+0000","last_became_peered":"2024-04-26T17:53:03.619944+0000","last_unstale":"2024-04-26T17:54:48.357814+0000","last_undegraded":"2024-04-26T17:54:48.357814+0000","last_fullsized":"2024-04-26T17:54:48.357814+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:08:31.801342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":140,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:55:09.857135+0000","last_change":"2024-04-26T17:53:00.587733+0000","last_active":"2024-04-26T17:55:09.857135+0000","last_peered":"2024-04-26T17:55:09.857135+0000","last_clean":"2024-04-26T17:55:09.857135+0000","last_became_active":"2024-04-26T17:53:00.587373+0000","last_became_peered":"2024-04-26T17:53:00.587373+0000","last_unstale":"2024-04-26T17:55:09.857135+0000","last_undegraded":"2024-04-26T17:55:09.857135+0000","last_fullsized":"2024-04-26T17:55:09.857135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:10.978944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.352086+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:54:48.352086+0000","last_peered":"2024-04-26T17:54:48.352086+0000","last_clean":"2024-04-26T17:54:48.352086+0000","last_became_active":"2024-04-26T17:53:03.616588+0000","last_became_peered":"2024-04-26T17:53:03.616588+0000","last_unstale":"2024-04-26T17:54:48.352086+0000","last_undegraded":"2024-04-26T17:54:48.352086+0000","last_fullsized":"2024-04-26T17:54:48.352086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:46:38.419613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767233+0000","last_change":"2024-04-26T17:53:01.146922+0000","last_active":"2024-04-26T17:53:49.767233+0000","last_peered":"2024-04-26T17:53:49.767233+0000","last_clean":"2024-04-26T17:53:49.767233+0000","last_became_active":"2024-04-26T17:53:01.146520+0000","last_became_peered":"2024-04-26T17:53:01.146520+0000","last_unstale":"2024-04-26T17:53:49.767233+0000","last_undegraded":"2024-04-26T17:53:49.767233+0000","last_fullsized":"2024-04-26T17:53:49.767233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:12:49.238012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.350055+0000","last_change":"2024-04-26T17:53:03.608513+0000","last_active":"2024-04-26T17:54:48.350055+0000","last_peered":"2024-04-26T17:54:48.350055+0000","last_clean":"2024-04-26T17:54:48.350055+0000","last_became_active":"2024-04-26T17:53:03.608167+0000","last_became_peered":"2024-04-26T17:53:03.608167+0000","last_unstale":"2024-04-26T17:54:48.350055+0000","last_undegraded":"2024-04-26T17:54:48.350055+0000","last_fullsized":"2024-04-26T17:54:48.350055+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:03.002756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769191+0000","last_change":"2024-04-26T17:53:00.583878+0000","last_active":"2024-04-26T17:53:49.769191+0000","last_peered":"2024-04-26T17:53:49.769191+0000","last_clean":"2024-04-26T17:53:49.769191+0000","last_became_active":"2024-04-26T17:53:00.583159+0000","last_became_peered":"2024-04-26T17:53:00.583159+0000","last_unstale":"2024-04-26T17:53:49.769191+0000","last_undegraded":"2024-04-26T17:53:49.769191+0000","last_fullsized":"2024-04-26T17:53:49.769191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:49.850942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.349848+0000","last_change":"2024-04-26T17:53:03.614467+0000","last_active":"2024-04-26T17:54:48.349848+0000","last_peered":"2024-04-26T17:54:48.349848+0000","last_clean":"2024-04-26T17:54:48.349848+0000","last_became_active":"2024-04-26T17:53:03.613972+0000","last_became_peered":"2024-04-26T17:53:03.613972+0000","last_unstale":"2024-04-26T17:54:48.349848+0000","last_undegraded":"2024-04-26T17:54:48.349848+0000","last_fullsized":"2024-04-26T17:54:48.349848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:27.259777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.771037+0000","last_change":"2024-04-26T17:53:49.767174+0000","last_active":"2024-04-26T17:53:49.771037+0000","last_peered":"2024-04-26T17:53:49.771037+0000","last_clean":"2024-04-26T17:53:49.771037+0000","last_became_active":"2024-04-26T17:53:49.766778+0000","last_became_peered":"2024-04-26T17:53:49.766778+0000","last_unstale":"2024-04-26T17:53:49.771037+0000","last_undegraded":"2024-04-26T17:53:49.771037+0000","last_fullsized":"2024-04-26T17:53:49.771037+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:59.240159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.397305+0000","last_change":"2024-04-26T17:53:03.613888+0000","last_active":"2024-04-26T17:54:48.397305+0000","last_peered":"2024-04-26T17:54:48.397305+0000","last_clean":"2024-04-26T17:54:48.397305+0000","last_became_active":"2024-04-26T17:53:03.613499+0000","last_became_peered":"2024-04-26T17:53:03.613499+0000","last_unstale":"2024-04-26T17:54:48.397305+0000","last_undegraded":"2024-04-26T17:54:48.397305+0000","last_fullsized":"2024-04-26T17:54:48.397305+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:15.549419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767443+0000","last_change":"2024-04-26T17:53:00.587803+0000","last_active":"2024-04-26T17:53:49.767443+0000","last_peered":"2024-04-26T17:53:49.767443+0000","last_clean":"2024-04-26T17:53:49.767443+0000","last_became_active":"2024-04-26T17:53:00.587382+0000","last_became_peered":"2024-04-26T17:53:00.587382+0000","last_unstale":"2024-04-26T17:53:49.767443+0000","last_undegraded":"2024-04-26T17:53:49.767443+0000","last_fullsized":"2024-04-26T17:53:49.767443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:38.249478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.355737+0000","last_change":"2024-04-26T17:53:49.769008+0000","last_active":"2024-04-26T17:54:48.355737+0000","last_peered":"2024-04-26T17:54:48.355737+0000","last_clean":"2024-04-26T17:54:48.355737+0000","last_became_active":"2024-04-26T17:53:49.768625+0000","last_became_peered":"2024-04-26T17:53:49.768625+0000","last_unstale":"2024-04-26T17:54:48.355737+0000","last_undegraded":"2024-04-26T17:54:48.355737+0000","last_fullsized":"2024-04-26T17:54:48.355737+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:27:46.412625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.355983+0000","last_change":"2024-04-26T17:53:03.612899+0000","last_active":"2024-04-26T17:54:48.355983+0000","last_peered":"2024-04-26T17:54:48.355983+0000","last_clean":"2024-04-26T17:54:48.355983+0000","last_became_active":"2024-04-26T17:53:03.612564+0000","last_became_peered":"2024-04-26T17:53:03.612564+0000","last_unstale":"2024-04-26T17:54:48.355983+0000","last_undegraded":"2024-04-26T17:54:48.355983+0000","last_fullsized":"2024-04-26T17:54:48.355983+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:07.368256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":142,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:55:11.859166+0000","last_change":"2024-04-26T17:53:01.145471+0000","last_active":"2024-04-26T17:55:11.859166+0000","last_peered":"2024-04-26T17:55:11.859166+0000","last_clean":"2024-04-26T17:55:11.859166+0000","last_became_active":"2024-04-26T17:53:01.145040+0000","last_became_peered":"2024-04-26T17:53:01.145040+0000","last_unstale":"2024-04-26T17:55:11.859166+0000","last_undegraded":"2024-04-26T17:55:11.859166+0000","last_fullsized":"2024-04-26T17:55:11.859166+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:46:27.437329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.392882+0000","last_change":"2024-04-26T17:53:03.618600+0000","last_active":"2024-04-26T17:54:48.392882+0000","last_peered":"2024-04-26T17:54:48.392882+0000","last_clean":"2024-04-26T17:54:48.392882+0000","last_became_active":"2024-04-26T17:53:03.618139+0000","last_became_peered":"2024-04-26T17:53:03.618139+0000","last_unstale":"2024-04-26T17:54:48.392882+0000","last_undegraded":"2024-04-26T17:54:48.392882+0000","last_fullsized":"2024-04-26T17:54:48.392882+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:56.007659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891129+0000","last_change":"2024-04-26T17:53:00.586386+0000","last_active":"2024-04-26T17:53:49.891129+0000","last_peered":"2024-04-26T17:53:49.891129+0000","last_clean":"2024-04-26T17:53:49.891129+0000","last_became_active":"2024-04-26T17:53:00.585953+0000","last_became_peered":"2024-04-26T17:53:00.585953+0000","last_unstale":"2024-04-26T17:53:49.891129+0000","last_undegraded":"2024-04-26T17:53:49.891129+0000","last_fullsized":"2024-04-26T17:53:49.891129+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:39:57.160983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":358,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767311+0000","last_change":"2024-04-26T17:51:38.839853+0000","last_active":"2024-04-26T17:53:49.767311+0000","last_peered":"2024-04-26T17:53:49.767311+0000","last_clean":"2024-04-26T17:53:49.767311+0000","last_became_active":"2024-04-26T17:51:38.839318+0000","last_became_peered":"2024-04-26T17:51:38.839318+0000","last_unstale":"2024-04-26T17:53:49.767311+0000","last_undegraded":"2024-04-26T17:53:49.767311+0000","last_fullsized":"2024-04-26T17:53:49.767311+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_clean_scrub_stamp":"2024-04-26T17:48:12.577543+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:08:50.683663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.349869+0000","last_change":"2024-04-26T17:53:03.614170+0000","last_active":"2024-04-26T17:54:48.349869+0000","last_peered":"2024-04-26T17:54:48.349869+0000","last_clean":"2024-04-26T17:54:48.349869+0000","last_became_active":"2024-04-26T17:53:03.613836+0000","last_became_peered":"2024-04-26T17:53:03.613836+0000","last_unstale":"2024-04-26T17:54:48.349869+0000","last_undegraded":"2024-04-26T17:54:48.349869+0000","last_fullsized":"2024-04-26T17:54:48.349869+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:48.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767483+0000","last_change":"2024-04-26T17:53:00.587319+0000","last_active":"2024-04-26T17:53:49.767483+0000","last_peered":"2024-04-26T17:53:49.767483+0000","last_clean":"2024-04-26T17:53:49.767483+0000","last_became_active":"2024-04-26T17:53:00.586983+0000","last_became_peered":"2024-04-26T17:53:00.586983+0000","last_unstale":"2024-04-26T17:53:49.767483+0000","last_undegraded":"2024-04-26T17:53:49.767483+0000","last_fullsized":"2024-04-26T17:53:49.767483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:03:43.470013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768791+0000","last_change":"2024-04-26T17:53:01.145672+0000","last_active":"2024-04-26T17:53:49.768791+0000","last_peered":"2024-04-26T17:53:49.768791+0000","last_clean":"2024-04-26T17:53:49.768791+0000","last_became_active":"2024-04-26T17:53:01.145274+0000","last_became_peered":"2024-04-26T17:53:01.145274+0000","last_unstale":"2024-04-26T17:53:49.768791+0000","last_undegraded":"2024-04-26T17:53:49.768791+0000","last_fullsized":"2024-04-26T17:53:49.768791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:21.122352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.358944+0000","last_change":"2024-04-26T17:53:03.611232+0000","last_active":"2024-04-26T17:54:48.358944+0000","last_peered":"2024-04-26T17:54:48.358944+0000","last_clean":"2024-04-26T17:54:48.358944+0000","last_became_active":"2024-04-26T17:53:03.610905+0000","last_became_peered":"2024-04-26T17:53:03.610905+0000","last_unstale":"2024-04-26T17:54:48.358944+0000","last_undegraded":"2024-04-26T17:54:48.358944+0000","last_fullsized":"2024-04-26T17:54:48.358944+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:31.426237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'12","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.633241+0000","last_change":"2024-04-26T17:53:00.580220+0000","last_active":"2024-04-26T17:54:48.633241+0000","last_peered":"2024-04-26T17:54:48.633241+0000","last_clean":"2024-04-26T17:54:48.633241+0000","last_became_active":"2024-04-26T17:53:00.579818+0000","last_became_peered":"2024-04-26T17:53:00.579818+0000","last_unstale":"2024-04-26T17:54:48.633241+0000","last_undegraded":"2024-04-26T17:54:48.633241+0000","last_fullsized":"2024-04-26T17:54:48.633241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:15.260489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.397707+0000","last_change":"2024-04-26T17:53:03.610882+0000","last_active":"2024-04-26T17:54:48.397707+0000","last_peered":"2024-04-26T17:54:48.397707+0000","last_clean":"2024-04-26T17:54:48.397707+0000","last_became_active":"2024-04-26T17:53:03.609979+0000","last_became_peered":"2024-04-26T17:53:03.609979+0000","last_unstale":"2024-04-26T17:54:48.397707+0000","last_undegraded":"2024-04-26T17:54:48.397707+0000","last_fullsized":"2024-04-26T17:54:48.397707+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:33.167579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891614+0000","last_change":"2024-04-26T17:53:01.358555+0000","last_active":"2024-04-26T17:53:49.891614+0000","last_peered":"2024-04-26T17:53:49.891614+0000","last_clean":"2024-04-26T17:53:49.891614+0000","last_became_active":"2024-04-26T17:53:01.358138+0000","last_became_peered":"2024-04-26T17:53:01.358138+0000","last_unstale":"2024-04-26T17:53:49.891614+0000","last_undegraded":"2024-04-26T17:53:49.891614+0000","last_fullsized":"2024-04-26T17:53:49.891614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:14.886609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.399154+0000","last_change":"2024-04-26T17:53:03.618816+0000","last_active":"2024-04-26T17:54:48.399154+0000","last_peered":"2024-04-26T17:54:48.399154+0000","last_clean":"2024-04-26T17:54:48.399154+0000","last_became_active":"2024-04-26T17:53:03.618184+0000","last_became_peered":"2024-04-26T17:53:03.618184+0000","last_unstale":"2024-04-26T17:54:48.399154+0000","last_undegraded":"2024-04-26T17:54:48.399154+0000","last_fullsized":"2024-04-26T17:54:48.399154+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:07.514516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758666+0000","last_change":"2024-04-26T17:53:01.147081+0000","last_active":"2024-04-26T17:53:49.758666+0000","last_peered":"2024-04-26T17:53:49.758666+0000","last_clean":"2024-04-26T17:53:49.758666+0000","last_became_active":"2024-04-26T17:53:01.146698+0000","last_became_peered":"2024-04-26T17:53:01.146698+0000","last_unstale":"2024-04-26T17:53:49.758666+0000","last_undegraded":"2024-04-26T17:53:49.758666+0000","last_fullsized":"2024-04-26T17:53:49.758666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:44:19.927316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.316221+0000","last_change":"2024-04-26T17:53:03.612592+0000","last_active":"2024-04-26T17:54:48.316221+0000","last_peered":"2024-04-26T17:54:48.316221+0000","last_clean":"2024-04-26T17:54:48.316221+0000","last_became_active":"2024-04-26T17:53:03.612274+0000","last_became_peered":"2024-04-26T17:53:03.612274+0000","last_unstale":"2024-04-26T17:54:48.316221+0000","last_undegraded":"2024-04-26T17:54:48.316221+0000","last_fullsized":"2024-04-26T17:54:48.316221+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:30:48.461947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764888+0000","last_change":"2024-04-26T17:53:00.586430+0000","last_active":"2024-04-26T17:53:49.764888+0000","last_peered":"2024-04-26T17:53:49.764888+0000","last_clean":"2024-04-26T17:53:49.764888+0000","last_became_active":"2024-04-26T17:53:00.586004+0000","last_became_peered":"2024-04-26T17:53:00.586004+0000","last_unstale":"2024-04-26T17:53:49.764888+0000","last_undegraded":"2024-04-26T17:53:49.764888+0000","last_fullsized":"2024-04-26T17:53:49.764888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:47:18.224890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.347177+0000","last_change":"2024-04-26T17:53:03.623081+0000","last_active":"2024-04-26T17:54:48.347177+0000","last_peered":"2024-04-26T17:54:48.347177+0000","last_clean":"2024-04-26T17:54:48.347177+0000","last_became_active":"2024-04-26T17:53:03.622761+0000","last_became_peered":"2024-04-26T17:53:03.622761+0000","last_unstale":"2024-04-26T17:54:48.347177+0000","last_undegraded":"2024-04-26T17:54:48.347177+0000","last_fullsized":"2024-04-26T17:54:48.347177+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:39.520075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768898+0000","last_change":"2024-04-26T17:53:00.583013+0000","last_active":"2024-04-26T17:53:49.768898+0000","last_peered":"2024-04-26T17:53:49.768898+0000","last_clean":"2024-04-26T17:53:49.768898+0000","last_became_active":"2024-04-26T17:53:00.582653+0000","last_became_peered":"2024-04-26T17:53:00.582653+0000","last_unstale":"2024-04-26T17:53:49.768898+0000","last_undegraded":"2024-04-26T17:53:49.768898+0000","last_fullsized":"2024-04-26T17:53:49.768898+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:09.448621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.398074+0000","last_change":"2024-04-26T17:53:03.618232+0000","last_active":"2024-04-26T17:54:48.398074+0000","last_peered":"2024-04-26T17:54:48.398074+0000","last_clean":"2024-04-26T17:54:48.398074+0000","last_became_active":"2024-04-26T17:53:03.617906+0000","last_became_peered":"2024-04-26T17:53:03.617906+0000","last_unstale":"2024-04-26T17:54:48.398074+0000","last_undegraded":"2024-04-26T17:54:48.398074+0000","last_fullsized":"2024-04-26T17:54:48.398074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:17.357398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768301+0000","last_change":"2024-04-26T17:53:00.586565+0000","last_active":"2024-04-26T17:53:49.768301+0000","last_peered":"2024-04-26T17:53:49.768301+0000","last_clean":"2024-04-26T17:53:49.768301+0000","last_became_active":"2024-04-26T17:53:00.586161+0000","last_became_peered":"2024-04-26T17:53:00.586161+0000","last_unstale":"2024-04-26T17:53:49.768301+0000","last_undegraded":"2024-04-26T17:53:49.768301+0000","last_fullsized":"2024-04-26T17:53:49.768301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:56:25.294055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.363184+0000","last_change":"2024-04-26T17:53:03.620340+0000","last_active":"2024-04-26T17:54:48.363184+0000","last_peered":"2024-04-26T17:54:48.363184+0000","last_clean":"2024-04-26T17:54:48.363184+0000","last_became_active":"2024-04-26T17:53:03.619897+0000","last_became_peered":"2024-04-26T17:53:03.619897+0000","last_unstale":"2024-04-26T17:54:48.363184+0000","last_undegraded":"2024-04-26T17:54:48.363184+0000","last_fullsized":"2024-04-26T17:54:48.363184+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:12:40.662070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769496+0000","last_change":"2024-04-26T17:53:01.358131+0000","last_active":"2024-04-26T17:53:49.769496+0000","last_peered":"2024-04-26T17:53:49.769496+0000","last_clean":"2024-04-26T17:53:49.769496+0000","last_became_active":"2024-04-26T17:53:01.357796+0000","last_became_peered":"2024-04-26T17:53:01.357796+0000","last_unstale":"2024-04-26T17:53:49.769496+0000","last_undegraded":"2024-04-26T17:53:49.769496+0000","last_fullsized":"2024-04-26T17:53:49.769496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:47.849873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.362259+0000","last_change":"2024-04-26T17:53:03.616133+0000","last_active":"2024-04-26T17:54:48.362259+0000","last_peered":"2024-04-26T17:54:48.362259+0000","last_clean":"2024-04-26T17:54:48.362259+0000","last_became_active":"2024-04-26T17:53:03.615797+0000","last_became_peered":"2024-04-26T17:53:03.615797+0000","last_unstale":"2024-04-26T17:54:48.362259+0000","last_undegraded":"2024-04-26T17:54:48.362259+0000","last_fullsized":"2024-04-26T17:54:48.362259+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:05.698489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759297+0000","last_change":"2024-04-26T17:53:00.587696+0000","last_active":"2024-04-26T17:53:49.759297+0000","last_peered":"2024-04-26T17:53:49.759297+0000","last_clean":"2024-04-26T17:53:49.759297+0000","last_became_active":"2024-04-26T17:53:00.587305+0000","last_became_peered":"2024-04-26T17:53:00.587305+0000","last_unstale":"2024-04-26T17:53:49.759297+0000","last_undegraded":"2024-04-26T17:53:49.759297+0000","last_fullsized":"2024-04-26T17:53:49.759297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:39.185753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.397406+0000","last_change":"2024-04-26T17:53:03.613076+0000","last_active":"2024-04-26T17:54:48.397406+0000","last_peered":"2024-04-26T17:54:48.397406+0000","last_clean":"2024-04-26T17:54:48.397406+0000","last_became_active":"2024-04-26T17:53:03.612746+0000","last_became_peered":"2024-04-26T17:53:03.612746+0000","last_unstale":"2024-04-26T17:54:48.397406+0000","last_undegraded":"2024-04-26T17:54:48.397406+0000","last_fullsized":"2024-04-26T17:54:48.397406+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:20.920154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760620+0000","last_change":"2024-04-26T17:53:00.582575+0000","last_active":"2024-04-26T17:53:49.760620+0000","last_peered":"2024-04-26T17:53:49.760620+0000","last_clean":"2024-04-26T17:53:49.760620+0000","last_became_active":"2024-04-26T17:53:00.582215+0000","last_became_peered":"2024-04-26T17:53:00.582215+0000","last_unstale":"2024-04-26T17:53:49.760620+0000","last_undegraded":"2024-04-26T17:53:49.760620+0000","last_fullsized":"2024-04-26T17:53:49.760620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:35:12.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.351391+0000","last_change":"2024-04-26T17:53:03.616269+0000","last_active":"2024-04-26T17:54:48.351391+0000","last_peered":"2024-04-26T17:54:48.351391+0000","last_clean":"2024-04-26T17:54:48.351391+0000","last_became_active":"2024-04-26T17:53:03.615896+0000","last_became_peered":"2024-04-26T17:53:03.615896+0000","last_unstale":"2024-04-26T17:54:48.351391+0000","last_undegraded":"2024-04-26T17:54:48.351391+0000","last_fullsized":"2024-04-26T17:54:48.351391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:12.111392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.350985+0000","last_change":"2024-04-26T17:53:03.621589+0000","last_active":"2024-04-26T17:54:48.350985+0000","last_peered":"2024-04-26T17:54:48.350985+0000","last_clean":"2024-04-26T17:54:48.350985+0000","last_became_active":"2024-04-26T17:53:03.621211+0000","last_became_peered":"2024-04-26T17:53:03.621211+0000","last_unstale":"2024-04-26T17:54:48.350985+0000","last_undegraded":"2024-04-26T17:54:48.350985+0000","last_fullsized":"2024-04-26T17:54:48.350985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.487476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.351103+0000","last_change":"2024-04-26T17:53:03.624536+0000","last_active":"2024-04-26T17:54:48.351103+0000","last_peered":"2024-04-26T17:54:48.351103+0000","last_clean":"2024-04-26T17:54:48.351103+0000","last_became_active":"2024-04-26T17:53:03.624234+0000","last_became_peered":"2024-04-26T17:53:03.624234+0000","last_unstale":"2024-04-26T17:54:48.351103+0000","last_undegraded":"2024-04-26T17:54:48.351103+0000","last_fullsized":"2024-04-26T17:54:48.351103+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:31:36.322688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.373052+0000","last_change":"2024-04-26T17:53:03.620096+0000","last_active":"2024-04-26T17:54:48.373052+0000","last_peered":"2024-04-26T17:54:48.373052+0000","last_clean":"2024-04-26T17:54:48.373052+0000","last_became_active":"2024-04-26T17:53:03.619729+0000","last_became_peered":"2024-04-26T17:53:03.619729+0000","last_unstale":"2024-04-26T17:54:48.373052+0000","last_undegraded":"2024-04-26T17:54:48.373052+0000","last_fullsized":"2024-04-26T17:54:48.373052+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:17.987728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.387974+0000","last_change":"2024-04-26T17:53:49.768829+0000","last_active":"2024-04-26T17:54:48.387974+0000","last_peered":"2024-04-26T17:54:48.387974+0000","last_clean":"2024-04-26T17:54:48.387974+0000","last_became_active":"2024-04-26T17:53:49.768415+0000","last_became_peered":"2024-04-26T17:53:49.768415+0000","last_unstale":"2024-04-26T17:54:48.387974+0000","last_undegraded":"2024-04-26T17:54:48.387974+0000","last_fullsized":"2024-04-26T17:54:48.387974+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:57.364355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.386318+0000","last_change":"2024-04-26T17:53:03.619193+0000","last_active":"2024-04-26T17:54:48.386318+0000","last_peered":"2024-04-26T17:54:48.386318+0000","last_clean":"2024-04-26T17:54:48.386318+0000","last_became_active":"2024-04-26T17:53:03.618776+0000","last_became_peered":"2024-04-26T17:53:03.618776+0000","last_unstale":"2024-04-26T17:54:48.386318+0000","last_undegraded":"2024-04-26T17:54:48.386318+0000","last_fullsized":"2024-04-26T17:54:48.386318+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:07:24.811683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.349713+0000","last_change":"2024-04-26T17:53:03.621366+0000","last_active":"2024-04-26T17:54:48.349713+0000","last_peered":"2024-04-26T17:54:48.349713+0000","last_clean":"2024-04-26T17:54:48.349713+0000","last_became_active":"2024-04-26T17:53:03.621048+0000","last_became_peered":"2024-04-26T17:53:03.621048+0000","last_unstale":"2024-04-26T17:54:48.349713+0000","last_undegraded":"2024-04-26T17:54:48.349713+0000","last_fullsized":"2024-04-26T17:54:48.349713+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:31:06.484880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.335563+0000","last_change":"2024-04-26T17:53:03.619383+0000","last_active":"2024-04-26T17:54:48.335563+0000","last_peered":"2024-04-26T17:54:48.335563+0000","last_clean":"2024-04-26T17:54:48.335563+0000","last_became_active":"2024-04-26T17:53:03.619017+0000","last_became_peered":"2024-04-26T17:53:03.619017+0000","last_unstale":"2024-04-26T17:54:48.335563+0000","last_undegraded":"2024-04-26T17:54:48.335563+0000","last_fullsized":"2024-04-26T17:54:48.335563+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:20.091959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.391997+0000","last_change":"2024-04-26T17:53:49.770695+0000","last_active":"2024-04-26T17:54:48.391997+0000","last_peered":"2024-04-26T17:54:48.391997+0000","last_clean":"2024-04-26T17:54:48.391997+0000","last_became_active":"2024-04-26T17:53:49.770303+0000","last_became_peered":"2024-04-26T17:53:49.770303+0000","last_unstale":"2024-04-26T17:54:48.391997+0000","last_undegraded":"2024-04-26T17:54:48.391997+0000","last_fullsized":"2024-04-26T17:54:48.391997+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:22.830129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.359311+0000","last_change":"2024-04-26T17:53:03.605521+0000","last_active":"2024-04-26T17:54:48.359311+0000","last_peered":"2024-04-26T17:54:48.359311+0000","last_clean":"2024-04-26T17:54:48.359311+0000","last_became_active":"2024-04-26T17:53:03.605145+0000","last_became_peered":"2024-04-26T17:53:03.605145+0000","last_unstale":"2024-04-26T17:54:48.359311+0000","last_undegraded":"2024-04-26T17:54:48.359311+0000","last_fullsized":"2024-04-26T17:54:48.359311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:58:09.856021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.394012+0000","last_change":"2024-04-26T17:53:03.610866+0000","last_active":"2024-04-26T17:54:48.394012+0000","last_peered":"2024-04-26T17:54:48.394012+0000","last_clean":"2024-04-26T17:54:48.394012+0000","last_became_active":"2024-04-26T17:53:03.610326+0000","last_became_peered":"2024-04-26T17:53:03.610326+0000","last_unstale":"2024-04-26T17:54:48.394012+0000","last_undegraded":"2024-04-26T17:54:48.394012+0000","last_fullsized":"2024-04-26T17:54:48.394012+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:20.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.394333+0000","last_change":"2024-04-26T17:53:03.615336+0000","last_active":"2024-04-26T17:54:48.394333+0000","last_peered":"2024-04-26T17:54:48.394333+0000","last_clean":"2024-04-26T17:54:48.394333+0000","last_became_active":"2024-04-26T17:53:03.614447+0000","last_became_peered":"2024-04-26T17:53:03.614447+0000","last_unstale":"2024-04-26T17:54:48.394333+0000","last_undegraded":"2024-04-26T17:54:48.394333+0000","last_fullsized":"2024-04-26T17:54:48.394333+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:36:10.091929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.372822+0000","last_change":"2024-04-26T17:53:03.614863+0000","last_active":"2024-04-26T17:54:48.372822+0000","last_peered":"2024-04-26T17:54:48.372822+0000","last_clean":"2024-04-26T17:54:48.372822+0000","last_became_active":"2024-04-26T17:53:03.614520+0000","last_became_peered":"2024-04-26T17:53:03.614520+0000","last_unstale":"2024-04-26T17:54:48.372822+0000","last_undegraded":"2024-04-26T17:54:48.372822+0000","last_fullsized":"2024-04-26T17:54:48.372822+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:51.367785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.339955+0000","last_change":"2024-04-26T17:53:03.618521+0000","last_active":"2024-04-26T17:54:48.339955+0000","last_peered":"2024-04-26T17:54:48.339955+0000","last_clean":"2024-04-26T17:54:48.339955+0000","last_became_active":"2024-04-26T17:53:03.617658+0000","last_became_peered":"2024-04-26T17:53:03.617658+0000","last_unstale":"2024-04-26T17:54:48.339955+0000","last_undegraded":"2024-04-26T17:54:48.339955+0000","last_fullsized":"2024-04-26T17:54:48.339955+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:39.813098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280045,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66726,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":702,"num_read_kb":423,"num_write":172,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":200178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":479802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":1133867,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":480038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":479958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":1072029,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":479427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":541804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":480698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":543686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":479738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":479864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1071713,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T17:55:18.486 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T17:55:18.737 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:19.101 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:18 smithi031 ceph-mon[29164]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:55:19.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:18 smithi097 ceph-mon[32419]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:55:19.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:18 smithi186 ceph-mon[32797]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:55:20.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:19 smithi097 ceph-mon[32419]: pgmap v365: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:20.169 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:19 smithi031 ceph-mon[29164]: pgmap v365: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:20.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:19 smithi186 ceph-mon[32797]: pgmap v365: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:20.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.399+0000 7f7951aa4700 1 -- 172.21.15.31:0/89792910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f794c103470 msgr2=0x7f794c1038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:20.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.399+0000 7f7951aa4700 1 --2- 172.21.15.31:0/89792910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f794c103470 0x7f794c1038f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7940009990 tx=0x7f7940009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:20.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.399+0000 7f7951aa4700 1 -- 172.21.15.31:0/89792910 shutdown_connections 2024-04-26T17:55:20.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.399+0000 7f7951aa4700 1 --2- 172.21.15.31:0/89792910 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f794c1094c0 0x7f794c1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:20.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.399+0000 7f7951aa4700 1 --2- 172.21.15.31:0/89792910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f794c103e30 0x7f794c110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:20.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.399+0000 7f7951aa4700 1 --2- 172.21.15.31:0/89792910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f794c103470 0x7f794c1038f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:20.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.399+0000 7f7951aa4700 1 -- 172.21.15.31:0/89792910 >> 172.21.15.31:0/89792910 conn(0x7f794c07a580 msgr2=0x7f794c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:20.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.399+0000 7f7951aa4700 1 -- 172.21.15.31:0/89792910 shutdown_connections 2024-04-26T17:55:20.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.399+0000 7f7951aa4700 1 -- 172.21.15.31:0/89792910 wait complete. 2024-04-26T17:55:20.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.400+0000 7f7951aa4700 1 Processor -- start 2024-04-26T17:55:20.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.400+0000 7f7951aa4700 1 -- start start 2024-04-26T17:55:20.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f7951aa4700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f794c103470 0x7f794c19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:20.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f7951aa4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f794c103e30 0x7f794c19fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:20.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f7951aa4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f794c1094c0 0x7f794c199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:20.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f7951aa4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f794c113640 con 0x7f794c1094c0 2024-04-26T17:55:20.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f7951aa4700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f794c1134c0 con 0x7f794c103470 2024-04-26T17:55:20.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f7951aa4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f794c1137c0 con 0x7f794c103e30 2024-04-26T17:55:20.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f794bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f794c1094c0 0x7f794c199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:20.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f794bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f794c1094c0 0x7f794c199980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39780/0 (socket says 172.21.15.31:39780) 2024-04-26T17:55:20.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f794bfff700 1 -- 172.21.15.31:0/2009560007 learned_addr learned my addr 172.21.15.31:0/2009560007 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:20.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f794b7fe700 1 --2- 172.21.15.31:0/2009560007 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f794c103470 0x7f794c19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:20.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f794affd700 1 --2- 172.21.15.31:0/2009560007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f794c103e30 0x7f794c19fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:20.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f794bfff700 1 -- 172.21.15.31:0/2009560007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f794c103e30 msgr2=0x7f794c19fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:20.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f794bfff700 1 --2- 172.21.15.31:0/2009560007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f794c103e30 0x7f794c19fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:20.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f794bfff700 1 -- 172.21.15.31:0/2009560007 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f794c103470 msgr2=0x7f794c19f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:20.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f794bfff700 1 --2- 172.21.15.31:0/2009560007 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f794c103470 0x7f794c19f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:20.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f794bfff700 1 -- 172.21.15.31:0/2009560007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794c19a0e0 con 0x7f794c1094c0 2024-04-26T17:55:20.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f794bfff700 1 --2- 172.21.15.31:0/2009560007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f794c1094c0 0x7f794c199980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f794400b970 tx=0x7f794400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:20.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f7948ff9700 1 -- 172.21.15.31:0/2009560007 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7944019070 con 0x7f794c1094c0 2024-04-26T17:55:20.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f7948ff9700 1 -- 172.21.15.31:0/2009560007 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f794400cde0 con 0x7f794c1094c0 2024-04-26T17:55:20.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f7951aa4700 1 -- 172.21.15.31:0/2009560007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f794c19a3d0 con 0x7f794c1094c0 2024-04-26T17:55:20.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.401+0000 7f7948ff9700 1 -- 172.21.15.31:0/2009560007 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7944007370 con 0x7f794c1094c0 2024-04-26T17:55:20.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.402+0000 7f7951aa4700 1 -- 172.21.15.31:0/2009560007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f794c1a6690 con 0x7f794c1094c0 2024-04-26T17:55:20.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.403+0000 7f7951aa4700 1 -- 172.21.15.31:0/2009560007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f794c06a110 con 0x7f794c1094c0 2024-04-26T17:55:20.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.404+0000 7f7948ff9700 1 -- 172.21.15.31:0/2009560007 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f794401c070 con 0x7f794c1094c0 2024-04-26T17:55:20.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.404+0000 7f7948ff9700 1 --2- 172.21.15.31:0/2009560007 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f79340a3cc0 0x7f79340a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:20.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.404+0000 7f794b7fe700 1 --2- 172.21.15.31:0/2009560007 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f79340a3cc0 0x7f79340a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:20.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.404+0000 7f7948ff9700 1 -- 172.21.15.31:0/2009560007 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f79440cf7d0 con 0x7f794c1094c0 2024-04-26T17:55:20.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.405+0000 7f794b7fe700 1 --2- 172.21.15.31:0/2009560007 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f79340a3cc0 0x7f79340a6180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f794c19aac0 tx=0x7f793c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:20.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.409+0000 7f7948ff9700 1 -- 172.21.15.31:0/2009560007 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f794409c1e0 con 0x7f794c1094c0 2024-04-26T17:55:20.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.613+0000 7f7951aa4700 1 -- 172.21.15.31:0/2009560007 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f794c19aeb0 con 0x7f79340a3cc0 2024-04-26T17:55:20.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.621+0000 7f7948ff9700 1 -- 172.21.15.31:0/2009560007 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259002 (secure 0 0 0) 0x7f794c19aeb0 con 0x7f79340a3cc0 2024-04-26T17:55:20.625 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:55:20.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.624+0000 7f7951aa4700 1 -- 172.21.15.31:0/2009560007 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f79340a3cc0 msgr2=0x7f79340a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:20.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.624+0000 7f7951aa4700 1 --2- 172.21.15.31:0/2009560007 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f79340a3cc0 0x7f79340a6180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f794c19aac0 tx=0x7f793c00b410 comp rx=0 tx=0).stop 2024-04-26T17:55:20.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.624+0000 7f7951aa4700 1 -- 172.21.15.31:0/2009560007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f794c1094c0 msgr2=0x7f794c199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:20.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.624+0000 7f7951aa4700 1 --2- 172.21.15.31:0/2009560007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f794c1094c0 0x7f794c199980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f794400b970 tx=0x7f794400bc80 comp rx=0 tx=0).stop 2024-04-26T17:55:20.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.624+0000 7f7951aa4700 1 -- 172.21.15.31:0/2009560007 shutdown_connections 2024-04-26T17:55:20.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.624+0000 7f7951aa4700 1 --2- 172.21.15.31:0/2009560007 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f79340a3cc0 0x7f79340a6180 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:20.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.624+0000 7f7951aa4700 1 --2- 172.21.15.31:0/2009560007 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f794c103470 0x7f794c19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:20.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.624+0000 7f7951aa4700 1 --2- 172.21.15.31:0/2009560007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f794c1094c0 0x7f794c199980 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:20.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.624+0000 7f7951aa4700 1 --2- 172.21.15.31:0/2009560007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f794c103e30 0x7f794c19fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:20.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.624+0000 7f7951aa4700 1 -- 172.21.15.31:0/2009560007 >> 172.21.15.31:0/2009560007 conn(0x7f794c07a580 msgr2=0x7f794c1022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:20.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.624+0000 7f7951aa4700 1 -- 172.21.15.31:0/2009560007 shutdown_connections 2024-04-26T17:55:20.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:20.624+0000 7f7951aa4700 1 -- 172.21.15.31:0/2009560007 wait complete. 2024-04-26T17:55:20.628 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-26T17:55:21.402 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-04-26T17:55:20.341065+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657328,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":848,"num_read_kb":547,"num_write":355,"num_write_kb":3058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361164,"kb_used_data":12012,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580660,"statfs":{"total":1151940427776,"available":1151570595840,"internally_reserved":0,"allocated":12300288,"data_stored":7722624,"data_compressed":106863,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002912"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.349263+0000","last_change":"2024-04-26T17:53:03.616592+0000","last_active":"2024-04-26T17:54:48.349263+0000","last_peered":"2024-04-26T17:54:48.349263+0000","last_clean":"2024-04-26T17:54:48.349263+0000","last_became_active":"2024-04-26T17:53:03.616255+0000","last_became_peered":"2024-04-26T17:53:03.616255+0000","last_unstale":"2024-04-26T17:54:48.349263+0000","last_undegraded":"2024-04-26T17:54:48.349263+0000","last_fullsized":"2024-04-26T17:54:48.349263+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:43:31.183679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.380447+0000","last_change":"2024-04-26T17:53:03.615402+0000","last_active":"2024-04-26T17:54:48.380447+0000","last_peered":"2024-04-26T17:54:48.380447+0000","last_clean":"2024-04-26T17:54:48.380447+0000","last_became_active":"2024-04-26T17:53:03.615048+0000","last_became_peered":"2024-04-26T17:53:03.615048+0000","last_unstale":"2024-04-26T17:54:48.380447+0000","last_undegraded":"2024-04-26T17:54:48.380447+0000","last_fullsized":"2024-04-26T17:54:48.380447+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:05:12.454770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.386996+0000","last_change":"2024-04-26T17:53:03.622625+0000","last_active":"2024-04-26T17:54:48.386996+0000","last_peered":"2024-04-26T17:54:48.386996+0000","last_clean":"2024-04-26T17:54:48.386996+0000","last_became_active":"2024-04-26T17:53:03.622305+0000","last_became_peered":"2024-04-26T17:53:03.622305+0000","last_unstale":"2024-04-26T17:54:48.386996+0000","last_undegraded":"2024-04-26T17:54:48.386996+0000","last_fullsized":"2024-04-26T17:54:48.386996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:06.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.340614+0000","last_change":"2024-04-26T17:53:03.617650+0000","last_active":"2024-04-26T17:54:48.340614+0000","last_peered":"2024-04-26T17:54:48.340614+0000","last_clean":"2024-04-26T17:54:48.340614+0000","last_became_active":"2024-04-26T17:53:03.617220+0000","last_became_peered":"2024-04-26T17:53:03.617220+0000","last_unstale":"2024-04-26T17:54:48.340614+0000","last_undegraded":"2024-04-26T17:54:48.340614+0000","last_fullsized":"2024-04-26T17:54:48.340614+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:31.765577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.344350+0000","last_change":"2024-04-26T17:53:03.615429+0000","last_active":"2024-04-26T17:54:48.344350+0000","last_peered":"2024-04-26T17:54:48.344350+0000","last_clean":"2024-04-26T17:54:48.344350+0000","last_became_active":"2024-04-26T17:53:03.615079+0000","last_became_peered":"2024-04-26T17:53:03.615079+0000","last_unstale":"2024-04-26T17:54:48.344350+0000","last_undegraded":"2024-04-26T17:54:48.344350+0000","last_fullsized":"2024-04-26T17:54:48.344350+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:20.994016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.355757+0000","last_change":"2024-04-26T17:53:03.624619+0000","last_active":"2024-04-26T17:54:48.355757+0000","last_peered":"2024-04-26T17:54:48.355757+0000","last_clean":"2024-04-26T17:54:48.355757+0000","last_became_active":"2024-04-26T17:53:03.624251+0000","last_became_peered":"2024-04-26T17:53:03.624251+0000","last_unstale":"2024-04-26T17:54:48.355757+0000","last_undegraded":"2024-04-26T17:54:48.355757+0000","last_fullsized":"2024-04-26T17:54:48.355757+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:50:17.535235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.352676+0000","last_change":"2024-04-26T17:53:03.619325+0000","last_active":"2024-04-26T17:54:48.352676+0000","last_peered":"2024-04-26T17:54:48.352676+0000","last_clean":"2024-04-26T17:54:48.352676+0000","last_became_active":"2024-04-26T17:53:03.618989+0000","last_became_peered":"2024-04-26T17:53:03.618989+0000","last_unstale":"2024-04-26T17:54:48.352676+0000","last_undegraded":"2024-04-26T17:54:48.352676+0000","last_fullsized":"2024-04-26T17:54:48.352676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:10.195742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.396614+0000","last_change":"2024-04-26T17:53:03.609346+0000","last_active":"2024-04-26T17:54:48.396614+0000","last_peered":"2024-04-26T17:54:48.396614+0000","last_clean":"2024-04-26T17:54:48.396614+0000","last_became_active":"2024-04-26T17:53:03.609021+0000","last_became_peered":"2024-04-26T17:53:03.609021+0000","last_unstale":"2024-04-26T17:54:48.396614+0000","last_undegraded":"2024-04-26T17:54:48.396614+0000","last_fullsized":"2024-04-26T17:54:48.396614+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:44.328599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.355179+0000","last_change":"2024-04-26T17:53:03.618573+0000","last_active":"2024-04-26T17:54:48.355179+0000","last_peered":"2024-04-26T17:54:48.355179+0000","last_clean":"2024-04-26T17:54:48.355179+0000","last_became_active":"2024-04-26T17:53:03.618223+0000","last_became_peered":"2024-04-26T17:53:03.618223+0000","last_unstale":"2024-04-26T17:54:48.355179+0000","last_undegraded":"2024-04-26T17:54:48.355179+0000","last_fullsized":"2024-04-26T17:54:48.355179+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:16.749205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.346295+0000","last_change":"2024-04-26T17:53:03.621530+0000","last_active":"2024-04-26T17:54:48.346295+0000","last_peered":"2024-04-26T17:54:48.346295+0000","last_clean":"2024-04-26T17:54:48.346295+0000","last_became_active":"2024-04-26T17:53:03.621174+0000","last_became_peered":"2024-04-26T17:53:03.621174+0000","last_unstale":"2024-04-26T17:54:48.346295+0000","last_undegraded":"2024-04-26T17:54:48.346295+0000","last_fullsized":"2024-04-26T17:54:48.346295+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:31.388232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.361254+0000","last_change":"2024-04-26T17:53:03.617682+0000","last_active":"2024-04-26T17:54:48.361254+0000","last_peered":"2024-04-26T17:54:48.361254+0000","last_clean":"2024-04-26T17:54:48.361254+0000","last_became_active":"2024-04-26T17:53:03.617348+0000","last_became_peered":"2024-04-26T17:53:03.617348+0000","last_unstale":"2024-04-26T17:54:48.361254+0000","last_undegraded":"2024-04-26T17:54:48.361254+0000","last_fullsized":"2024-04-26T17:54:48.361254+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:21.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.333063+0000","last_change":"2024-04-26T17:53:03.617955+0000","last_active":"2024-04-26T17:54:48.333063+0000","last_peered":"2024-04-26T17:54:48.333063+0000","last_clean":"2024-04-26T17:54:48.333063+0000","last_became_active":"2024-04-26T17:53:03.616765+0000","last_became_peered":"2024-04-26T17:53:03.616765+0000","last_unstale":"2024-04-26T17:54:48.333063+0000","last_undegraded":"2024-04-26T17:54:48.333063+0000","last_fullsized":"2024-04-26T17:54:48.333063+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:59.292277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.358178+0000","last_change":"2024-04-26T17:53:49.767807+0000","last_active":"2024-04-26T17:54:48.358178+0000","last_peered":"2024-04-26T17:54:48.358178+0000","last_clean":"2024-04-26T17:54:48.358178+0000","last_became_active":"2024-04-26T17:53:49.767404+0000","last_became_peered":"2024-04-26T17:53:49.767404+0000","last_unstale":"2024-04-26T17:54:48.358178+0000","last_undegraded":"2024-04-26T17:54:48.358178+0000","last_fullsized":"2024-04-26T17:54:48.358178+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:20:45.482580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.386973+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:54:48.386973+0000","last_peered":"2024-04-26T17:54:48.386973+0000","last_clean":"2024-04-26T17:54:48.386973+0000","last_became_active":"2024-04-26T17:53:03.616623+0000","last_became_peered":"2024-04-26T17:53:03.616623+0000","last_unstale":"2024-04-26T17:54:48.386973+0000","last_undegraded":"2024-04-26T17:54:48.386973+0000","last_fullsized":"2024-04-26T17:54:48.386973+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:04.231549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.387143+0000","last_change":"2024-04-26T17:53:03.622147+0000","last_active":"2024-04-26T17:54:48.387143+0000","last_peered":"2024-04-26T17:54:48.387143+0000","last_clean":"2024-04-26T17:54:48.387143+0000","last_became_active":"2024-04-26T17:53:03.621542+0000","last_became_peered":"2024-04-26T17:53:03.621542+0000","last_unstale":"2024-04-26T17:54:48.387143+0000","last_undegraded":"2024-04-26T17:54:48.387143+0000","last_fullsized":"2024-04-26T17:54:48.387143+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:01.746954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.385788+0000","last_change":"2024-04-26T17:53:03.624433+0000","last_active":"2024-04-26T17:54:48.385788+0000","last_peered":"2024-04-26T17:54:48.385788+0000","last_clean":"2024-04-26T17:54:48.385788+0000","last_became_active":"2024-04-26T17:53:03.624135+0000","last_became_peered":"2024-04-26T17:53:03.624135+0000","last_unstale":"2024-04-26T17:54:48.385788+0000","last_undegraded":"2024-04-26T17:54:48.385788+0000","last_fullsized":"2024-04-26T17:54:48.385788+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:32:20.521917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.348971+0000","last_change":"2024-04-26T17:53:03.623607+0000","last_active":"2024-04-26T17:54:48.348971+0000","last_peered":"2024-04-26T17:54:48.348971+0000","last_clean":"2024-04-26T17:54:48.348971+0000","last_became_active":"2024-04-26T17:53:03.621783+0000","last_became_peered":"2024-04-26T17:53:03.621783+0000","last_unstale":"2024-04-26T17:54:48.348971+0000","last_undegraded":"2024-04-26T17:54:48.348971+0000","last_fullsized":"2024-04-26T17:54:48.348971+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:38:59.872868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.359683+0000","last_change":"2024-04-26T17:53:03.614974+0000","last_active":"2024-04-26T17:54:48.359683+0000","last_peered":"2024-04-26T17:54:48.359683+0000","last_clean":"2024-04-26T17:54:48.359683+0000","last_became_active":"2024-04-26T17:53:03.614619+0000","last_became_peered":"2024-04-26T17:53:03.614619+0000","last_unstale":"2024-04-26T17:54:48.359683+0000","last_undegraded":"2024-04-26T17:54:48.359683+0000","last_fullsized":"2024-04-26T17:54:48.359683+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:50.729531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.373272+0000","last_change":"2024-04-26T17:53:03.611140+0000","last_active":"2024-04-26T17:54:48.373272+0000","last_peered":"2024-04-26T17:54:48.373272+0000","last_clean":"2024-04-26T17:54:48.373272+0000","last_became_active":"2024-04-26T17:53:03.610549+0000","last_became_peered":"2024-04-26T17:53:03.610549+0000","last_unstale":"2024-04-26T17:54:48.373272+0000","last_undegraded":"2024-04-26T17:54:48.373272+0000","last_fullsized":"2024-04-26T17:54:48.373272+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:46:17.178598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.761876+0000","last_change":"2024-04-26T17:53:00.585971+0000","last_active":"2024-04-26T17:53:49.761876+0000","last_peered":"2024-04-26T17:53:49.761876+0000","last_clean":"2024-04-26T17:53:49.761876+0000","last_became_active":"2024-04-26T17:53:00.585648+0000","last_became_peered":"2024-04-26T17:53:00.585648+0000","last_unstale":"2024-04-26T17:53:49.761876+0000","last_undegraded":"2024-04-26T17:53:49.761876+0000","last_fullsized":"2024-04-26T17:53:49.761876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:00:24.082773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.358423+0000","last_change":"2024-04-26T17:53:49.770579+0000","last_active":"2024-04-26T17:54:48.358423+0000","last_peered":"2024-04-26T17:54:48.358423+0000","last_clean":"2024-04-26T17:54:48.358423+0000","last_became_active":"2024-04-26T17:53:49.770250+0000","last_became_peered":"2024-04-26T17:53:49.770250+0000","last_unstale":"2024-04-26T17:54:48.358423+0000","last_undegraded":"2024-04-26T17:54:48.358423+0000","last_fullsized":"2024-04-26T17:54:48.358423+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:35.758790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.766078+0000","last_change":"2024-04-26T17:53:01.145438+0000","last_active":"2024-04-26T17:53:49.766078+0000","last_peered":"2024-04-26T17:53:49.766078+0000","last_clean":"2024-04-26T17:53:49.766078+0000","last_became_active":"2024-04-26T17:53:01.145030+0000","last_became_peered":"2024-04-26T17:53:01.145030+0000","last_unstale":"2024-04-26T17:53:49.766078+0000","last_undegraded":"2024-04-26T17:53:49.766078+0000","last_fullsized":"2024-04-26T17:53:49.766078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:15.650247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.387694+0000","last_change":"2024-04-26T17:53:03.622647+0000","last_active":"2024-04-26T17:54:48.387694+0000","last_peered":"2024-04-26T17:54:48.387694+0000","last_clean":"2024-04-26T17:54:48.387694+0000","last_became_active":"2024-04-26T17:53:03.622298+0000","last_became_peered":"2024-04-26T17:53:03.622298+0000","last_unstale":"2024-04-26T17:54:48.387694+0000","last_undegraded":"2024-04-26T17:54:48.387694+0000","last_fullsized":"2024-04-26T17:54:48.387694+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:37.230539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.775803+0000","last_change":"2024-04-26T17:53:49.775405+0000","last_active":"2024-04-26T17:53:49.775803+0000","last_peered":"2024-04-26T17:53:49.775803+0000","last_clean":"2024-04-26T17:53:49.775803+0000","last_became_active":"2024-04-26T17:53:49.770536+0000","last_became_peered":"2024-04-26T17:53:49.770536+0000","last_unstale":"2024-04-26T17:53:49.775803+0000","last_undegraded":"2024-04-26T17:53:49.775803+0000","last_fullsized":"2024-04-26T17:53:49.775803+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:34.643452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.362560+0000","last_change":"2024-04-26T17:53:03.617462+0000","last_active":"2024-04-26T17:54:48.362560+0000","last_peered":"2024-04-26T17:54:48.362560+0000","last_clean":"2024-04-26T17:54:48.362560+0000","last_became_active":"2024-04-26T17:53:03.617115+0000","last_became_peered":"2024-04-26T17:53:03.617115+0000","last_unstale":"2024-04-26T17:54:48.362560+0000","last_undegraded":"2024-04-26T17:54:48.362560+0000","last_fullsized":"2024-04-26T17:54:48.362560+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:11.406146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767810+0000","last_change":"2024-04-26T17:53:00.583378+0000","last_active":"2024-04-26T17:53:49.767810+0000","last_peered":"2024-04-26T17:53:49.767810+0000","last_clean":"2024-04-26T17:53:49.767810+0000","last_became_active":"2024-04-26T17:53:00.583020+0000","last_became_peered":"2024-04-26T17:53:00.583020+0000","last_unstale":"2024-04-26T17:53:49.767810+0000","last_undegraded":"2024-04-26T17:53:49.767810+0000","last_fullsized":"2024-04-26T17:53:49.767810+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:24.089996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.356230+0000","last_change":"2024-04-26T17:53:03.609557+0000","last_active":"2024-04-26T17:54:48.356230+0000","last_peered":"2024-04-26T17:54:48.356230+0000","last_clean":"2024-04-26T17:54:48.356230+0000","last_became_active":"2024-04-26T17:53:03.609028+0000","last_became_peered":"2024-04-26T17:53:03.609028+0000","last_unstale":"2024-04-26T17:54:48.356230+0000","last_undegraded":"2024-04-26T17:54:48.356230+0000","last_fullsized":"2024-04-26T17:54:48.356230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:39:22.860877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":150,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:55:13.864356+0000","last_change":"2024-04-26T17:53:00.586683+0000","last_active":"2024-04-26T17:55:13.864356+0000","last_peered":"2024-04-26T17:55:13.864356+0000","last_clean":"2024-04-26T17:55:13.864356+0000","last_became_active":"2024-04-26T17:53:00.586272+0000","last_became_peered":"2024-04-26T17:53:00.586272+0000","last_unstale":"2024-04-26T17:55:13.864356+0000","last_undegraded":"2024-04-26T17:55:13.864356+0000","last_fullsized":"2024-04-26T17:55:13.864356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:20:29.068607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.352410+0000","last_change":"2024-04-26T17:53:03.611073+0000","last_active":"2024-04-26T17:54:48.352410+0000","last_peered":"2024-04-26T17:54:48.352410+0000","last_clean":"2024-04-26T17:54:48.352410+0000","last_became_active":"2024-04-26T17:53:03.610701+0000","last_became_peered":"2024-04-26T17:53:03.610701+0000","last_unstale":"2024-04-26T17:54:48.352410+0000","last_undegraded":"2024-04-26T17:54:48.352410+0000","last_fullsized":"2024-04-26T17:54:48.352410+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:01:00.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764638+0000","last_change":"2024-04-26T17:53:00.586082+0000","last_active":"2024-04-26T17:53:49.764638+0000","last_peered":"2024-04-26T17:53:49.764638+0000","last_clean":"2024-04-26T17:53:49.764638+0000","last_became_active":"2024-04-26T17:53:00.585678+0000","last_became_peered":"2024-04-26T17:53:00.585678+0000","last_unstale":"2024-04-26T17:53:49.764638+0000","last_undegraded":"2024-04-26T17:53:49.764638+0000","last_fullsized":"2024-04-26T17:53:49.764638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:55:25.523846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.356370+0000","last_change":"2024-04-26T17:53:03.600230+0000","last_active":"2024-04-26T17:54:48.356370+0000","last_peered":"2024-04-26T17:54:48.356370+0000","last_clean":"2024-04-26T17:54:48.356370+0000","last_became_active":"2024-04-26T17:53:03.599713+0000","last_became_peered":"2024-04-26T17:53:03.599713+0000","last_unstale":"2024-04-26T17:54:48.356370+0000","last_undegraded":"2024-04-26T17:54:48.356370+0000","last_fullsized":"2024-04-26T17:54:48.356370+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:45.822561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760523+0000","last_change":"2024-04-26T17:53:00.584588+0000","last_active":"2024-04-26T17:53:49.760523+0000","last_peered":"2024-04-26T17:53:49.760523+0000","last_clean":"2024-04-26T17:53:49.760523+0000","last_became_active":"2024-04-26T17:53:00.584284+0000","last_became_peered":"2024-04-26T17:53:00.584284+0000","last_unstale":"2024-04-26T17:53:49.760523+0000","last_undegraded":"2024-04-26T17:53:49.760523+0000","last_fullsized":"2024-04-26T17:53:49.760523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:08.428904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.393973+0000","last_change":"2024-04-26T17:53:03.615237+0000","last_active":"2024-04-26T17:54:48.393973+0000","last_peered":"2024-04-26T17:54:48.393973+0000","last_clean":"2024-04-26T17:54:48.393973+0000","last_became_active":"2024-04-26T17:53:03.614704+0000","last_became_peered":"2024-04-26T17:53:03.614704+0000","last_unstale":"2024-04-26T17:54:48.393973+0000","last_undegraded":"2024-04-26T17:54:48.393973+0000","last_fullsized":"2024-04-26T17:54:48.393973+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:27.251839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758907+0000","last_change":"2024-04-26T17:53:00.589287+0000","last_active":"2024-04-26T17:53:49.758907+0000","last_peered":"2024-04-26T17:53:49.758907+0000","last_clean":"2024-04-26T17:53:49.758907+0000","last_became_active":"2024-04-26T17:53:00.588119+0000","last_became_peered":"2024-04-26T17:53:00.588119+0000","last_unstale":"2024-04-26T17:53:49.758907+0000","last_undegraded":"2024-04-26T17:53:49.758907+0000","last_fullsized":"2024-04-26T17:53:49.758907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:25.389777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.392842+0000","last_change":"2024-04-26T17:53:03.608061+0000","last_active":"2024-04-26T17:54:48.392842+0000","last_peered":"2024-04-26T17:54:48.392842+0000","last_clean":"2024-04-26T17:54:48.392842+0000","last_became_active":"2024-04-26T17:53:03.607624+0000","last_became_peered":"2024-04-26T17:53:03.607624+0000","last_unstale":"2024-04-26T17:54:48.392842+0000","last_undegraded":"2024-04-26T17:54:48.392842+0000","last_fullsized":"2024-04-26T17:54:48.392842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:15.111011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":143,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:55:13.863837+0000","last_change":"2024-04-26T17:53:00.588621+0000","last_active":"2024-04-26T17:55:13.863837+0000","last_peered":"2024-04-26T17:55:13.863837+0000","last_clean":"2024-04-26T17:55:13.863837+0000","last_became_active":"2024-04-26T17:53:00.587330+0000","last_became_peered":"2024-04-26T17:53:00.587330+0000","last_unstale":"2024-04-26T17:55:13.863837+0000","last_undegraded":"2024-04-26T17:55:13.863837+0000","last_fullsized":"2024-04-26T17:55:13.863837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:01:12.178610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.363628+0000","last_change":"2024-04-26T17:53:03.615723+0000","last_active":"2024-04-26T17:54:48.363628+0000","last_peered":"2024-04-26T17:54:48.363628+0000","last_clean":"2024-04-26T17:54:48.363628+0000","last_became_active":"2024-04-26T17:53:03.614931+0000","last_became_peered":"2024-04-26T17:53:03.614931+0000","last_unstale":"2024-04-26T17:54:48.363628+0000","last_undegraded":"2024-04-26T17:54:48.363628+0000","last_fullsized":"2024-04-26T17:54:48.363628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:09.663876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767693+0000","last_change":"2024-04-26T17:53:00.589232+0000","last_active":"2024-04-26T17:53:49.767693+0000","last_peered":"2024-04-26T17:53:49.767693+0000","last_clean":"2024-04-26T17:53:49.767693+0000","last_became_active":"2024-04-26T17:53:00.588863+0000","last_became_peered":"2024-04-26T17:53:00.588863+0000","last_unstale":"2024-04-26T17:53:49.767693+0000","last_undegraded":"2024-04-26T17:53:49.767693+0000","last_fullsized":"2024-04-26T17:53:49.767693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:14.360782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.386612+0000","last_change":"2024-04-26T17:53:03.624923+0000","last_active":"2024-04-26T17:54:48.386612+0000","last_peered":"2024-04-26T17:54:48.386612+0000","last_clean":"2024-04-26T17:54:48.386612+0000","last_became_active":"2024-04-26T17:53:03.624620+0000","last_became_peered":"2024-04-26T17:53:03.624620+0000","last_unstale":"2024-04-26T17:54:48.386612+0000","last_undegraded":"2024-04-26T17:54:48.386612+0000","last_fullsized":"2024-04-26T17:54:48.386612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:50:51.259563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759526+0000","last_change":"2024-04-26T17:53:01.146950+0000","last_active":"2024-04-26T17:53:49.759526+0000","last_peered":"2024-04-26T17:53:49.759526+0000","last_clean":"2024-04-26T17:53:49.759526+0000","last_became_active":"2024-04-26T17:53:01.146591+0000","last_became_peered":"2024-04-26T17:53:01.146591+0000","last_unstale":"2024-04-26T17:53:49.759526+0000","last_undegraded":"2024-04-26T17:53:49.759526+0000","last_fullsized":"2024-04-26T17:53:49.759526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:17.756660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.361614+0000","last_change":"2024-04-26T17:53:03.626944+0000","last_active":"2024-04-26T17:54:48.361614+0000","last_peered":"2024-04-26T17:54:48.361614+0000","last_clean":"2024-04-26T17:54:48.361614+0000","last_became_active":"2024-04-26T17:53:03.626594+0000","last_became_peered":"2024-04-26T17:53:03.626594+0000","last_unstale":"2024-04-26T17:54:48.361614+0000","last_undegraded":"2024-04-26T17:54:48.361614+0000","last_fullsized":"2024-04-26T17:54:48.361614+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:52:42.997251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'33","reported_seq":295,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:55:14.864787+0000","last_change":"2024-04-26T17:53:00.588353+0000","last_active":"2024-04-26T17:55:14.864787+0000","last_peered":"2024-04-26T17:55:14.864787+0000","last_clean":"2024-04-26T17:55:14.864787+0000","last_became_active":"2024-04-26T17:53:00.586351+0000","last_became_peered":"2024-04-26T17:53:00.586351+0000","last_unstale":"2024-04-26T17:55:14.864787+0000","last_undegraded":"2024-04-26T17:55:14.864787+0000","last_fullsized":"2024-04-26T17:55:14.864787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:41.660525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62584,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":236,"num_read_kb":179,"num_write":34,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.348671+0000","last_change":"2024-04-26T17:53:03.612052+0000","last_active":"2024-04-26T17:54:48.348671+0000","last_peered":"2024-04-26T17:54:48.348671+0000","last_clean":"2024-04-26T17:54:48.348671+0000","last_became_active":"2024-04-26T17:53:03.611733+0000","last_became_peered":"2024-04-26T17:53:03.611733+0000","last_unstale":"2024-04-26T17:54:48.348671+0000","last_undegraded":"2024-04-26T17:54:48.348671+0000","last_fullsized":"2024-04-26T17:54:48.348671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:05:23.958317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769171+0000","last_change":"2024-04-26T17:53:00.588284+0000","last_active":"2024-04-26T17:53:49.769171+0000","last_peered":"2024-04-26T17:53:49.769171+0000","last_clean":"2024-04-26T17:53:49.769171+0000","last_became_active":"2024-04-26T17:53:00.587942+0000","last_became_peered":"2024-04-26T17:53:00.587942+0000","last_unstale":"2024-04-26T17:53:49.769171+0000","last_undegraded":"2024-04-26T17:53:49.769171+0000","last_fullsized":"2024-04-26T17:53:49.769171+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:43.782971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.374913+0000","last_change":"2024-04-26T17:53:03.613092+0000","last_active":"2024-04-26T17:54:48.374913+0000","last_peered":"2024-04-26T17:54:48.374913+0000","last_clean":"2024-04-26T17:54:48.374913+0000","last_became_active":"2024-04-26T17:53:03.612773+0000","last_became_peered":"2024-04-26T17:53:03.612773+0000","last_unstale":"2024-04-26T17:54:48.374913+0000","last_undegraded":"2024-04-26T17:54:48.374913+0000","last_fullsized":"2024-04-26T17:54:48.374913+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:41:21.340375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759054+0000","last_change":"2024-04-26T17:53:00.588777+0000","last_active":"2024-04-26T17:53:49.759054+0000","last_peered":"2024-04-26T17:53:49.759054+0000","last_clean":"2024-04-26T17:53:49.759054+0000","last_became_active":"2024-04-26T17:53:00.588380+0000","last_became_peered":"2024-04-26T17:53:00.588380+0000","last_unstale":"2024-04-26T17:53:49.759054+0000","last_undegraded":"2024-04-26T17:53:49.759054+0000","last_fullsized":"2024-04-26T17:53:49.759054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:44.587381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.357814+0000","last_change":"2024-04-26T17:53:03.620344+0000","last_active":"2024-04-26T17:54:48.357814+0000","last_peered":"2024-04-26T17:54:48.357814+0000","last_clean":"2024-04-26T17:54:48.357814+0000","last_became_active":"2024-04-26T17:53:03.619944+0000","last_became_peered":"2024-04-26T17:53:03.619944+0000","last_unstale":"2024-04-26T17:54:48.357814+0000","last_undegraded":"2024-04-26T17:54:48.357814+0000","last_fullsized":"2024-04-26T17:54:48.357814+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:08:31.801342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":145,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:55:14.864167+0000","last_change":"2024-04-26T17:53:00.587733+0000","last_active":"2024-04-26T17:55:14.864167+0000","last_peered":"2024-04-26T17:55:14.864167+0000","last_clean":"2024-04-26T17:55:14.864167+0000","last_became_active":"2024-04-26T17:53:00.587373+0000","last_became_peered":"2024-04-26T17:53:00.587373+0000","last_unstale":"2024-04-26T17:55:14.864167+0000","last_undegraded":"2024-04-26T17:55:14.864167+0000","last_fullsized":"2024-04-26T17:55:14.864167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:10.978944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.352086+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:54:48.352086+0000","last_peered":"2024-04-26T17:54:48.352086+0000","last_clean":"2024-04-26T17:54:48.352086+0000","last_became_active":"2024-04-26T17:53:03.616588+0000","last_became_peered":"2024-04-26T17:53:03.616588+0000","last_unstale":"2024-04-26T17:54:48.352086+0000","last_undegraded":"2024-04-26T17:54:48.352086+0000","last_fullsized":"2024-04-26T17:54:48.352086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:46:38.419613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767233+0000","last_change":"2024-04-26T17:53:01.146922+0000","last_active":"2024-04-26T17:53:49.767233+0000","last_peered":"2024-04-26T17:53:49.767233+0000","last_clean":"2024-04-26T17:53:49.767233+0000","last_became_active":"2024-04-26T17:53:01.146520+0000","last_became_peered":"2024-04-26T17:53:01.146520+0000","last_unstale":"2024-04-26T17:53:49.767233+0000","last_undegraded":"2024-04-26T17:53:49.767233+0000","last_fullsized":"2024-04-26T17:53:49.767233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:12:49.238012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.350055+0000","last_change":"2024-04-26T17:53:03.608513+0000","last_active":"2024-04-26T17:54:48.350055+0000","last_peered":"2024-04-26T17:54:48.350055+0000","last_clean":"2024-04-26T17:54:48.350055+0000","last_became_active":"2024-04-26T17:53:03.608167+0000","last_became_peered":"2024-04-26T17:53:03.608167+0000","last_unstale":"2024-04-26T17:54:48.350055+0000","last_undegraded":"2024-04-26T17:54:48.350055+0000","last_fullsized":"2024-04-26T17:54:48.350055+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:03.002756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769240+0000","last_change":"2024-04-26T17:53:00.583878+0000","last_active":"2024-04-26T17:53:49.769240+0000","last_peered":"2024-04-26T17:53:49.769240+0000","last_clean":"2024-04-26T17:53:49.769240+0000","last_became_active":"2024-04-26T17:53:00.583159+0000","last_became_peered":"2024-04-26T17:53:00.583159+0000","last_unstale":"2024-04-26T17:53:49.769240+0000","last_undegraded":"2024-04-26T17:53:49.769240+0000","last_fullsized":"2024-04-26T17:53:49.769240+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:49.850942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.349848+0000","last_change":"2024-04-26T17:53:03.614467+0000","last_active":"2024-04-26T17:54:48.349848+0000","last_peered":"2024-04-26T17:54:48.349848+0000","last_clean":"2024-04-26T17:54:48.349848+0000","last_became_active":"2024-04-26T17:53:03.613972+0000","last_became_peered":"2024-04-26T17:53:03.613972+0000","last_unstale":"2024-04-26T17:54:48.349848+0000","last_undegraded":"2024-04-26T17:54:48.349848+0000","last_fullsized":"2024-04-26T17:54:48.349848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:27.259777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.771037+0000","last_change":"2024-04-26T17:53:49.767174+0000","last_active":"2024-04-26T17:53:49.771037+0000","last_peered":"2024-04-26T17:53:49.771037+0000","last_clean":"2024-04-26T17:53:49.771037+0000","last_became_active":"2024-04-26T17:53:49.766778+0000","last_became_peered":"2024-04-26T17:53:49.766778+0000","last_unstale":"2024-04-26T17:53:49.771037+0000","last_undegraded":"2024-04-26T17:53:49.771037+0000","last_fullsized":"2024-04-26T17:53:49.771037+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:59.240159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.397305+0000","last_change":"2024-04-26T17:53:03.613888+0000","last_active":"2024-04-26T17:54:48.397305+0000","last_peered":"2024-04-26T17:54:48.397305+0000","last_clean":"2024-04-26T17:54:48.397305+0000","last_became_active":"2024-04-26T17:53:03.613499+0000","last_became_peered":"2024-04-26T17:53:03.613499+0000","last_unstale":"2024-04-26T17:54:48.397305+0000","last_undegraded":"2024-04-26T17:54:48.397305+0000","last_fullsized":"2024-04-26T17:54:48.397305+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:15.549419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767443+0000","last_change":"2024-04-26T17:53:00.587803+0000","last_active":"2024-04-26T17:53:49.767443+0000","last_peered":"2024-04-26T17:53:49.767443+0000","last_clean":"2024-04-26T17:53:49.767443+0000","last_became_active":"2024-04-26T17:53:00.587382+0000","last_became_peered":"2024-04-26T17:53:00.587382+0000","last_unstale":"2024-04-26T17:53:49.767443+0000","last_undegraded":"2024-04-26T17:53:49.767443+0000","last_fullsized":"2024-04-26T17:53:49.767443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:38.249478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.355737+0000","last_change":"2024-04-26T17:53:49.769008+0000","last_active":"2024-04-26T17:54:48.355737+0000","last_peered":"2024-04-26T17:54:48.355737+0000","last_clean":"2024-04-26T17:54:48.355737+0000","last_became_active":"2024-04-26T17:53:49.768625+0000","last_became_peered":"2024-04-26T17:53:49.768625+0000","last_unstale":"2024-04-26T17:54:48.355737+0000","last_undegraded":"2024-04-26T17:54:48.355737+0000","last_fullsized":"2024-04-26T17:54:48.355737+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:27:46.412625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.355983+0000","last_change":"2024-04-26T17:53:03.612899+0000","last_active":"2024-04-26T17:54:48.355983+0000","last_peered":"2024-04-26T17:54:48.355983+0000","last_clean":"2024-04-26T17:54:48.355983+0000","last_became_active":"2024-04-26T17:53:03.612564+0000","last_became_peered":"2024-04-26T17:53:03.612564+0000","last_unstale":"2024-04-26T17:54:48.355983+0000","last_undegraded":"2024-04-26T17:54:48.355983+0000","last_fullsized":"2024-04-26T17:54:48.355983+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:07.368256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":147,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:55:16.866796+0000","last_change":"2024-04-26T17:53:01.145471+0000","last_active":"2024-04-26T17:55:16.866796+0000","last_peered":"2024-04-26T17:55:16.866796+0000","last_clean":"2024-04-26T17:55:16.866796+0000","last_became_active":"2024-04-26T17:53:01.145040+0000","last_became_peered":"2024-04-26T17:53:01.145040+0000","last_unstale":"2024-04-26T17:55:16.866796+0000","last_undegraded":"2024-04-26T17:55:16.866796+0000","last_fullsized":"2024-04-26T17:55:16.866796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:46:27.437329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.392882+0000","last_change":"2024-04-26T17:53:03.618600+0000","last_active":"2024-04-26T17:54:48.392882+0000","last_peered":"2024-04-26T17:54:48.392882+0000","last_clean":"2024-04-26T17:54:48.392882+0000","last_became_active":"2024-04-26T17:53:03.618139+0000","last_became_peered":"2024-04-26T17:53:03.618139+0000","last_unstale":"2024-04-26T17:54:48.392882+0000","last_undegraded":"2024-04-26T17:54:48.392882+0000","last_fullsized":"2024-04-26T17:54:48.392882+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:56.007659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891129+0000","last_change":"2024-04-26T17:53:00.586386+0000","last_active":"2024-04-26T17:53:49.891129+0000","last_peered":"2024-04-26T17:53:49.891129+0000","last_clean":"2024-04-26T17:53:49.891129+0000","last_became_active":"2024-04-26T17:53:00.585953+0000","last_became_peered":"2024-04-26T17:53:00.585953+0000","last_unstale":"2024-04-26T17:53:49.891129+0000","last_undegraded":"2024-04-26T17:53:49.891129+0000","last_fullsized":"2024-04-26T17:53:49.891129+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:39:57.160983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":358,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767311+0000","last_change":"2024-04-26T17:51:38.839853+0000","last_active":"2024-04-26T17:53:49.767311+0000","last_peered":"2024-04-26T17:53:49.767311+0000","last_clean":"2024-04-26T17:53:49.767311+0000","last_became_active":"2024-04-26T17:51:38.839318+0000","last_became_peered":"2024-04-26T17:51:38.839318+0000","last_unstale":"2024-04-26T17:53:49.767311+0000","last_undegraded":"2024-04-26T17:53:49.767311+0000","last_fullsized":"2024-04-26T17:53:49.767311+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_clean_scrub_stamp":"2024-04-26T17:48:12.577543+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:08:50.683663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.349869+0000","last_change":"2024-04-26T17:53:03.614170+0000","last_active":"2024-04-26T17:54:48.349869+0000","last_peered":"2024-04-26T17:54:48.349869+0000","last_clean":"2024-04-26T17:54:48.349869+0000","last_became_active":"2024-04-26T17:53:03.613836+0000","last_became_peered":"2024-04-26T17:53:03.613836+0000","last_unstale":"2024-04-26T17:54:48.349869+0000","last_undegraded":"2024-04-26T17:54:48.349869+0000","last_fullsized":"2024-04-26T17:54:48.349869+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:48.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.767483+0000","last_change":"2024-04-26T17:53:00.587319+0000","last_active":"2024-04-26T17:53:49.767483+0000","last_peered":"2024-04-26T17:53:49.767483+0000","last_clean":"2024-04-26T17:53:49.767483+0000","last_became_active":"2024-04-26T17:53:00.586983+0000","last_became_peered":"2024-04-26T17:53:00.586983+0000","last_unstale":"2024-04-26T17:53:49.767483+0000","last_undegraded":"2024-04-26T17:53:49.767483+0000","last_fullsized":"2024-04-26T17:53:49.767483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:03:43.470013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768791+0000","last_change":"2024-04-26T17:53:01.145672+0000","last_active":"2024-04-26T17:53:49.768791+0000","last_peered":"2024-04-26T17:53:49.768791+0000","last_clean":"2024-04-26T17:53:49.768791+0000","last_became_active":"2024-04-26T17:53:01.145274+0000","last_became_peered":"2024-04-26T17:53:01.145274+0000","last_unstale":"2024-04-26T17:53:49.768791+0000","last_undegraded":"2024-04-26T17:53:49.768791+0000","last_fullsized":"2024-04-26T17:53:49.768791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:21.122352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.358944+0000","last_change":"2024-04-26T17:53:03.611232+0000","last_active":"2024-04-26T17:54:48.358944+0000","last_peered":"2024-04-26T17:54:48.358944+0000","last_clean":"2024-04-26T17:54:48.358944+0000","last_became_active":"2024-04-26T17:53:03.610905+0000","last_became_peered":"2024-04-26T17:53:03.610905+0000","last_unstale":"2024-04-26T17:54:48.358944+0000","last_undegraded":"2024-04-26T17:54:48.358944+0000","last_fullsized":"2024-04-26T17:54:48.358944+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:31.426237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'12","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.633241+0000","last_change":"2024-04-26T17:53:00.580220+0000","last_active":"2024-04-26T17:54:48.633241+0000","last_peered":"2024-04-26T17:54:48.633241+0000","last_clean":"2024-04-26T17:54:48.633241+0000","last_became_active":"2024-04-26T17:53:00.579818+0000","last_became_peered":"2024-04-26T17:53:00.579818+0000","last_unstale":"2024-04-26T17:54:48.633241+0000","last_undegraded":"2024-04-26T17:54:48.633241+0000","last_fullsized":"2024-04-26T17:54:48.633241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:15.260489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.397707+0000","last_change":"2024-04-26T17:53:03.610882+0000","last_active":"2024-04-26T17:54:48.397707+0000","last_peered":"2024-04-26T17:54:48.397707+0000","last_clean":"2024-04-26T17:54:48.397707+0000","last_became_active":"2024-04-26T17:53:03.609979+0000","last_became_peered":"2024-04-26T17:53:03.609979+0000","last_unstale":"2024-04-26T17:54:48.397707+0000","last_undegraded":"2024-04-26T17:54:48.397707+0000","last_fullsized":"2024-04-26T17:54:48.397707+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:33.167579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.891614+0000","last_change":"2024-04-26T17:53:01.358555+0000","last_active":"2024-04-26T17:53:49.891614+0000","last_peered":"2024-04-26T17:53:49.891614+0000","last_clean":"2024-04-26T17:53:49.891614+0000","last_became_active":"2024-04-26T17:53:01.358138+0000","last_became_peered":"2024-04-26T17:53:01.358138+0000","last_unstale":"2024-04-26T17:53:49.891614+0000","last_undegraded":"2024-04-26T17:53:49.891614+0000","last_fullsized":"2024-04-26T17:53:49.891614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:14.886609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.399154+0000","last_change":"2024-04-26T17:53:03.618816+0000","last_active":"2024-04-26T17:54:48.399154+0000","last_peered":"2024-04-26T17:54:48.399154+0000","last_clean":"2024-04-26T17:54:48.399154+0000","last_became_active":"2024-04-26T17:53:03.618184+0000","last_became_peered":"2024-04-26T17:53:03.618184+0000","last_unstale":"2024-04-26T17:54:48.399154+0000","last_undegraded":"2024-04-26T17:54:48.399154+0000","last_fullsized":"2024-04-26T17:54:48.399154+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:07.514516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.758666+0000","last_change":"2024-04-26T17:53:01.147081+0000","last_active":"2024-04-26T17:53:49.758666+0000","last_peered":"2024-04-26T17:53:49.758666+0000","last_clean":"2024-04-26T17:53:49.758666+0000","last_became_active":"2024-04-26T17:53:01.146698+0000","last_became_peered":"2024-04-26T17:53:01.146698+0000","last_unstale":"2024-04-26T17:53:49.758666+0000","last_undegraded":"2024-04-26T17:53:49.758666+0000","last_fullsized":"2024-04-26T17:53:49.758666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:44:19.927316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.339819+0000","last_change":"2024-04-26T17:53:03.612592+0000","last_active":"2024-04-26T17:54:48.339819+0000","last_peered":"2024-04-26T17:54:48.339819+0000","last_clean":"2024-04-26T17:54:48.339819+0000","last_became_active":"2024-04-26T17:53:03.612274+0000","last_became_peered":"2024-04-26T17:53:03.612274+0000","last_unstale":"2024-04-26T17:54:48.339819+0000","last_undegraded":"2024-04-26T17:54:48.339819+0000","last_fullsized":"2024-04-26T17:54:48.339819+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:30:48.461947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.764888+0000","last_change":"2024-04-26T17:53:00.586430+0000","last_active":"2024-04-26T17:53:49.764888+0000","last_peered":"2024-04-26T17:53:49.764888+0000","last_clean":"2024-04-26T17:53:49.764888+0000","last_became_active":"2024-04-26T17:53:00.586004+0000","last_became_peered":"2024-04-26T17:53:00.586004+0000","last_unstale":"2024-04-26T17:53:49.764888+0000","last_undegraded":"2024-04-26T17:53:49.764888+0000","last_fullsized":"2024-04-26T17:53:49.764888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:47:18.224890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.347426+0000","last_change":"2024-04-26T17:53:03.623081+0000","last_active":"2024-04-26T17:54:48.347426+0000","last_peered":"2024-04-26T17:54:48.347426+0000","last_clean":"2024-04-26T17:54:48.347426+0000","last_became_active":"2024-04-26T17:53:03.622761+0000","last_became_peered":"2024-04-26T17:53:03.622761+0000","last_unstale":"2024-04-26T17:54:48.347426+0000","last_undegraded":"2024-04-26T17:54:48.347426+0000","last_fullsized":"2024-04-26T17:54:48.347426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:39.520075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768898+0000","last_change":"2024-04-26T17:53:00.583013+0000","last_active":"2024-04-26T17:53:49.768898+0000","last_peered":"2024-04-26T17:53:49.768898+0000","last_clean":"2024-04-26T17:53:49.768898+0000","last_became_active":"2024-04-26T17:53:00.582653+0000","last_became_peered":"2024-04-26T17:53:00.582653+0000","last_unstale":"2024-04-26T17:53:49.768898+0000","last_undegraded":"2024-04-26T17:53:49.768898+0000","last_fullsized":"2024-04-26T17:53:49.768898+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:09.448621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.398074+0000","last_change":"2024-04-26T17:53:03.618232+0000","last_active":"2024-04-26T17:54:48.398074+0000","last_peered":"2024-04-26T17:54:48.398074+0000","last_clean":"2024-04-26T17:54:48.398074+0000","last_became_active":"2024-04-26T17:53:03.617906+0000","last_became_peered":"2024-04-26T17:53:03.617906+0000","last_unstale":"2024-04-26T17:54:48.398074+0000","last_undegraded":"2024-04-26T17:54:48.398074+0000","last_fullsized":"2024-04-26T17:54:48.398074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:17.357398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.768357+0000","last_change":"2024-04-26T17:53:00.586565+0000","last_active":"2024-04-26T17:53:49.768357+0000","last_peered":"2024-04-26T17:53:49.768357+0000","last_clean":"2024-04-26T17:53:49.768357+0000","last_became_active":"2024-04-26T17:53:00.586161+0000","last_became_peered":"2024-04-26T17:53:00.586161+0000","last_unstale":"2024-04-26T17:53:49.768357+0000","last_undegraded":"2024-04-26T17:53:49.768357+0000","last_fullsized":"2024-04-26T17:53:49.768357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:56:25.294055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.363298+0000","last_change":"2024-04-26T17:53:03.620340+0000","last_active":"2024-04-26T17:54:48.363298+0000","last_peered":"2024-04-26T17:54:48.363298+0000","last_clean":"2024-04-26T17:54:48.363298+0000","last_became_active":"2024-04-26T17:53:03.619897+0000","last_became_peered":"2024-04-26T17:53:03.619897+0000","last_unstale":"2024-04-26T17:54:48.363298+0000","last_undegraded":"2024-04-26T17:54:48.363298+0000","last_fullsized":"2024-04-26T17:54:48.363298+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:12:40.662070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.769556+0000","last_change":"2024-04-26T17:53:01.358131+0000","last_active":"2024-04-26T17:53:49.769556+0000","last_peered":"2024-04-26T17:53:49.769556+0000","last_clean":"2024-04-26T17:53:49.769556+0000","last_became_active":"2024-04-26T17:53:01.357796+0000","last_became_peered":"2024-04-26T17:53:01.357796+0000","last_unstale":"2024-04-26T17:53:49.769556+0000","last_undegraded":"2024-04-26T17:53:49.769556+0000","last_fullsized":"2024-04-26T17:53:49.769556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:47.849873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.362259+0000","last_change":"2024-04-26T17:53:03.616133+0000","last_active":"2024-04-26T17:54:48.362259+0000","last_peered":"2024-04-26T17:54:48.362259+0000","last_clean":"2024-04-26T17:54:48.362259+0000","last_became_active":"2024-04-26T17:53:03.615797+0000","last_became_peered":"2024-04-26T17:53:03.615797+0000","last_unstale":"2024-04-26T17:54:48.362259+0000","last_undegraded":"2024-04-26T17:54:48.362259+0000","last_fullsized":"2024-04-26T17:54:48.362259+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:05.698489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.759297+0000","last_change":"2024-04-26T17:53:00.587696+0000","last_active":"2024-04-26T17:53:49.759297+0000","last_peered":"2024-04-26T17:53:49.759297+0000","last_clean":"2024-04-26T17:53:49.759297+0000","last_became_active":"2024-04-26T17:53:00.587305+0000","last_became_peered":"2024-04-26T17:53:00.587305+0000","last_unstale":"2024-04-26T17:53:49.759297+0000","last_undegraded":"2024-04-26T17:53:49.759297+0000","last_fullsized":"2024-04-26T17:53:49.759297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:39.185753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.397406+0000","last_change":"2024-04-26T17:53:03.613076+0000","last_active":"2024-04-26T17:54:48.397406+0000","last_peered":"2024-04-26T17:54:48.397406+0000","last_clean":"2024-04-26T17:54:48.397406+0000","last_became_active":"2024-04-26T17:53:03.612746+0000","last_became_peered":"2024-04-26T17:53:03.612746+0000","last_unstale":"2024-04-26T17:54:48.397406+0000","last_undegraded":"2024-04-26T17:54:48.397406+0000","last_fullsized":"2024-04-26T17:54:48.397406+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:20.920154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:53:49.760669+0000","last_change":"2024-04-26T17:53:00.582575+0000","last_active":"2024-04-26T17:53:49.760669+0000","last_peered":"2024-04-26T17:53:49.760669+0000","last_clean":"2024-04-26T17:53:49.760669+0000","last_became_active":"2024-04-26T17:53:00.582215+0000","last_became_peered":"2024-04-26T17:53:00.582215+0000","last_unstale":"2024-04-26T17:53:49.760669+0000","last_undegraded":"2024-04-26T17:53:49.760669+0000","last_fullsized":"2024-04-26T17:53:49.760669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:35:12.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.351391+0000","last_change":"2024-04-26T17:53:03.616269+0000","last_active":"2024-04-26T17:54:48.351391+0000","last_peered":"2024-04-26T17:54:48.351391+0000","last_clean":"2024-04-26T17:54:48.351391+0000","last_became_active":"2024-04-26T17:53:03.615896+0000","last_became_peered":"2024-04-26T17:53:03.615896+0000","last_unstale":"2024-04-26T17:54:48.351391+0000","last_undegraded":"2024-04-26T17:54:48.351391+0000","last_fullsized":"2024-04-26T17:54:48.351391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:12.111392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.350985+0000","last_change":"2024-04-26T17:53:03.621589+0000","last_active":"2024-04-26T17:54:48.350985+0000","last_peered":"2024-04-26T17:54:48.350985+0000","last_clean":"2024-04-26T17:54:48.350985+0000","last_became_active":"2024-04-26T17:53:03.621211+0000","last_became_peered":"2024-04-26T17:53:03.621211+0000","last_unstale":"2024-04-26T17:54:48.350985+0000","last_undegraded":"2024-04-26T17:54:48.350985+0000","last_fullsized":"2024-04-26T17:54:48.350985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.487476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.351103+0000","last_change":"2024-04-26T17:53:03.624536+0000","last_active":"2024-04-26T17:54:48.351103+0000","last_peered":"2024-04-26T17:54:48.351103+0000","last_clean":"2024-04-26T17:54:48.351103+0000","last_became_active":"2024-04-26T17:53:03.624234+0000","last_became_peered":"2024-04-26T17:53:03.624234+0000","last_unstale":"2024-04-26T17:54:48.351103+0000","last_undegraded":"2024-04-26T17:54:48.351103+0000","last_fullsized":"2024-04-26T17:54:48.351103+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:31:36.322688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.373052+0000","last_change":"2024-04-26T17:53:03.620096+0000","last_active":"2024-04-26T17:54:48.373052+0000","last_peered":"2024-04-26T17:54:48.373052+0000","last_clean":"2024-04-26T17:54:48.373052+0000","last_became_active":"2024-04-26T17:53:03.619729+0000","last_became_peered":"2024-04-26T17:53:03.619729+0000","last_unstale":"2024-04-26T17:54:48.373052+0000","last_undegraded":"2024-04-26T17:54:48.373052+0000","last_fullsized":"2024-04-26T17:54:48.373052+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:17.987728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.387974+0000","last_change":"2024-04-26T17:53:49.768829+0000","last_active":"2024-04-26T17:54:48.387974+0000","last_peered":"2024-04-26T17:54:48.387974+0000","last_clean":"2024-04-26T17:54:48.387974+0000","last_became_active":"2024-04-26T17:53:49.768415+0000","last_became_peered":"2024-04-26T17:53:49.768415+0000","last_unstale":"2024-04-26T17:54:48.387974+0000","last_undegraded":"2024-04-26T17:54:48.387974+0000","last_fullsized":"2024-04-26T17:54:48.387974+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:57.364355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.386318+0000","last_change":"2024-04-26T17:53:03.619193+0000","last_active":"2024-04-26T17:54:48.386318+0000","last_peered":"2024-04-26T17:54:48.386318+0000","last_clean":"2024-04-26T17:54:48.386318+0000","last_became_active":"2024-04-26T17:53:03.618776+0000","last_became_peered":"2024-04-26T17:53:03.618776+0000","last_unstale":"2024-04-26T17:54:48.386318+0000","last_undegraded":"2024-04-26T17:54:48.386318+0000","last_fullsized":"2024-04-26T17:54:48.386318+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:07:24.811683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.349855+0000","last_change":"2024-04-26T17:53:03.621366+0000","last_active":"2024-04-26T17:54:48.349855+0000","last_peered":"2024-04-26T17:54:48.349855+0000","last_clean":"2024-04-26T17:54:48.349855+0000","last_became_active":"2024-04-26T17:53:03.621048+0000","last_became_peered":"2024-04-26T17:53:03.621048+0000","last_unstale":"2024-04-26T17:54:48.349855+0000","last_undegraded":"2024-04-26T17:54:48.349855+0000","last_fullsized":"2024-04-26T17:54:48.349855+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:31:06.484880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.336155+0000","last_change":"2024-04-26T17:53:03.619383+0000","last_active":"2024-04-26T17:54:48.336155+0000","last_peered":"2024-04-26T17:54:48.336155+0000","last_clean":"2024-04-26T17:54:48.336155+0000","last_became_active":"2024-04-26T17:53:03.619017+0000","last_became_peered":"2024-04-26T17:53:03.619017+0000","last_unstale":"2024-04-26T17:54:48.336155+0000","last_undegraded":"2024-04-26T17:54:48.336155+0000","last_fullsized":"2024-04-26T17:54:48.336155+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:20.091959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.391997+0000","last_change":"2024-04-26T17:53:49.770695+0000","last_active":"2024-04-26T17:54:48.391997+0000","last_peered":"2024-04-26T17:54:48.391997+0000","last_clean":"2024-04-26T17:54:48.391997+0000","last_became_active":"2024-04-26T17:53:49.770303+0000","last_became_peered":"2024-04-26T17:53:49.770303+0000","last_unstale":"2024-04-26T17:54:48.391997+0000","last_undegraded":"2024-04-26T17:54:48.391997+0000","last_fullsized":"2024-04-26T17:54:48.391997+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:22.830129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.359311+0000","last_change":"2024-04-26T17:53:03.605521+0000","last_active":"2024-04-26T17:54:48.359311+0000","last_peered":"2024-04-26T17:54:48.359311+0000","last_clean":"2024-04-26T17:54:48.359311+0000","last_became_active":"2024-04-26T17:53:03.605145+0000","last_became_peered":"2024-04-26T17:53:03.605145+0000","last_unstale":"2024-04-26T17:54:48.359311+0000","last_undegraded":"2024-04-26T17:54:48.359311+0000","last_fullsized":"2024-04-26T17:54:48.359311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:58:09.856021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.394012+0000","last_change":"2024-04-26T17:53:03.610866+0000","last_active":"2024-04-26T17:54:48.394012+0000","last_peered":"2024-04-26T17:54:48.394012+0000","last_clean":"2024-04-26T17:54:48.394012+0000","last_became_active":"2024-04-26T17:53:03.610326+0000","last_became_peered":"2024-04-26T17:53:03.610326+0000","last_unstale":"2024-04-26T17:54:48.394012+0000","last_undegraded":"2024-04-26T17:54:48.394012+0000","last_fullsized":"2024-04-26T17:54:48.394012+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:20.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.394434+0000","last_change":"2024-04-26T17:53:03.615336+0000","last_active":"2024-04-26T17:54:48.394434+0000","last_peered":"2024-04-26T17:54:48.394434+0000","last_clean":"2024-04-26T17:54:48.394434+0000","last_became_active":"2024-04-26T17:53:03.614447+0000","last_became_peered":"2024-04-26T17:53:03.614447+0000","last_unstale":"2024-04-26T17:54:48.394434+0000","last_undegraded":"2024-04-26T17:54:48.394434+0000","last_fullsized":"2024-04-26T17:54:48.394434+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:36:10.091929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.372980+0000","last_change":"2024-04-26T17:53:03.614863+0000","last_active":"2024-04-26T17:54:48.372980+0000","last_peered":"2024-04-26T17:54:48.372980+0000","last_clean":"2024-04-26T17:54:48.372980+0000","last_became_active":"2024-04-26T17:53:03.614520+0000","last_became_peered":"2024-04-26T17:53:03.614520+0000","last_unstale":"2024-04-26T17:54:48.372980+0000","last_undegraded":"2024-04-26T17:54:48.372980+0000","last_fullsized":"2024-04-26T17:54:48.372980+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:51.367785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T17:54:48.339955+0000","last_change":"2024-04-26T17:53:03.618521+0000","last_active":"2024-04-26T17:54:48.339955+0000","last_peered":"2024-04-26T17:54:48.339955+0000","last_clean":"2024-04-26T17:54:48.339955+0000","last_became_active":"2024-04-26T17:53:03.617658+0000","last_became_peered":"2024-04-26T17:53:03.617658+0000","last_unstale":"2024-04-26T17:54:48.339955+0000","last_undegraded":"2024-04-26T17:54:48.339955+0000","last_fullsized":"2024-04-26T17:54:48.339955+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:39.813098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280045,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66726,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":712,"num_read_kb":428,"num_write":172,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":200178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383723,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":479802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":1133867,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":480038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070521,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":479958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":1072029,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":479427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":541804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":480698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":543686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":479738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":479864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1071713,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33980,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T17:55:21.406 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T17:55:21.406 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T17:55:21.407 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-26T17:55:21.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-26T17:55:21.656 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:22.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:21 smithi031 ceph-mon[29164]: pgmap v366: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:22.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:21 smithi031 ceph-mon[29164]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:55:22.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:21 smithi097 ceph-mon[32419]: pgmap v366: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:22.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:21 smithi097 ceph-mon[32419]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:55:22.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:21 smithi186 ceph-mon[32797]: pgmap v366: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:22.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:21 smithi186 ceph-mon[32797]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:55:23.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.292+0000 7f4c55ca5700 1 -- 172.21.15.31:0/2444881465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c50103e30 msgr2=0x7f4c50110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:23.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.292+0000 7f4c55ca5700 1 --2- 172.21.15.31:0/2444881465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c50103e30 0x7f4c50110290 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800b0a0 tx=0x7f4c4800b3b0 comp rx=0 tx=0).stop 2024-04-26T17:55:23.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.292+0000 7f4c55ca5700 1 -- 172.21.15.31:0/2444881465 shutdown_connections 2024-04-26T17:55:23.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.292+0000 7f4c55ca5700 1 --2- 172.21.15.31:0/2444881465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c501094c0 0x7f4c501098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:23.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.292+0000 7f4c55ca5700 1 --2- 172.21.15.31:0/2444881465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c50103e30 0x7f4c50110290 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:23.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.292+0000 7f4c55ca5700 1 --2- 172.21.15.31:0/2444881465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c50103470 0x7f4c501038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:23.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.292+0000 7f4c55ca5700 1 -- 172.21.15.31:0/2444881465 >> 172.21.15.31:0/2444881465 conn(0x7f4c5007a580 msgr2=0x7f4c5007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:23.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.293+0000 7f4c55ca5700 1 -- 172.21.15.31:0/2444881465 shutdown_connections 2024-04-26T17:55:23.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.293+0000 7f4c55ca5700 1 -- 172.21.15.31:0/2444881465 wait complete. 2024-04-26T17:55:23.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.293+0000 7f4c55ca5700 1 Processor -- start 2024-04-26T17:55:23.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.293+0000 7f4c55ca5700 1 -- start start 2024-04-26T17:55:23.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c55ca5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c50103470 0x7f4c5019f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:23.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c55ca5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c50103e30 0x7f4c5019fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:23.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c55ca5700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c501094c0 0x7f4c501999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:23.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c55ca5700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4c50113690 con 0x7f4c50103470 2024-04-26T17:55:23.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c55ca5700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4c50113510 con 0x7f4c501094c0 2024-04-26T17:55:23.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c55ca5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4c50113810 con 0x7f4c50103e30 2024-04-26T17:55:23.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c4f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c50103470 0x7f4c5019f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:23.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c4f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c50103470 0x7f4c5019f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39796/0 (socket says 172.21.15.31:39796) 2024-04-26T17:55:23.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c4f7fe700 1 -- 172.21.15.31:0/3390402538 learned_addr learned my addr 172.21.15.31:0/3390402538 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:23.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c4ffff700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c501094c0 0x7f4c501999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:23.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c4effd700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c50103e30 0x7f4c5019fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:23.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c4f7fe700 1 -- 172.21.15.31:0/3390402538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c50103e30 msgr2=0x7f4c5019fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:23.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c4f7fe700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c50103e30 0x7f4c5019fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:23.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c4f7fe700 1 -- 172.21.15.31:0/3390402538 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c501094c0 msgr2=0x7f4c501999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:23.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c4f7fe700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c501094c0 0x7f4c501999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:23.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c4f7fe700 1 -- 172.21.15.31:0/3390402538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5019a110 con 0x7f4c50103470 2024-04-26T17:55:23.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c4effd700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c50103e30 0x7f4c5019fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:55:23.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c4f7fe700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c50103470 0x7f4c5019f7a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000c920 tx=0x7f4c4000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:23.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.294+0000 7f4c4cff9700 1 -- 172.21.15.31:0/3390402538 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c40007b00 con 0x7f4c50103470 2024-04-26T17:55:23.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.295+0000 7f4c4cff9700 1 -- 172.21.15.31:0/3390402538 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c40004ce0 con 0x7f4c50103470 2024-04-26T17:55:23.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.295+0000 7f4c55ca5700 1 -- 172.21.15.31:0/3390402538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5019a400 con 0x7f4c50103470 2024-04-26T17:55:23.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.295+0000 7f4c4cff9700 1 -- 172.21.15.31:0/3390402538 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c40005550 con 0x7f4c50103470 2024-04-26T17:55:23.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.295+0000 7f4c55ca5700 1 -- 172.21.15.31:0/3390402538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c501a6670 con 0x7f4c50103470 2024-04-26T17:55:23.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.296+0000 7f4c55ca5700 1 -- 172.21.15.31:0/3390402538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c5006a110 con 0x7f4c50103470 2024-04-26T17:55:23.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.297+0000 7f4c4cff9700 1 -- 172.21.15.31:0/3390402538 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4c40007ca0 con 0x7f4c50103470 2024-04-26T17:55:23.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.297+0000 7f4c4cff9700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f4c380a3c40 0x7f4c380a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:23.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.297+0000 7f4c4effd700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f4c380a3c40 0x7f4c380a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:23.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.297+0000 7f4c4cff9700 1 -- 172.21.15.31:0/3390402538 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f4c400d02c0 con 0x7f4c50103470 2024-04-26T17:55:23.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.298+0000 7f4c4effd700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f4c380a3c40 0x7f4c380a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4c5007ab80 tx=0x7f4c44008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:23.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.302+0000 7f4c4cff9700 1 -- 172.21.15.31:0/3390402538 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c40014030 con 0x7f4c50103470 2024-04-26T17:55:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.557+0000 7f4c55ca5700 1 -- 172.21.15.31:0/3390402538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f4c5019aec0 con 0x7f4c50103470 2024-04-26T17:55:23.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.557+0000 7f4c4cff9700 1 -- 172.21.15.31:0/3390402538 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f4c5019aec0 con 0x7f4c50103470 2024-04-26T17:55:23.558 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:55:23.558 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T17:55:23.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.560+0000 7f4c55ca5700 1 -- 172.21.15.31:0/3390402538 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f4c380a3c40 msgr2=0x7f4c380a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:23.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.560+0000 7f4c55ca5700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f4c380a3c40 0x7f4c380a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4c5007ab80 tx=0x7f4c44008040 comp rx=0 tx=0).stop 2024-04-26T17:55:23.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.560+0000 7f4c55ca5700 1 -- 172.21.15.31:0/3390402538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c50103470 msgr2=0x7f4c5019f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:23.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.560+0000 7f4c55ca5700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c50103470 0x7f4c5019f7a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000c920 tx=0x7f4c4000cc30 comp rx=0 tx=0).stop 2024-04-26T17:55:23.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.560+0000 7f4c55ca5700 1 -- 172.21.15.31:0/3390402538 shutdown_connections 2024-04-26T17:55:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.560+0000 7f4c55ca5700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f4c380a3c40 0x7f4c380a6100 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.560+0000 7f4c55ca5700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c501094c0 0x7f4c501999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.560+0000 7f4c55ca5700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c50103470 0x7f4c5019f7a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.560+0000 7f4c55ca5700 1 --2- 172.21.15.31:0/3390402538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c50103e30 0x7f4c5019fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.560+0000 7f4c55ca5700 1 -- 172.21.15.31:0/3390402538 >> 172.21.15.31:0/3390402538 conn(0x7f4c5007a580 msgr2=0x7f4c50102180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.560+0000 7f4c55ca5700 1 -- 172.21.15.31:0/3390402538 shutdown_connections 2024-04-26T17:55:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:23.560+0000 7f4c55ca5700 1 -- 172.21.15.31:0/3390402538 wait complete. 2024-04-26T17:55:24.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:23 smithi097 ceph-mon[32419]: pgmap v367: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:24.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:23 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3390402538' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T17:55:24.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:23 smithi031 ceph-mon[29164]: pgmap v367: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:24.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:23 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3390402538' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T17:55:24.266 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-26T17:55:24.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:55:24.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:23 smithi186 ceph-mon[32797]: pgmap v367: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:24.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:23 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3390402538' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T17:55:24.522 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:25.941 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:25 smithi031 ceph-mon[29164]: pgmap v368: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:26.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:25 smithi097 ceph-mon[32419]: pgmap v368: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:26.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.185+0000 7f3550bc1700 1 -- 172.21.15.31:0/2435083130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f354c075650 msgr2=0x7f354c075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:26.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.185+0000 7f3550bc1700 1 --2- 172.21.15.31:0/2435083130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f354c075650 0x7f354c075a30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3540009990 tx=0x7f3540009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:26.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.185+0000 7f3550bc1700 1 -- 172.21.15.31:0/2435083130 shutdown_connections 2024-04-26T17:55:26.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.185+0000 7f3550bc1700 1 --2- 172.21.15.31:0/2435083130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f354c10fe20 0x7f354c112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:26.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.185+0000 7f3550bc1700 1 --2- 172.21.15.31:0/2435083130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f354c075650 0x7f354c075a30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:26.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.185+0000 7f3550bc1700 1 --2- 172.21.15.31:0/2435083130 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f354c075f70 0x7f354c10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:26.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.185+0000 7f3550bc1700 1 -- 172.21.15.31:0/2435083130 >> 172.21.15.31:0/2435083130 conn(0x7f354c07a580 msgr2=0x7f354c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:26.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.185+0000 7f3550bc1700 1 -- 172.21.15.31:0/2435083130 shutdown_connections 2024-04-26T17:55:26.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.185+0000 7f3550bc1700 1 -- 172.21.15.31:0/2435083130 wait complete. 2024-04-26T17:55:26.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f3550bc1700 1 Processor -- start 2024-04-26T17:55:26.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f3550bc1700 1 -- start start 2024-04-26T17:55:26.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f3550bc1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f354c075650 0x7f354c19bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:26.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f3550bc1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f354c075f70 0x7f354c19c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f3550bc1700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f354c10fe20 0x7f354c1a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f3550bc1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f354c115820 con 0x7f354c075650 2024-04-26T17:55:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f3550bc1700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f354c1156a0 con 0x7f354c10fe20 2024-04-26T17:55:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f3550bc1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f354c1159a0 con 0x7f354c075f70 2024-04-26T17:55:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f354a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f354c075650 0x7f354c19bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f354a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f354c075650 0x7f354c19bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40630/0 (socket says 172.21.15.31:40630) 2024-04-26T17:55:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f354a59c700 1 -- 172.21.15.31:0/1462995041 learned_addr learned my addr 172.21.15.31:0/1462995041 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f3549d9b700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f354c075f70 0x7f354c19c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f354ad9d700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f354c10fe20 0x7f354c1a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f354a59c700 1 -- 172.21.15.31:0/1462995041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f354c075f70 msgr2=0x7f354c19c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f354a59c700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f354c075f70 0x7f354c19c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f354a59c700 1 -- 172.21.15.31:0/1462995041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f354c10fe20 msgr2=0x7f354c1a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f354a59c700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f354c10fe20 0x7f354c1a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f354a59c700 1 -- 172.21.15.31:0/1462995041 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354c1a0d30 con 0x7f354c075650 2024-04-26T17:55:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.186+0000 7f3549d9b700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f354c075f70 0x7f354c19c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:55:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.187+0000 7f354ad9d700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f354c10fe20 0x7f354c1a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:55:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.187+0000 7f354a59c700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f354c075650 0x7f354c19bd70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3540000c00 tx=0x7f35400047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.187+0000 7f353b7fe700 1 -- 172.21.15.31:0/1462995041 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3540027070 con 0x7f354c075650 2024-04-26T17:55:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.187+0000 7f353b7fe700 1 -- 172.21.15.31:0/1462995041 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3540003fa0 con 0x7f354c075650 2024-04-26T17:55:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.187+0000 7f353b7fe700 1 -- 172.21.15.31:0/1462995041 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3540022db0 con 0x7f354c075650 2024-04-26T17:55:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.187+0000 7f3550bc1700 1 -- 172.21.15.31:0/1462995041 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f354c1a0fc0 con 0x7f354c075650 2024-04-26T17:55:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.187+0000 7f3550bc1700 1 -- 172.21.15.31:0/1462995041 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f354c1a88a0 con 0x7f354c075650 2024-04-26T17:55:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.188+0000 7f3550bc1700 1 -- 172.21.15.31:0/1462995041 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f354c076d40 con 0x7f354c075650 2024-04-26T17:55:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.189+0000 7f353b7fe700 1 -- 172.21.15.31:0/1462995041 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f354002a070 con 0x7f354c075650 2024-04-26T17:55:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.190+0000 7f353b7fe700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35340a3cc0 0x7f35340a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.190+0000 7f3549d9b700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35340a3cc0 0x7f35340a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.190+0000 7f353b7fe700 1 -- 172.21.15.31:0/1462995041 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f35400d10e0 con 0x7f354c075650 2024-04-26T17:55:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.190+0000 7f3549d9b700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35340a3cc0 0x7f35340a6180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f353c005950 tx=0x7f353c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.195+0000 7f353b7fe700 1 -- 172.21.15.31:0/1462995041 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3540013070 con 0x7f354c075650 2024-04-26T17:55:26.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:25 smithi186 ceph-mon[32797]: pgmap v368: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:26.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.451+0000 7f3550bc1700 1 -- 172.21.15.31:0/1462995041 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f354c02e490 con 0x7f354c075650 2024-04-26T17:55:26.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.452+0000 7f353b7fe700 1 -- 172.21.15.31:0/1462995041 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11871 (secure 0 0 0) 0x7f354009db20 con 0x7f354c075650 2024-04-26T17:55:26.453 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:55:26.453 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T17:53:25.754649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24457,24473,24446]},"id":1}]} 2024-04-26T17:55:26.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.454+0000 7f3550bc1700 1 -- 172.21.15.31:0/1462995041 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35340a3cc0 msgr2=0x7f35340a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:26.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.454+0000 7f3550bc1700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35340a3cc0 0x7f35340a6180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f353c005950 tx=0x7f353c0058e0 comp rx=0 tx=0).stop 2024-04-26T17:55:26.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.454+0000 7f3550bc1700 1 -- 172.21.15.31:0/1462995041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f354c075650 msgr2=0x7f354c19bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:26.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.454+0000 7f3550bc1700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f354c075650 0x7f354c19bd70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3540000c00 tx=0x7f35400047b0 comp rx=0 tx=0).stop 2024-04-26T17:55:26.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.454+0000 7f3550bc1700 1 -- 172.21.15.31:0/1462995041 shutdown_connections 2024-04-26T17:55:26.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.454+0000 7f3550bc1700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35340a3cc0 0x7f35340a6180 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:26.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.454+0000 7f3550bc1700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f354c10fe20 0x7f354c1a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:26.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.454+0000 7f3550bc1700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f354c075650 0x7f354c19bd70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:26.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.454+0000 7f3550bc1700 1 --2- 172.21.15.31:0/1462995041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f354c075f70 0x7f354c19c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:26.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.454+0000 7f3550bc1700 1 -- 172.21.15.31:0/1462995041 >> 172.21.15.31:0/1462995041 conn(0x7f354c07a580 msgr2=0x7f354c1006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:26.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.454+0000 7f3550bc1700 1 -- 172.21.15.31:0/1462995041 shutdown_connections 2024-04-26T17:55:26.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:26.454+0000 7f3550bc1700 1 -- 172.21.15.31:0/1462995041 wait complete. 2024-04-26T17:55:26.456 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-04-26T17:55:27.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:26 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1462995041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:55:27.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:55:27.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:26 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1462995041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:55:27.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:26 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1462995041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:55:27.361 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:28.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:27 smithi031 ceph-mon[29164]: pgmap v369: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:28.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:27 smithi097 ceph-mon[32419]: pgmap v369: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:28.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:27 smithi186 ceph-mon[32797]: pgmap v369: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.055+0000 7f1441c84700 1 -- 172.21.15.31:0/3439755824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f143c102320 msgr2=0x7f143c1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.055+0000 7f1441c84700 1 --2- 172.21.15.31:0/3439755824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f143c102320 0x7f143c1027a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1430009990 tx=0x7f1430009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.055+0000 7f1441c84700 1 -- 172.21.15.31:0/3439755824 shutdown_connections 2024-04-26T17:55:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.055+0000 7f1441c84700 1 --2- 172.21.15.31:0/3439755824 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f143c102ce0 0x7f143c10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.055+0000 7f1441c84700 1 --2- 172.21.15.31:0/3439755824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f143c108390 0x7f143c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.055+0000 7f1441c84700 1 --2- 172.21.15.31:0/3439755824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f143c102320 0x7f143c1027a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.055+0000 7f1441c84700 1 -- 172.21.15.31:0/3439755824 >> 172.21.15.31:0/3439755824 conn(0x7f143c0792d0 msgr2=0x7f143c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.055+0000 7f1441c84700 1 -- 172.21.15.31:0/3439755824 shutdown_connections 2024-04-26T17:55:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.055+0000 7f1441c84700 1 -- 172.21.15.31:0/3439755824 wait complete. 2024-04-26T17:55:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.056+0000 7f1441c84700 1 Processor -- start 2024-04-26T17:55:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.056+0000 7f1441c84700 1 -- start start 2024-04-26T17:55:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.056+0000 7f1441c84700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f143c102320 0x7f143c19f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.056+0000 7f1441c84700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f143c102ce0 0x7f143c19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.056+0000 7f1441c84700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f143c108390 0x7f143c199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.056+0000 7f1441c84700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f143c112530 con 0x7f143c102ce0 2024-04-26T17:55:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.056+0000 7f1441c84700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f143c1123b0 con 0x7f143c102320 2024-04-26T17:55:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.056+0000 7f1441c84700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f143c1126b0 con 0x7f143c108390 2024-04-26T17:55:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.056+0000 7f143affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f143c102ce0 0x7f143c19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.056+0000 7f143affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f143c102ce0 0x7f143c19fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40662/0 (socket says 172.21.15.31:40662) 2024-04-26T17:55:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.056+0000 7f143affd700 1 -- 172.21.15.31:0/1844494389 learned_addr learned my addr 172.21.15.31:0/1844494389 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.057+0000 7f143bfff700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f143c108390 0x7f143c199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.057+0000 7f143b7fe700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f143c102320 0x7f143c19f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.057+0000 7f143affd700 1 -- 172.21.15.31:0/1844494389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f143c108390 msgr2=0x7f143c199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.057+0000 7f143affd700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f143c108390 0x7f143c199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.057+0000 7f143affd700 1 -- 172.21.15.31:0/1844494389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f143c102320 msgr2=0x7f143c19f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.057+0000 7f143affd700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f143c102320 0x7f143c19f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.057+0000 7f143affd700 1 -- 172.21.15.31:0/1844494389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f143c19a0a0 con 0x7f143c102ce0 2024-04-26T17:55:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.057+0000 7f143b7fe700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f143c102320 0x7f143c19f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:55:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.057+0000 7f143affd700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f143c102ce0 0x7f143c19fcc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1430000c00 tx=0x7f14300047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.057+0000 7f1438ff9700 1 -- 172.21.15.31:0/1844494389 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1430027070 con 0x7f143c102ce0 2024-04-26T17:55:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.057+0000 7f1441c84700 1 -- 172.21.15.31:0/1844494389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f143c19a330 con 0x7f143c102ce0 2024-04-26T17:55:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.057+0000 7f1438ff9700 1 -- 172.21.15.31:0/1844494389 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14300041d0 con 0x7f143c102ce0 2024-04-26T17:55:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.057+0000 7f1438ff9700 1 -- 172.21.15.31:0/1844494389 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143000f460 con 0x7f143c102ce0 2024-04-26T17:55:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.057+0000 7f1441c84700 1 -- 172.21.15.31:0/1844494389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f143c1a6650 con 0x7f143c102ce0 2024-04-26T17:55:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.058+0000 7f1441c84700 1 -- 172.21.15.31:0/1844494389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f143c06a110 con 0x7f143c102ce0 2024-04-26T17:55:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.059+0000 7f1438ff9700 1 -- 172.21.15.31:0/1844494389 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f143002a070 con 0x7f143c102ce0 2024-04-26T17:55:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.060+0000 7f1438ff9700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f14240a3c40 0x7f14240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.060+0000 7f143b7fe700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f14240a3c40 0x7f14240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.060+0000 7f1438ff9700 1 -- 172.21.15.31:0/1844494389 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f14300d13d0 con 0x7f143c102ce0 2024-04-26T17:55:29.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.060+0000 7f143b7fe700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f14240a3c40 0x7f14240a6100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f142c005fd0 tx=0x7f142c005de0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:29.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.065+0000 7f1438ff9700 1 -- 172.21.15.31:0/1844494389 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f143005eff0 con 0x7f143c102ce0 2024-04-26T17:55:29.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.319+0000 7f1441c84700 1 -- 172.21.15.31:0/1844494389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f143c1a6820 con 0x7f143c102ce0 2024-04-26T17:55:29.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.320+0000 7f1438ff9700 1 -- 172.21.15.31:0/1844494389 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11871 (secure 0 0 0) 0x7f1430013070 con 0x7f143c102ce0 2024-04-26T17:55:29.321 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:55:29.321 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T17:53:25.754649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24457,24473,24446]},"id":1}]} 2024-04-26T17:55:29.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.322+0000 7f1441c84700 1 -- 172.21.15.31:0/1844494389 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f14240a3c40 msgr2=0x7f14240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:29.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.322+0000 7f1441c84700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f14240a3c40 0x7f14240a6100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f142c005fd0 tx=0x7f142c005de0 comp rx=0 tx=0).stop 2024-04-26T17:55:29.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.322+0000 7f1441c84700 1 -- 172.21.15.31:0/1844494389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f143c102ce0 msgr2=0x7f143c19fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:29.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.322+0000 7f1441c84700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f143c102ce0 0x7f143c19fcc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1430000c00 tx=0x7f14300047b0 comp rx=0 tx=0).stop 2024-04-26T17:55:29.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.322+0000 7f1441c84700 1 -- 172.21.15.31:0/1844494389 shutdown_connections 2024-04-26T17:55:29.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.322+0000 7f1441c84700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f14240a3c40 0x7f14240a6100 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:29.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.322+0000 7f1441c84700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f143c102320 0x7f143c19f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:29.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.322+0000 7f1441c84700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f143c108390 0x7f143c199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:29.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.322+0000 7f1441c84700 1 --2- 172.21.15.31:0/1844494389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f143c102ce0 0x7f143c19fcc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:29.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.322+0000 7f1441c84700 1 -- 172.21.15.31:0/1844494389 >> 172.21.15.31:0/1844494389 conn(0x7f143c0792d0 msgr2=0x7f143c1015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:29.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.322+0000 7f1441c84700 1 -- 172.21.15.31:0/1844494389 shutdown_connections 2024-04-26T17:55:29.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:29.322+0000 7f1441c84700 1 -- 172.21.15.31:0/1844494389 wait complete. 2024-04-26T17:55:29.323 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-04-26T17:55:29.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:29 smithi031 ceph-mon[29164]: pgmap v370: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:29.991 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:29 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1844494389' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:55:30.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T17:55:30.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:29 smithi097 ceph-mon[32419]: pgmap v370: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:30.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:29 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1844494389' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:55:30.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:29 smithi186 ceph-mon[32797]: pgmap v370: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:30.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:29 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1844494389' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:55:30.269 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:31.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.957+0000 7f695b8f1700 1 -- 172.21.15.31:0/3556160937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f695406bbd0 msgr2=0x7f695406c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:31.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.957+0000 7f695b8f1700 1 --2- 172.21.15.31:0/3556160937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f695406bbd0 0x7f695406c050 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6944009990 tx=0x7f6944009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:31.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.957+0000 7f695b8f1700 1 -- 172.21.15.31:0/3556160937 shutdown_connections 2024-04-26T17:55:31.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.957+0000 7f695b8f1700 1 --2- 172.21.15.31:0/3556160937 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f695406c590 0x7f6954110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:31.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.957+0000 7f695b8f1700 1 --2- 172.21.15.31:0/3556160937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69541094d0 0x7f69541098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:31.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.957+0000 7f695b8f1700 1 --2- 172.21.15.31:0/3556160937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f695406bbd0 0x7f695406c050 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:31.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.957+0000 7f695b8f1700 1 -- 172.21.15.31:0/3556160937 >> 172.21.15.31:0/3556160937 conn(0x7f69540793c0 msgr2=0x7f69540797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:31.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.958+0000 7f695b8f1700 1 -- 172.21.15.31:0/3556160937 shutdown_connections 2024-04-26T17:55:31.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.958+0000 7f695b8f1700 1 -- 172.21.15.31:0/3556160937 wait complete. 2024-04-26T17:55:31.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:31 smithi031 ceph-mon[29164]: pgmap v371: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.958+0000 7f695b8f1700 1 Processor -- start 2024-04-26T17:55:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.958+0000 7f695b8f1700 1 -- start start 2024-04-26T17:55:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f695b8f1700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f695406bbd0 0x7f69541a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f695b8f1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f695406c590 0x7f69541a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f695b8f1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69541094d0 0x7f695419bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f695b8f1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6954113600 con 0x7f69541094d0 2024-04-26T17:55:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f695b8f1700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6954113480 con 0x7f695406bbd0 2024-04-26T17:55:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f695b8f1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6954113780 con 0x7f695406c590 2024-04-26T17:55:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f6959e8e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69541094d0 0x7f695419bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f6959e8e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69541094d0 0x7f695419bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40672/0 (socket says 172.21.15.31:40672) 2024-04-26T17:55:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f6959e8e700 1 -- 172.21.15.31:0/2197354743 learned_addr learned my addr 172.21.15.31:0/2197354743 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f695968d700 1 --2- 172.21.15.31:0/2197354743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f695406bbd0 0x7f69541a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f6958e8c700 1 --2- 172.21.15.31:0/2197354743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f695406c590 0x7f69541a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f6959e8e700 1 -- 172.21.15.31:0/2197354743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f695406c590 msgr2=0x7f69541a1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f6959e8e700 1 --2- 172.21.15.31:0/2197354743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f695406c590 0x7f69541a1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f6959e8e700 1 -- 172.21.15.31:0/2197354743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f695406bbd0 msgr2=0x7f69541a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f6959e8e700 1 --2- 172.21.15.31:0/2197354743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f695406bbd0 0x7f69541a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f6959e8e700 1 -- 172.21.15.31:0/2197354743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f695419c260 con 0x7f69541094d0 2024-04-26T17:55:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.959+0000 7f6959e8e700 1 --2- 172.21.15.31:0/2197354743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69541094d0 0x7f695419bb00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f694c007ee0 tx=0x7f694c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.960+0000 7f694a7fc700 1 -- 172.21.15.31:0/2197354743 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694c00dd20 con 0x7f69541094d0 2024-04-26T17:55:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.960+0000 7f695b8f1700 1 -- 172.21.15.31:0/2197354743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f695419c550 con 0x7f69541094d0 2024-04-26T17:55:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.960+0000 7f694a7fc700 1 -- 172.21.15.31:0/2197354743 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f694c004020 con 0x7f69541094d0 2024-04-26T17:55:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.960+0000 7f694a7fc700 1 -- 172.21.15.31:0/2197354743 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694c015660 con 0x7f69541094d0 2024-04-26T17:55:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.960+0000 7f695b8f1700 1 -- 172.21.15.31:0/2197354743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69541a8810 con 0x7f69541094d0 2024-04-26T17:55:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.961+0000 7f695b8f1700 1 -- 172.21.15.31:0/2197354743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6954104b80 con 0x7f69541094d0 2024-04-26T17:55:31.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.962+0000 7f694a7fc700 1 -- 172.21.15.31:0/2197354743 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f694c0041d0 con 0x7f69541094d0 2024-04-26T17:55:31.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.963+0000 7f694a7fc700 1 --2- 172.21.15.31:0/2197354743 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f69400a3c40 0x7f69400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:31.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.963+0000 7f695968d700 1 --2- 172.21.15.31:0/2197354743 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f69400a3c40 0x7f69400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:31.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.963+0000 7f694a7fc700 1 -- 172.21.15.31:0/2197354743 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f694c0cfdf0 con 0x7f69541094d0 2024-04-26T17:55:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.963+0000 7f695968d700 1 --2- 172.21.15.31:0/2197354743 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f69400a3c40 0x7f69400a6100 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f69500045b0 tx=0x7f695000b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:31.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:31.967+0000 7f694a7fc700 1 -- 172.21.15.31:0/2197354743 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f694c010070 con 0x7f69541094d0 2024-04-26T17:55:32.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:31 smithi097 ceph-mon[32419]: pgmap v371: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:32.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:32.177+0000 7f695b8f1700 1 -- 172.21.15.31:0/2197354743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f695402e490 con 0x7f69541094d0 2024-04-26T17:55:32.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:32.179+0000 7f694a7fc700 1 -- 172.21.15.31:0/2197354743 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22577 (secure 0 0 0) 0x7f694c09c7a0 con 0x7f69541094d0 2024-04-26T17:55:32.179 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:55:32.179 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","created":"2024-04-26T17:44:48.618250+0000","modified":"2024-04-26T17:53:49.754504+0000","last_up_change":"2024-04-26T17:52:00.798886+0000","last_in_change":"2024-04-26T17:51:40.613593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T17:48:12.307250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T17:52:58.650338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T17:53:02.488144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f684ee5-eeee-454a-a00b-e5d6d2901572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6803","nonce":1957398328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6805","nonce":1957398328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6809","nonce":1957398328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6807","nonce":1957398328}]},"public_addr":"172.21.15.31:6803/1957398328","cluster_addr":"172.21.15.31:6805/1957398328","heartbeat_back_addr":"172.21.15.31:6809/1957398328","heartbeat_front_addr":"172.21.15.31:6807/1957398328","state":["exists","up"]},{"osd":1,"uuid":"08be6b26-cb60-41f7-be8b-01a0c93ebafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6801","nonce":4209866427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6803","nonce":4209866427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6807","nonce":4209866427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6805","nonce":4209866427}]},"public_addr":"172.21.15.97:6801/4209866427","cluster_addr":"172.21.15.97:6803/4209866427","heartbeat_back_addr":"172.21.15.97:6807/4209866427","heartbeat_front_addr":"172.21.15.97:6805/4209866427","state":["exists","up"]},{"osd":2,"uuid":"498216b1-a0ac-4923-ade8-ce50fb97d472","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6801","nonce":223535147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6803","nonce":223535147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6807","nonce":223535147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6805","nonce":223535147}]},"public_addr":"172.21.15.186:6801/223535147","cluster_addr":"172.21.15.186:6803/223535147","heartbeat_back_addr":"172.21.15.186:6807/223535147","heartbeat_front_addr":"172.21.15.186:6805/223535147","state":["exists","up"]},{"osd":3,"uuid":"568a352d-a430-4f0f-9a49-6f992e0ceae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6811","nonce":2640112341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6813","nonce":2640112341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6817","nonce":2640112341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6815","nonce":2640112341}]},"public_addr":"172.21.15.31:6811/2640112341","cluster_addr":"172.21.15.31:6813/2640112341","heartbeat_back_addr":"172.21.15.31:6817/2640112341","heartbeat_front_addr":"172.21.15.31:6815/2640112341","state":["exists","up"]},{"osd":4,"uuid":"112ccbf7-24e8-473c-9852-2e7167dd3108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6809","nonce":1359263143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6811","nonce":1359263143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6815","nonce":1359263143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6813","nonce":1359263143}]},"public_addr":"172.21.15.97:6809/1359263143","cluster_addr":"172.21.15.97:6811/1359263143","heartbeat_back_addr":"172.21.15.97:6815/1359263143","heartbeat_front_addr":"172.21.15.97:6813/1359263143","state":["exists","up"]},{"osd":5,"uuid":"eb17759e-1ad8-4bda-9f61-54d2a9781535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6809","nonce":2006134626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6811","nonce":2006134626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6815","nonce":2006134626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6813","nonce":2006134626}]},"public_addr":"172.21.15.186:6809/2006134626","cluster_addr":"172.21.15.186:6811/2006134626","heartbeat_back_addr":"172.21.15.186:6815/2006134626","heartbeat_front_addr":"172.21.15.186:6813/2006134626","state":["exists","up"]},{"osd":6,"uuid":"f1d11e38-40e9-4db4-868f-145f1ffc0d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6819","nonce":2955759395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6821","nonce":2955759395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6825","nonce":2955759395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6823","nonce":2955759395}]},"public_addr":"172.21.15.31:6819/2955759395","cluster_addr":"172.21.15.31:6821/2955759395","heartbeat_back_addr":"172.21.15.31:6825/2955759395","heartbeat_front_addr":"172.21.15.31:6823/2955759395","state":["exists","up"]},{"osd":7,"uuid":"4e95f728-23ff-4bca-9b94-afde2662a5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6817","nonce":3940083075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6819","nonce":3940083075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6823","nonce":3940083075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6821","nonce":3940083075}]},"public_addr":"172.21.15.97:6817/3940083075","cluster_addr":"172.21.15.97:6819/3940083075","heartbeat_back_addr":"172.21.15.97:6823/3940083075","heartbeat_front_addr":"172.21.15.97:6821/3940083075","state":["exists","up"]},{"osd":8,"uuid":"5f91c915-2568-4d20-b645-00dad2efe952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6817","nonce":3193558170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6819","nonce":3193558170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6823","nonce":3193558170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6821","nonce":3193558170}]},"public_addr":"172.21.15.186:6817/3193558170","cluster_addr":"172.21.15.186:6819/3193558170","heartbeat_back_addr":"172.21.15.186:6823/3193558170","heartbeat_front_addr":"172.21.15.186:6821/3193558170","state":["exists","up"]},{"osd":9,"uuid":"12fda87d-830a-461f-a6cb-4699c52dc5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6827","nonce":3660801905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6829","nonce":3660801905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6833","nonce":3660801905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6831","nonce":3660801905}]},"public_addr":"172.21.15.31:6827/3660801905","cluster_addr":"172.21.15.31:6829/3660801905","heartbeat_back_addr":"172.21.15.31:6833/3660801905","heartbeat_front_addr":"172.21.15.31:6831/3660801905","state":["exists","up"]},{"osd":10,"uuid":"92a639b3-918b-485c-bd30-258b4224993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6825","nonce":4097344903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6827","nonce":4097344903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6831","nonce":4097344903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6829","nonce":4097344903}]},"public_addr":"172.21.15.97:6825/4097344903","cluster_addr":"172.21.15.97:6827/4097344903","heartbeat_back_addr":"172.21.15.97:6831/4097344903","heartbeat_front_addr":"172.21.15.97:6829/4097344903","state":["exists","up"]},{"osd":11,"uuid":"842aa599-d21d-4e4a-b872-e9ac20d8c0ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6825","nonce":1063973352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6827","nonce":1063973352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6831","nonce":1063973352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6829","nonce":1063973352}]},"public_addr":"172.21.15.186:6825/1063973352","cluster_addr":"172.21.15.186:6827/1063973352","heartbeat_back_addr":"172.21.15.186:6831/1063973352","heartbeat_front_addr":"172.21.15.186:6829/1063973352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:21.305331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:44.877932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:08.784665+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:34.381762+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:58.999358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:23.620237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:49.961719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:15.363702+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:40.599703+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:07.391645+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:32.800851+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:59.384148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/718966776":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/2035551671":"2024-04-27T17:45:48.218816+0000","172.21.15.31:6801/153341213":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/810461143":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/3148082955":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6801/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/288292084":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/3101596726":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/153341213":"2024-04-27T17:45:48.218816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T17:55:32.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:32.181+0000 7f695b8f1700 1 -- 172.21.15.31:0/2197354743 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f69400a3c40 msgr2=0x7f69400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:32.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:32.181+0000 7f695b8f1700 1 --2- 172.21.15.31:0/2197354743 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f69400a3c40 0x7f69400a6100 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f69500045b0 tx=0x7f695000b3c0 comp rx=0 tx=0).stop 2024-04-26T17:55:32.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:32.181+0000 7f695b8f1700 1 -- 172.21.15.31:0/2197354743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69541094d0 msgr2=0x7f695419bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:32.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:32.181+0000 7f695b8f1700 1 --2- 172.21.15.31:0/2197354743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69541094d0 0x7f695419bb00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f694c007ee0 tx=0x7f694c00d3b0 comp rx=0 tx=0).stop 2024-04-26T17:55:32.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:32.181+0000 7f695b8f1700 1 -- 172.21.15.31:0/2197354743 shutdown_connections 2024-04-26T17:55:32.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:32.181+0000 7f695b8f1700 1 --2- 172.21.15.31:0/2197354743 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f69400a3c40 0x7f69400a6100 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:32.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:32.181+0000 7f695b8f1700 1 --2- 172.21.15.31:0/2197354743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f695406bbd0 0x7f69541a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:32.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:32.181+0000 7f695b8f1700 1 --2- 172.21.15.31:0/2197354743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69541094d0 0x7f695419bb00 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:32.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:32.181+0000 7f695b8f1700 1 --2- 172.21.15.31:0/2197354743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f695406c590 0x7f69541a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:32.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:32.181+0000 7f695b8f1700 1 -- 172.21.15.31:0/2197354743 >> 172.21.15.31:0/2197354743 conn(0x7f69540793c0 msgr2=0x7f695410e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:32.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:32.181+0000 7f695b8f1700 1 -- 172.21.15.31:0/2197354743 shutdown_connections 2024-04-26T17:55:32.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:32.181+0000 7f695b8f1700 1 -- 172.21.15.31:0/2197354743 wait complete. 2024-04-26T17:55:32.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:31 smithi186 ceph-mon[32797]: pgmap v371: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:32.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:55:33.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:32 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2197354743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:55:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:33.266 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:32 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2197354743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:55:33.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:32 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2197354743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:55:34.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:33 smithi097 ceph-mon[32419]: pgmap v372: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:34.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:33 smithi186 ceph-mon[32797]: pgmap v372: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:34.272 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:33 smithi031 ceph-mon[29164]: pgmap v372: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:34.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.913+0000 7f801f7a2700 1 -- 172.21.15.31:0/1060341069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f801806b130 msgr2=0x7f801806b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:34.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.913+0000 7f801f7a2700 1 --2- 172.21.15.31:0/1060341069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f801806b130 0x7f801806b590 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f800c009990 tx=0x7f800c009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.913+0000 7f801f7a2700 1 -- 172.21.15.31:0/1060341069 shutdown_connections 2024-04-26T17:55:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.913+0000 7f801f7a2700 1 --2- 172.21.15.31:0/1060341069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f80181094a0 0x7f8018109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.913+0000 7f801f7a2700 1 --2- 172.21.15.31:0/1060341069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f801806bad0 0x7f80181102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.913+0000 7f801f7a2700 1 --2- 172.21.15.31:0/1060341069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f801806b130 0x7f801806b590 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.913+0000 7f801f7a2700 1 -- 172.21.15.31:0/1060341069 >> 172.21.15.31:0/1060341069 conn(0x7f80180792d0 msgr2=0x7f80180796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.913+0000 7f801f7a2700 1 -- 172.21.15.31:0/1060341069 shutdown_connections 2024-04-26T17:55:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.913+0000 7f801f7a2700 1 -- 172.21.15.31:0/1060341069 wait complete. 2024-04-26T17:55:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801f7a2700 1 Processor -- start 2024-04-26T17:55:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801f7a2700 1 -- start start 2024-04-26T17:55:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801f7a2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f801806b130 0x7f801819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801f7a2700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f801806bad0 0x7f801819fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801f7a2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80181094a0 0x7f80181999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801f7a2700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f80181135b0 con 0x7f80181094a0 2024-04-26T17:55:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801f7a2700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f8018113430 con 0x7f801806bad0 2024-04-26T17:55:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801f7a2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8018113730 con 0x7f801806b130 2024-04-26T17:55:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801dd3f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80181094a0 0x7f80181999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801dd3f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80181094a0 0x7f80181999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52590/0 (socket says 172.21.15.31:52590) 2024-04-26T17:55:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801dd3f700 1 -- 172.21.15.31:0/3408174623 learned_addr learned my addr 172.21.15.31:0/3408174623 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801d53e700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f801806b130 0x7f801819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801cd3d700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f801806bad0 0x7f801819fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801dd3f700 1 -- 172.21.15.31:0/3408174623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f801806b130 msgr2=0x7f801819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.914+0000 7f801dd3f700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f801806b130 0x7f801819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.915+0000 7f801dd3f700 1 -- 172.21.15.31:0/3408174623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f801806bad0 msgr2=0x7f801819fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.915+0000 7f801dd3f700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f801806bad0 0x7f801819fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.915+0000 7f801dd3f700 1 -- 172.21.15.31:0/3408174623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f801819a100 con 0x7f80181094a0 2024-04-26T17:55:34.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.915+0000 7f801cd3d700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f801806bad0 0x7f801819fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:55:34.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.915+0000 7f801dd3f700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80181094a0 0x7f80181999a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f800800d980 tx=0x7f800800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:34.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.915+0000 7f80167fc700 1 -- 172.21.15.31:0/3408174623 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8008016070 con 0x7f80181094a0 2024-04-26T17:55:34.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.915+0000 7f801f7a2700 1 -- 172.21.15.31:0/3408174623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f801819a3f0 con 0x7f80181094a0 2024-04-26T17:55:34.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.915+0000 7f80167fc700 1 -- 172.21.15.31:0/3408174623 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8008004830 con 0x7f80181094a0 2024-04-26T17:55:34.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.915+0000 7f80167fc700 1 -- 172.21.15.31:0/3408174623 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80080054b0 con 0x7f80181094a0 2024-04-26T17:55:34.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.915+0000 7f801f7a2700 1 -- 172.21.15.31:0/3408174623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80181a66b0 con 0x7f80181094a0 2024-04-26T17:55:34.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.916+0000 7f801f7a2700 1 -- 172.21.15.31:0/3408174623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f801806a110 con 0x7f80181094a0 2024-04-26T17:55:34.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.917+0000 7f80167fc700 1 -- 172.21.15.31:0/3408174623 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f80080049e0 con 0x7f80181094a0 2024-04-26T17:55:34.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.918+0000 7f80167fc700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f80000a3c40 0x7f80000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:34.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.918+0000 7f801d53e700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f80000a3c40 0x7f80000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:34.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.918+0000 7f80167fc700 1 -- 172.21.15.31:0/3408174623 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f80080d0760 con 0x7f80181094a0 2024-04-26T17:55:34.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.919+0000 7f801d53e700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f80000a3c40 0x7f80000a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8010009550 tx=0x7f80100091b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:34.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:34.922+0000 7f80167fc700 1 -- 172.21.15.31:0/3408174623 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f800801f020 con 0x7f80181094a0 2024-04-26T17:55:35.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:35.177+0000 7f801f7a2700 1 -- 172.21.15.31:0/3408174623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f801819aed0 con 0x7f80181094a0 2024-04-26T17:55:35.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:35.178+0000 7f80167fc700 1 -- 172.21.15.31:0/3408174623 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11871 (secure 0 0 0) 0x7f800809d170 con 0x7f80181094a0 2024-04-26T17:55:35.178 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:55:35.178 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T17:53:25.754649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24457,24473,24446]},"id":1}]} 2024-04-26T17:55:35.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:35.180+0000 7f801f7a2700 1 -- 172.21.15.31:0/3408174623 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f80000a3c40 msgr2=0x7f80000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:35.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:35.180+0000 7f801f7a2700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f80000a3c40 0x7f80000a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8010009550 tx=0x7f80100091b0 comp rx=0 tx=0).stop 2024-04-26T17:55:35.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:35.180+0000 7f801f7a2700 1 -- 172.21.15.31:0/3408174623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80181094a0 msgr2=0x7f80181999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:35.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:35.180+0000 7f801f7a2700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80181094a0 0x7f80181999a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f800800d980 tx=0x7f800800dc90 comp rx=0 tx=0).stop 2024-04-26T17:55:35.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:35.180+0000 7f801f7a2700 1 -- 172.21.15.31:0/3408174623 shutdown_connections 2024-04-26T17:55:35.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:35.180+0000 7f801f7a2700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f80000a3c40 0x7f80000a6100 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:35.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:35.180+0000 7f801f7a2700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80181094a0 0x7f80181999a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:35.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:35.180+0000 7f801f7a2700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f801806b130 0x7f801819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:35.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:35.180+0000 7f801f7a2700 1 --2- 172.21.15.31:0/3408174623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f801806bad0 0x7f801819fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:35.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:35.180+0000 7f801f7a2700 1 -- 172.21.15.31:0/3408174623 >> 172.21.15.31:0/3408174623 conn(0x7f80180792d0 msgr2=0x7f80181004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:35.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:35.180+0000 7f801f7a2700 1 -- 172.21.15.31:0/3408174623 shutdown_connections 2024-04-26T17:55:35.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:35.180+0000 7f801f7a2700 1 -- 172.21.15.31:0/3408174623 wait complete. 2024-04-26T17:55:35.181 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-04-26T17:55:35.896 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T17:53:05.425897+0000', 'modified': '2024-04-26T17:53:25.754649+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24473, 'mds_1': 24457, 'mds_2': 24446}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24428': {'gid': 24428, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.97:6835/1721162755', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 1721162755}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 1721162755}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24439': {'gid': 24439, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6835/580433615', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6834', 'nonce': 580433615}, {'type': 'v1', 'addr': '172.21.15.186:6835', 'nonce': 580433615}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6837/1510454985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6836', 'nonce': 1510454985}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 1510454985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24446': {'gid': 24446, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.97:6837/3334702590', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 3334702590}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3334702590}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.186:6839/1587952500', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 1587952500}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 1587952500}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.97:6839/3765855684', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6838', 'nonce': 3765855684}, {'type': 'v1', 'addr': '172.21.15.97:6839', 'nonce': 3765855684}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24473, 'qdb_cluster': [24457, 24473, 24446]} 2024-04-26T17:55:35.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T17:55:35.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell mds.h status 2024-04-26T17:55:36.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:35 smithi097 ceph-mon[32419]: pgmap v373: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:36.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:35 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3408174623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:55:36.147 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:36.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:35 smithi186 ceph-mon[32797]: pgmap v373: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:35 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3408174623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:55:36.269 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:35 smithi031 ceph-mon[29164]: pgmap v373: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:36.269 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:35 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3408174623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:55:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.819+0000 7f064b27d700 1 -- 172.21.15.31:0/2611945121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06441034a0 msgr2=0x7f0644103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.819+0000 7f064b27d700 1 --2- 172.21.15.31:0/2611945121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06441034a0 0x7f0644103920 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0634009990 tx=0x7f0634009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.819+0000 7f064b27d700 1 -- 172.21.15.31:0/2611945121 shutdown_connections 2024-04-26T17:55:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.819+0000 7f064b27d700 1 --2- 172.21.15.31:0/2611945121 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0644103e60 0x7f06441102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.819+0000 7f064b27d700 1 --2- 172.21.15.31:0/2611945121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06441094e0 0x7f06441098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.819+0000 7f064b27d700 1 --2- 172.21.15.31:0/2611945121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06441034a0 0x7f0644103920 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.819+0000 7f064b27d700 1 -- 172.21.15.31:0/2611945121 >> 172.21.15.31:0/2611945121 conn(0x7f064407a590 msgr2=0x7f064407a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.819+0000 7f064b27d700 1 -- 172.21.15.31:0/2611945121 shutdown_connections 2024-04-26T17:55:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.819+0000 7f064b27d700 1 -- 172.21.15.31:0/2611945121 wait complete. 2024-04-26T17:55:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.820+0000 7f064b27d700 1 Processor -- start 2024-04-26T17:55:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.820+0000 7f064b27d700 1 -- start start 2024-04-26T17:55:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.820+0000 7f064b27d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06441034a0 0x7f06441a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.820+0000 7f064b27d700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0644103e60 0x7f06441a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.820+0000 7f064b27d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06441094e0 0x7f064419bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.820+0000 7f064b27d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f06441135e0 con 0x7f06441034a0 2024-04-26T17:55:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.820+0000 7f064b27d700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0644113460 con 0x7f0644103e60 2024-04-26T17:55:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.820+0000 7f064b27d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0644113760 con 0x7f06441094e0 2024-04-26T17:55:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.820+0000 7f0649019700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06441034a0 0x7f06441a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.820+0000 7f0649019700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06441034a0 0x7f06441a1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52610/0 (socket says 172.21.15.31:52610) 2024-04-26T17:55:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.820+0000 7f0649019700 1 -- 172.21.15.31:0/901326373 learned_addr learned my addr 172.21.15.31:0/901326373 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.820+0000 7f064981a700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06441094e0 0x7f064419bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.820+0000 7f0648818700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0644103e60 0x7f06441a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.821+0000 7f0649019700 1 -- 172.21.15.31:0/901326373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06441094e0 msgr2=0x7f064419bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.821+0000 7f0649019700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06441094e0 0x7f064419bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.821+0000 7f0649019700 1 -- 172.21.15.31:0/901326373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0644103e60 msgr2=0x7f06441a1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.821+0000 7f0649019700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0644103e60 0x7f06441a1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.821+0000 7f0649019700 1 -- 172.21.15.31:0/901326373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064419c400 con 0x7f06441034a0 2024-04-26T17:55:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.821+0000 7f0648818700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0644103e60 0x7f06441a1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:55:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.821+0000 7f0649019700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06441034a0 0x7f06441a1a50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f064000d950 tx=0x7f064000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.821+0000 7f063a7fc700 1 -- 172.21.15.31:0/901326373 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0640016070 con 0x7f06441034a0 2024-04-26T17:55:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.821+0000 7f063a7fc700 1 -- 172.21.15.31:0/901326373 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0640009df0 con 0x7f06441034a0 2024-04-26T17:55:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.821+0000 7f064b27d700 1 -- 172.21.15.31:0/901326373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f064419c6f0 con 0x7f06441034a0 2024-04-26T17:55:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.821+0000 7f063a7fc700 1 -- 172.21.15.31:0/901326373 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064000f3c0 con 0x7f06441034a0 2024-04-26T17:55:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.821+0000 7f064b27d700 1 -- 172.21.15.31:0/901326373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06441a89d0 con 0x7f06441034a0 2024-04-26T17:55:37.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.823+0000 7f063a7fc700 1 -- 172.21.15.31:0/901326373 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0640019070 con 0x7f06441034a0 2024-04-26T17:55:37.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.824+0000 7f063a7fc700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f06300a3c40 0x7f06300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.824+0000 7f0648818700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f06300a3c40 0x7f06300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.824+0000 7f063a7fc700 1 -- 172.21.15.31:0/901326373 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f06400cff00 con 0x7f06441034a0 2024-04-26T17:55:37.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.825+0000 7f0648818700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f06300a3c40 0x7f06300a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0634006010 tx=0x7f063400b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:37.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.825+0000 7f0655b0fb80 1 Processor -- start 2024-04-26T17:55:37.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.825+0000 7f0655b0fb80 1 -- start start 2024-04-26T17:55:37.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.825+0000 7f0655b0fb80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b97846fa20 0x55b97846fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.825+0000 7f0655b0fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b978469d50 0x55b97846a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.825+0000 7f0655b0fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b978476770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.825+0000 7f0655b0fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b9783bc7e0 con 0x55b97846a6f0 2024-04-26T17:55:37.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.825+0000 7f0655b0fb80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b9783228e0 con 0x55b97846fa20 2024-04-26T17:55:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.825+0000 7f0655b0fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b9783b87a0 con 0x55b978469d50 2024-04-26T17:55:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.825+0000 7f064981a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b978476770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.825+0000 7f064981a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b978476770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52626/0 (socket says 172.21.15.31:52626) 2024-04-26T17:55:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.825+0000 7f064981a700 1 -- 172.21.15.31:0/3484153125 learned_addr learned my addr 172.21.15.31:0/3484153125 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.825+0000 7f0648818700 1 --2- 172.21.15.31:0/3484153125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b978469d50 0x55b97846a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.825+0000 7f0649019700 1 --2- 172.21.15.31:0/3484153125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b97846fa20 0x55b97846fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.826+0000 7f064981a700 1 -- 172.21.15.31:0/3484153125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b978469d50 msgr2=0x55b97846a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.826+0000 7f064981a700 1 --2- 172.21.15.31:0/3484153125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b978469d50 0x55b97846a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.826+0000 7f064981a700 1 -- 172.21.15.31:0/3484153125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b97846fa20 msgr2=0x55b97846fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.826+0000 7f064981a700 1 --2- 172.21.15.31:0/3484153125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b97846fa20 0x55b97846fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.826+0000 7f064981a700 1 -- 172.21.15.31:0/3484153125 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b978309880 con 0x55b97846a6f0 2024-04-26T17:55:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.826+0000 7f064981a700 1 --2- 172.21.15.31:0/3484153125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b978476770 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f063c003f90 tx=0x7f063c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c94a7f16ce949ef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.826+0000 7f062f7fe700 1 -- 172.21.15.31:0/3484153125 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c0046f0 con 0x55b97846a6f0 2024-04-26T17:55:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.826+0000 7f062f7fe700 1 -- 172.21.15.31:0/3484153125 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f063c01a070 con 0x55b97846a6f0 2024-04-26T17:55:37.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.826+0000 7f062f7fe700 1 -- 172.21.15.31:0/3484153125 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c0156e0 con 0x55b97846a6f0 2024-04-26T17:55:37.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.827+0000 7f0655b0fb80 1 -- 172.21.15.31:0/3484153125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 msgr2=0x55b978476770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:37.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.827+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/3484153125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b978476770 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f063c003f90 tx=0x7f063c009c20 comp rx=0 tx=0).stop 2024-04-26T17:55:37.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.827+0000 7f0655b0fb80 1 -- 172.21.15.31:0/3484153125 shutdown_connections 2024-04-26T17:55:37.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.827+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/3484153125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b97846fa20 0x55b97846fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.827+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/3484153125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b978476770 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.827+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/3484153125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b978469d50 0x55b97846a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.827+0000 7f0655b0fb80 1 -- 172.21.15.31:0/3484153125 >> 172.21.15.31:0/3484153125 conn(0x55b978398570 msgr2=0x55b978464900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:37.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.827+0000 7f0655b0fb80 1 -- 172.21.15.31:0/3484153125 shutdown_connections 2024-04-26T17:55:37.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.827+0000 7f0655b0fb80 1 -- 172.21.15.31:0/3484153125 wait complete. 2024-04-26T17:55:37.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.827+0000 7f0655b0fb80 1 -- start start 2024-04-26T17:55:37.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.827+0000 7f0655b0fb80 1 Processor -- start 2024-04-26T17:55:37.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0655b0fb80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b978469d50 0x55b978476310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0655b0fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b9784736b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0655b0fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b97846fa20 0x55b978473bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0655b0fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b978476cb0 con 0x55b97846a6f0 2024-04-26T17:55:37.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0655b0fb80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b978376a30 con 0x55b978469d50 2024-04-26T17:55:37.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0655b0fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b978476e30 con 0x55b97846fa20 2024-04-26T17:55:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0655b0fb80 10 client.0 _ll_register_callbacks cb 0x55b978354900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:55:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0648818700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b9784736b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0648818700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b9784736b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52642/0 (socket says 172.21.15.31:52642) 2024-04-26T17:55:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0648818700 1 -- 172.21.15.31:0/2798642842 learned_addr learned my addr 172.21.15.31:0/2798642842 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f064981a700 1 --2- 172.21.15.31:0/2798642842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b97846fa20 0x55b978473bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0649019700 1 --2- 172.21.15.31:0/2798642842 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b978469d50 0x55b978476310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0648818700 1 -- 172.21.15.31:0/2798642842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b97846fa20 msgr2=0x55b978473bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0648818700 1 --2- 172.21.15.31:0/2798642842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b97846fa20 0x55b978473bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0648818700 1 -- 172.21.15.31:0/2798642842 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b978469d50 msgr2=0x55b978476310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0648818700 1 --2- 172.21.15.31:0/2798642842 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b978469d50 0x55b978476310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0648818700 1 -- 172.21.15.31:0/2798642842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b97837a430 con 0x55b97846a6f0 2024-04-26T17:55:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0649019700 1 --2- 172.21.15.31:0/2798642842 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b978469d50 0x55b978476310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:55:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f0648818700 1 --2- 172.21.15.31:0/2798642842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b9784736b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0634009990 tx=0x7f063401a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9b2876b693e42 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f062cff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:55:37.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.828+0000 7f062cff9700 1 -- 172.21.15.31:0/2798642842 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0634035060 con 0x55b97846a6f0 2024-04-26T17:55:37.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.829+0000 7f062cff9700 1 -- 172.21.15.31:0/2798642842 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f063400fd90 con 0x55b97846a6f0 2024-04-26T17:55:37.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.829+0000 7f0655b0fb80 1 -- 172.21.15.31:0/2798642842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f063c01a070 con 0x55b97846a6f0 2024-04-26T17:55:37.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.829+0000 7f062cff9700 1 -- 172.21.15.31:0/2798642842 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0634003860 con 0x55b97846a6f0 2024-04-26T17:55:37.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.829+0000 7f0655b0fb80 10 client.14955 fetch_fsmap learned FSMap version 25 2024-04-26T17:55:37.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.829+0000 7f0655b0fb80 1 -- 172.21.15.31:0/2798642842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b978481480 con 0x55b97846a6f0 2024-04-26T17:55:37.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.829+0000 7f062cff9700 1 -- 172.21.15.31:0/2798642842 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f063c01a070 con 0x55b97846a6f0 2024-04-26T17:55:37.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.829+0000 7f062cff9700 1 -- 172.21.15.31:0/2798642842 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f0634030460 con 0x55b97846a6f0 2024-04-26T17:55:37.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.829+0000 7f0655b0fb80 10 client.14955 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T17:55:37.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.829+0000 7f0655b0fb80 10 client.14955 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T17:55:37.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.829+0000 7f0655b0fb80 20 client.14955 populate_metadata read hostname 'smithi031' 2024-04-26T17:55:37.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.829+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/2798642842 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55b978483b70 0x55b978484fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.829+0000 7f0655b0fb80 4 client.14955 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.829+0000 7f0655b0fb80 1 -- 172.21.15.31:0/2798642842 --> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9783ca1c0 con 0x55b978483b70 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.830+0000 7f0649019700 1 --2- 172.21.15.31:0/2798642842 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55b978483b70 0x55b978484fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.830+0000 7f0649019700 1 --2- 172.21.15.31:0/2798642842 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55b978483b70 0x55b978484fe0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f742fb6a4007fc0d server_cookie=6421da22c46a646d in_seq=0 out_seq=0 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.830+0000 7f062cff9700 10 client.14955 ms_handle_connect on v2:172.21.15.97:6836/3334702590 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f062cff9700 1 -- 172.21.15.31:0/2798642842 <== mds.2 v2:172.21.15.97:6836/3334702590 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b9783ca1c0 con 0x55b978483b70 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f062cff9700 10 client.14955 handle_command_reply: tid=0 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f0655b0fb80 1 client.14955 shutdown 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f0655b0fb80 1 -- 172.21.15.31:0/2798642842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 msgr2=0x55b9784736b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/2798642842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b9784736b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0634009990 tx=0x7f063401a050 comp rx=0 tx=0).stop 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f0655b0fb80 1 -- 172.21.15.31:0/2798642842 shutdown_connections 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/2798642842 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55b978483b70 0x55b978484fe0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/2798642842 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b978469d50 0x55b978476310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/2798642842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b97846fa20 0x55b978473bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f062cff9700 0 client.14955 ms_handle_reset on v2:172.21.15.97:6836/3334702590 2024-04-26T17:55:37.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/2798642842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b9784736b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f0655b0fb80 1 -- 172.21.15.31:0/2798642842 >> 172.21.15.31:0/2798642842 conn(0x55b978398570 msgr2=0x55b978468d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:37.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f0655b0fb80 1 -- 172.21.15.31:0/2798642842 shutdown_connections 2024-04-26T17:55:37.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f0655b0fb80 1 -- 172.21.15.31:0/2798642842 wait complete. 2024-04-26T17:55:37.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.831+0000 7f0655b0fb80 20 client.14955 trim_cache size 0 max 16384 2024-04-26T17:55:37.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.852+0000 7f0655b0fb80 1 Processor -- start 2024-04-26T17:55:37.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.852+0000 7f0655b0fb80 1 -- start start 2024-04-26T17:55:37.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.852+0000 7f0655b0fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b978469d50 0x55b978472f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.852+0000 7f0655b0fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b97846a6f0 0x55b97848e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.852+0000 7f0655b0fb80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b97846fa20 0x55b97848e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.852+0000 7f0655b0fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b978483320 con 0x55b978469d50 2024-04-26T17:55:37.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.852+0000 7f0655b0fb80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b9784839f0 con 0x55b97846fa20 2024-04-26T17:55:37.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.852+0000 7f0655b0fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b9784840c0 con 0x55b97846a6f0 2024-04-26T17:55:37.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.852+0000 7f0649019700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b978469d50 0x55b978472f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.852+0000 7f0649019700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b978469d50 0x55b978472f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52654/0 (socket says 172.21.15.31:52654) 2024-04-26T17:55:37.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.852+0000 7f0649019700 1 -- 172.21.15.31:0/4245447765 learned_addr learned my addr 172.21.15.31:0/4245447765 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:37.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.853+0000 7f064981a700 1 --2- 172.21.15.31:0/4245447765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b97846fa20 0x55b97848e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.853+0000 7f0649019700 1 -- 172.21.15.31:0/4245447765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b97846a6f0 msgr2=0x55b97848e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:37.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.853+0000 7f0649019700 1 --2- 172.21.15.31:0/4245447765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b97846a6f0 0x55b97848e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.853+0000 7f0649019700 1 -- 172.21.15.31:0/4245447765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b97846fa20 msgr2=0x55b97848e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:37.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.853+0000 7f0649019700 1 --2- 172.21.15.31:0/4245447765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b97846fa20 0x55b97848e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.853+0000 7f0649019700 1 -- 172.21.15.31:0/4245447765 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b978476a50 con 0x55b978469d50 2024-04-26T17:55:37.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.853+0000 7f0649019700 1 --2- 172.21.15.31:0/4245447765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b978469d50 0x55b978472f60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f064009c1a0 tx=0x7f064009f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5135b44274c57c9f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:37.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.853+0000 7f062effd700 1 -- 172.21.15.31:0/4245447765 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0640020040 con 0x55b978469d50 2024-04-26T17:55:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.853+0000 7f062effd700 1 -- 172.21.15.31:0/4245447765 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06400b5070 con 0x55b978469d50 2024-04-26T17:55:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.853+0000 7f062effd700 1 -- 172.21.15.31:0/4245447765 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06400a49f0 con 0x55b978469d50 2024-04-26T17:55:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.854+0000 7f0655b0fb80 1 -- 172.21.15.31:0/4245447765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b978469d50 msgr2=0x55b978472f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.854+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/4245447765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b978469d50 0x55b978472f60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f064009c1a0 tx=0x7f064009f040 comp rx=0 tx=0).stop 2024-04-26T17:55:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.854+0000 7f0655b0fb80 1 -- 172.21.15.31:0/4245447765 shutdown_connections 2024-04-26T17:55:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.854+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/4245447765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b97846fa20 0x55b97848e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.854+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/4245447765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b978469d50 0x55b978472f60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.854+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/4245447765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b97846a6f0 0x55b97848e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.854+0000 7f0655b0fb80 1 -- 172.21.15.31:0/4245447765 >> 172.21.15.31:0/4245447765 conn(0x55b978398570 msgr2=0x55b978474250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.854+0000 7f0655b0fb80 1 -- 172.21.15.31:0/4245447765 shutdown_connections 2024-04-26T17:55:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.854+0000 7f0655b0fb80 1 -- 172.21.15.31:0/4245447765 wait complete. 2024-04-26T17:55:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.854+0000 7f0655b0fb80 1 -- start start 2024-04-26T17:55:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.854+0000 7f0655b0fb80 1 Processor -- start 2024-04-26T17:55:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0655b0fb80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b978469d50 0x55b97848d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0655b0fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b97848d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0655b0fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b97846fa20 0x55b97848df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0655b0fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b978484e60 con 0x55b97846a6f0 2024-04-26T17:55:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0655b0fb80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b978484790 con 0x55b978469d50 2024-04-26T17:55:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0655b0fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b978485530 con 0x55b97846fa20 2024-04-26T17:55:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0655b0fb80 10 client.0 _ll_register_callbacks cb 0x55b978354900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:55:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0648818700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b97848d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0648818700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b97848d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52666/0 (socket says 172.21.15.31:52666) 2024-04-26T17:55:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0648818700 1 -- 172.21.15.31:0/98293460 learned_addr learned my addr 172.21.15.31:0/98293460 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0649019700 1 --2- 172.21.15.31:0/98293460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b978469d50 0x55b97848d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0648818700 1 -- 172.21.15.31:0/98293460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b97846fa20 msgr2=0x55b97848df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0648818700 1 --2- 172.21.15.31:0/98293460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b97846fa20 0x55b97848df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0648818700 1 -- 172.21.15.31:0/98293460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b978469d50 msgr2=0x55b97848d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0648818700 1 --2- 172.21.15.31:0/98293460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b978469d50 0x55b97848d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0648818700 1 -- 172.21.15.31:0/98293460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b97848e770 con 0x55b97846a6f0 2024-04-26T17:55:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0648818700 1 --2- 172.21.15.31:0/98293460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b97848d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0634030ac0 tx=0x7f0634030740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768b034e66e1e208 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f062cff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:55:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f062cff9700 1 -- 172.21.15.31:0/98293460 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0634040070 con 0x55b97846a6f0 2024-04-26T17:55:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f062cff9700 1 -- 172.21.15.31:0/98293460 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f0634003a00 con 0x55b97846a6f0 2024-04-26T17:55:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.855+0000 7f0655b0fb80 1 -- 172.21.15.31:0/98293460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06400b5070 con 0x55b97846a6f0 2024-04-26T17:55:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.856+0000 7f062cff9700 1 -- 172.21.15.31:0/98293460 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f063404c050 con 0x55b97846a6f0 2024-04-26T17:55:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.856+0000 7f0655b0fb80 10 client.14961 fetch_fsmap learned FSMap version 25 2024-04-26T17:55:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.856+0000 7f0655b0fb80 1 -- 172.21.15.31:0/98293460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b97846bea0 con 0x55b97846a6f0 2024-04-26T17:55:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.856+0000 7f062cff9700 1 -- 172.21.15.31:0/98293460 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06400b5070 con 0x55b97846a6f0 2024-04-26T17:55:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.856+0000 7f062cff9700 1 -- 172.21.15.31:0/98293460 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f063403c430 con 0x55b97846a6f0 2024-04-26T17:55:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.856+0000 7f0655b0fb80 10 client.14961 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T17:55:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.856+0000 7f0655b0fb80 10 client.14961 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T17:55:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.856+0000 7f0655b0fb80 20 client.14961 populate_metadata read hostname 'smithi031' 2024-04-26T17:55:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.856+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/98293460 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55b9784f03f0 0x55b9784ef100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.856+0000 7f0655b0fb80 4 client.14961 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T17:55:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.856+0000 7f0655b0fb80 1 -- 172.21.15.31:0/98293460 --> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] -- command(tid 0: {"prefix": "status"}) -- 0x55b9783c4260 con 0x55b9784f03f0 2024-04-26T17:55:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.856+0000 7f0649019700 1 --2- 172.21.15.31:0/98293460 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55b9784f03f0 0x55b9784ef100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.857+0000 7f062cff9700 10 client.14961 ms_handle_connect on v2:172.21.15.97:6836/3334702590 2024-04-26T17:55:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.857+0000 7f0649019700 1 --2- 172.21.15.31:0/98293460 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55b9784f03f0 0x55b9784ef100 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b5f823b094006a5d server_cookie=fb5ed88eb23f9a51 in_seq=0 out_seq=0 2024-04-26T17:55:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.857+0000 7f062cff9700 1 -- 172.21.15.31:0/98293460 <== mds.2 v2:172.21.15.97:6836/3334702590 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b9783c4260 con 0x55b9784f03f0 2024-04-26T17:55:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.857+0000 7f062cff9700 10 client.14961 handle_command_reply: tid=0 2024-04-26T17:55:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.857+0000 7f0655b0fb80 1 client.14961 shutdown 2024-04-26T17:55:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.857+0000 7f0655b0fb80 1 -- 172.21.15.31:0/98293460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 msgr2=0x55b97848d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.857+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/98293460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b97848d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0634030ac0 tx=0x7f0634030740 comp rx=0 tx=0).stop 2024-04-26T17:55:37.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.857+0000 7f0655b0fb80 1 -- 172.21.15.31:0/98293460 shutdown_connections 2024-04-26T17:55:37.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.857+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/98293460 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55b9784f03f0 0x55b9784ef100 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.857+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/98293460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b978469d50 0x55b97848d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.857+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/98293460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b97846fa20 0x55b97848df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.857+0000 7f0655b0fb80 1 --2- 172.21.15.31:0/98293460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b97846a6f0 0x55b97848d990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.857+0000 7f0655b0fb80 1 -- 172.21.15.31:0/98293460 >> 172.21.15.31:0/98293460 conn(0x55b978398570 msgr2=0x55b978473670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:37.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.857+0000 7f062cff9700 0 client.14961 ms_handle_reset on v2:172.21.15.97:6836/3334702590 2024-04-26T17:55:37.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.858+0000 7f0655b0fb80 1 -- 172.21.15.31:0/98293460 shutdown_connections 2024-04-26T17:55:37.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.858+0000 7f0655b0fb80 1 -- 172.21.15.31:0/98293460 wait complete. 2024-04-26T17:55:37.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.858+0000 7f0655b0fb80 20 client.14961 trim_cache size 0 max 16384 2024-04-26T17:55:37.863 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T17:55:37.864 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "7e53e050-03f4-11ef-bc93-c7b262605968", 2024-04-26T17:55:37.864 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 2, 2024-04-26T17:55:37.864 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24446, 2024-04-26T17:55:37.864 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-26T17:55:37.864 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-26T17:55:37.864 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-26T17:55:37.864 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 138.180267638, 2024-04-26T17:55:37.864 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 25, 2024-04-26T17:55:37.864 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 90, 2024-04-26T17:55:37.864 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T17:55:37.864 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 181.02312431300001 2024-04-26T17:55:37.864 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T17:55:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.858+0000 7f064b27d700 1 -- 172.21.15.31:0/901326373 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f06300a3c40 msgr2=0x7f06300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.858+0000 7f064b27d700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f06300a3c40 0x7f06300a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0634006010 tx=0x7f063400b540 comp rx=0 tx=0).stop 2024-04-26T17:55:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.858+0000 7f064b27d700 1 -- 172.21.15.31:0/901326373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06441034a0 msgr2=0x7f06441a1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.858+0000 7f064b27d700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06441034a0 0x7f06441a1a50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f064000d950 tx=0x7f064000dc60 comp rx=0 tx=0).stop 2024-04-26T17:55:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.859+0000 7f064b27d700 1 -- 172.21.15.31:0/901326373 shutdown_connections 2024-04-26T17:55:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.859+0000 7f064b27d700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f06300a3c40 0x7f06300a6100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.859+0000 7f064b27d700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06441094e0 0x7f064419bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.859+0000 7f064b27d700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06441034a0 0x7f06441a1a50 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.859+0000 7f064b27d700 1 --2- 172.21.15.31:0/901326373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0644103e60 0x7f06441a1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:37.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.859+0000 7f064b27d700 1 -- 172.21.15.31:0/901326373 >> 172.21.15.31:0/901326373 conn(0x7f064407a590 msgr2=0x7f064410e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:37.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.859+0000 7f064b27d700 1 -- 172.21.15.31:0/901326373 shutdown_connections 2024-04-26T17:55:37.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:37.859+0000 7f064b27d700 1 -- 172.21.15.31:0/901326373 wait complete. 2024-04-26T17:55:38.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:37 smithi097 ceph-mon[32419]: pgmap v374: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:38.231 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:37 smithi031 ceph-mon[29164]: pgmap v374: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:38.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:37 smithi186 ceph-mon[32797]: pgmap v374: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:38.615 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T17:55:38.867 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:40.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:39 smithi097 ceph-mon[32419]: pgmap v375: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:40.168 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:39 smithi031 ceph-mon[29164]: pgmap v375: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:40.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:39 smithi186 ceph-mon[32797]: pgmap v375: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.511+0000 7fc13ab01700 1 -- 172.21.15.31:0/2590471632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc134103470 msgr2=0x7fc1341038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.511+0000 7fc13ab01700 1 --2- 172.21.15.31:0/2590471632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc134103470 0x7fc1341038f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc120009990 tx=0x7fc120009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.511+0000 7fc13ab01700 1 -- 172.21.15.31:0/2590471632 shutdown_connections 2024-04-26T17:55:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.511+0000 7fc13ab01700 1 --2- 172.21.15.31:0/2590471632 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1341094c0 0x7fc1341098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.511+0000 7fc13ab01700 1 --2- 172.21.15.31:0/2590471632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc134103e30 0x7fc134110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.511+0000 7fc13ab01700 1 --2- 172.21.15.31:0/2590471632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc134103470 0x7fc1341038f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.511+0000 7fc13ab01700 1 -- 172.21.15.31:0/2590471632 >> 172.21.15.31:0/2590471632 conn(0x7fc13407a580 msgr2=0x7fc13407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.511+0000 7fc13ab01700 1 -- 172.21.15.31:0/2590471632 shutdown_connections 2024-04-26T17:55:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.511+0000 7fc13ab01700 1 -- 172.21.15.31:0/2590471632 wait complete. 2024-04-26T17:55:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.511+0000 7fc13ab01700 1 Processor -- start 2024-04-26T17:55:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.511+0000 7fc13ab01700 1 -- start start 2024-04-26T17:55:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc13ab01700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc134103470 0x7fc13419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc13ab01700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc134103e30 0x7fc13419fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc13ab01700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1341094c0 0x7fc134199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc13ab01700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc1341135c0 con 0x7fc134103e30 2024-04-26T17:55:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc13ab01700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc134113440 con 0x7fc134103470 2024-04-26T17:55:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc13ab01700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc134113740 con 0x7fc1341094c0 2024-04-26T17:55:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc133fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc134103e30 0x7fc13419fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc133fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc134103e30 0x7fc13419fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52694/0 (socket says 172.21.15.31:52694) 2024-04-26T17:55:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc133fff700 1 -- 172.21.15.31:0/3819578913 learned_addr learned my addr 172.21.15.31:0/3819578913 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc13889d700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc134103470 0x7fc13419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc13909e700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1341094c0 0x7fc134199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc133fff700 1 -- 172.21.15.31:0/3819578913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1341094c0 msgr2=0x7fc134199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc133fff700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1341094c0 0x7fc134199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc133fff700 1 -- 172.21.15.31:0/3819578913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc134103470 msgr2=0x7fc13419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:40.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc133fff700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc134103470 0x7fc13419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc133fff700 1 -- 172.21.15.31:0/3819578913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc13419a0a0 con 0x7fc134103e30 2024-04-26T17:55:40.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc13909e700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1341094c0 0x7fc134199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:55:40.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc133fff700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc134103e30 0x7fc13419fcc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc1200057c0 tx=0x7fc1200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:40.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc131ffb700 1 -- 172.21.15.31:0/3819578913 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc120027070 con 0x7fc134103e30 2024-04-26T17:55:40.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.512+0000 7fc131ffb700 1 -- 172.21.15.31:0/3819578913 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc120003f20 con 0x7fc134103e30 2024-04-26T17:55:40.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.513+0000 7fc13ab01700 1 -- 172.21.15.31:0/3819578913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc13419a330 con 0x7fc134103e30 2024-04-26T17:55:40.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.513+0000 7fc131ffb700 1 -- 172.21.15.31:0/3819578913 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc120022c30 con 0x7fc134103e30 2024-04-26T17:55:40.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.513+0000 7fc13ab01700 1 -- 172.21.15.31:0/3819578913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1341a6650 con 0x7fc134103e30 2024-04-26T17:55:40.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.515+0000 7fc131ffb700 1 -- 172.21.15.31:0/3819578913 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc12002a070 con 0x7fc134103e30 2024-04-26T17:55:40.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.516+0000 7fc131ffb700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fc1240a3c40 0x7fc1240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.516+0000 7fc13889d700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fc1240a3c40 0x7fc1240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.516+0000 7fc131ffb700 1 -- 172.21.15.31:0/3819578913 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fc1200d12f0 con 0x7fc134103e30 2024-04-26T17:55:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.516+0000 7fc13889d700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fc1240a3c40 0x7fc1240a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc13419aa80 tx=0x7fc12800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.516+0000 7fc145393b80 1 Processor -- start 2024-04-26T17:55:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.516+0000 7fc145393b80 1 -- start start 2024-04-26T17:55:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc145393b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aed9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc145393b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aeda3f0 0x55d70aed3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc145393b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed42d0 0x55d70aed4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc145393b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d70ae266f0 con 0x55d70aed9ad0 2024-04-26T17:55:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc145393b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55d70adc5590 con 0x55d70aed42d0 2024-04-26T17:55:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc145393b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d70ae226b0 con 0x55d70aeda3f0 2024-04-26T17:55:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc13889d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aed9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc13889d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aed9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52708/0 (socket says 172.21.15.31:52708) 2024-04-26T17:55:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc13889d700 1 -- 172.21.15.31:0/347077283 learned_addr learned my addr 172.21.15.31:0/347077283 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:40.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc133fff700 1 --2- 172.21.15.31:0/347077283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aeda3f0 0x55d70aed3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc13909e700 1 --2- 172.21.15.31:0/347077283 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed42d0 0x55d70aed4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc13889d700 1 -- 172.21.15.31:0/347077283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aeda3f0 msgr2=0x55d70aed3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:40.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc13889d700 1 --2- 172.21.15.31:0/347077283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aeda3f0 0x55d70aed3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc13889d700 1 -- 172.21.15.31:0/347077283 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed42d0 msgr2=0x55d70aed4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:40.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc13889d700 1 --2- 172.21.15.31:0/347077283 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed42d0 0x55d70aed4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc13889d700 1 -- 172.21.15.31:0/347077283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d70ad73880 con 0x55d70aed9ad0 2024-04-26T17:55:40.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc13889d700 1 --2- 172.21.15.31:0/347077283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aed9eb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc128010e00 tx=0x7fc128013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2e3f82c3df7a03 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:40.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.517+0000 7fc11effd700 1 -- 172.21.15.31:0/347077283 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc128011bb0 con 0x55d70aed9ad0 2024-04-26T17:55:40.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.518+0000 7fc11effd700 1 -- 172.21.15.31:0/347077283 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc128011d50 con 0x55d70aed9ad0 2024-04-26T17:55:40.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.518+0000 7fc11effd700 1 -- 172.21.15.31:0/347077283 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12801c6a0 con 0x55d70aed9ad0 2024-04-26T17:55:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.518+0000 7fc145393b80 1 -- 172.21.15.31:0/347077283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 msgr2=0x55d70aed9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.518+0000 7fc145393b80 1 --2- 172.21.15.31:0/347077283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aed9eb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc128010e00 tx=0x7fc128013000 comp rx=0 tx=0).stop 2024-04-26T17:55:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.518+0000 7fc145393b80 1 -- 172.21.15.31:0/347077283 shutdown_connections 2024-04-26T17:55:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.518+0000 7fc145393b80 1 --2- 172.21.15.31:0/347077283 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed42d0 0x55d70aed4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.518+0000 7fc145393b80 1 --2- 172.21.15.31:0/347077283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aed9eb0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.518+0000 7fc145393b80 1 --2- 172.21.15.31:0/347077283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aeda3f0 0x55d70aed3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.518+0000 7fc145393b80 1 -- 172.21.15.31:0/347077283 >> 172.21.15.31:0/347077283 conn(0x55d70ae00cf0 msgr2=0x55d70aecf1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.518+0000 7fc145393b80 1 -- 172.21.15.31:0/347077283 shutdown_connections 2024-04-26T17:55:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.518+0000 7fc145393b80 1 -- 172.21.15.31:0/347077283 wait complete. 2024-04-26T17:55:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.519+0000 7fc145393b80 1 -- start start 2024-04-26T17:55:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.519+0000 7fc145393b80 1 Processor -- start 2024-04-26T17:55:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.519+0000 7fc145393b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed42d0 0x55d70aee0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.519+0000 7fc145393b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aedd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.519+0000 7fc145393b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aeda3f0 0x55d70aedd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.519+0000 7fc145393b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d70aee0f10 con 0x55d70aed9ad0 2024-04-26T17:55:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.519+0000 7fc145393b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55d70ad8c900 con 0x55d70aed42d0 2024-04-26T17:55:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.519+0000 7fc145393b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d70aee1090 con 0x55d70aeda3f0 2024-04-26T17:55:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.519+0000 7fc145393b80 10 client.0 _ll_register_callbacks cb 0x55d70adba530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:55:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.519+0000 7fc133fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aedd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.519+0000 7fc133fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aedd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52716/0 (socket says 172.21.15.31:52716) 2024-04-26T17:55:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.519+0000 7fc133fff700 1 -- 172.21.15.31:0/1437347790 learned_addr learned my addr 172.21.15.31:0/1437347790 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.519+0000 7fc13889d700 1 --2- 172.21.15.31:0/1437347790 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed42d0 0x55d70aee0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc13909e700 1 --2- 172.21.15.31:0/1437347790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aeda3f0 0x55d70aedd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc133fff700 1 -- 172.21.15.31:0/1437347790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aeda3f0 msgr2=0x55d70aedd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc133fff700 1 --2- 172.21.15.31:0/1437347790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aeda3f0 0x55d70aedd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc133fff700 1 -- 172.21.15.31:0/1437347790 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed42d0 msgr2=0x55d70aee0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc133fff700 1 --2- 172.21.15.31:0/1437347790 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed42d0 0x55d70aee0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc133fff700 1 -- 172.21.15.31:0/1437347790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d70aede070 con 0x55d70aed9ad0 2024-04-26T17:55:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc13889d700 1 --2- 172.21.15.31:0/1437347790 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed42d0 0x55d70aee0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:55:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc133fff700 1 --2- 172.21.15.31:0/1437347790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aedd480 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc1200cf190 tx=0x7fc12000b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e9e8e2c7ca3538 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc113fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:55:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc113fff700 1 -- 172.21.15.31:0/1437347790 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1200a1a40 con 0x55d70aed9ad0 2024-04-26T17:55:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc113fff700 1 -- 172.21.15.31:0/1437347790 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fc1200040d0 con 0x55d70aed9ad0 2024-04-26T17:55:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc145393b80 1 -- 172.21.15.31:0/1437347790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc128011d50 con 0x55d70aed9ad0 2024-04-26T17:55:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc113fff700 1 -- 172.21.15.31:0/1437347790 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc1200bb050 con 0x55d70aed9ad0 2024-04-26T17:55:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc145393b80 10 client.14973 fetch_fsmap learned FSMap version 25 2024-04-26T17:55:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc145393b80 1 -- 172.21.15.31:0/1437347790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d70aeeb970 con 0x55d70aed9ad0 2024-04-26T17:55:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.520+0000 7fc113fff700 1 -- 172.21.15.31:0/1437347790 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc128011d50 con 0x55d70aed9ad0 2024-04-26T17:55:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.521+0000 7fc113fff700 1 -- 172.21.15.31:0/1437347790 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fc1200a5e70 con 0x55d70aed9ad0 2024-04-26T17:55:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.521+0000 7fc145393b80 10 client.14973 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T17:55:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.521+0000 7fc145393b80 10 client.14973 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T17:55:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.521+0000 7fc145393b80 20 client.14973 populate_metadata read hostname 'smithi031' 2024-04-26T17:55:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.521+0000 7fc145393b80 1 --2- 172.21.15.31:0/1437347790 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x55d70aeedf30 0x55d70aeef3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.521+0000 7fc145393b80 4 client.14973 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:55:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.521+0000 7fc145393b80 1 -- 172.21.15.31:0/1437347790 --> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d70ae340d0 con 0x55d70aeedf30 2024-04-26T17:55:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.521+0000 7fc13889d700 1 --2- 172.21.15.31:0/1437347790 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x55d70aeedf30 0x55d70aeef3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.521+0000 7fc113fff700 10 client.14973 ms_handle_connect on v2:172.21.15.186:6838/1587952500 2024-04-26T17:55:40.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.521+0000 7fc13889d700 1 --2- 172.21.15.31:0/1437347790 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x55d70aeedf30 0x55d70aeef3a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6207df635d97e86a server_cookie=40488e6bb5db761c in_seq=0 out_seq=0 2024-04-26T17:55:40.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.522+0000 7fc113fff700 1 -- 172.21.15.31:0/1437347790 <== mds.1 v2:172.21.15.186:6838/1587952500 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d70ae340d0 con 0x55d70aeedf30 2024-04-26T17:55:40.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.522+0000 7fc113fff700 10 client.14973 handle_command_reply: tid=0 2024-04-26T17:55:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.522+0000 7fc145393b80 1 client.14973 shutdown 2024-04-26T17:55:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.522+0000 7fc145393b80 1 -- 172.21.15.31:0/1437347790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 msgr2=0x55d70aedd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.522+0000 7fc145393b80 1 --2- 172.21.15.31:0/1437347790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aedd480 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc1200cf190 tx=0x7fc12000b540 comp rx=0 tx=0).stop 2024-04-26T17:55:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.523+0000 7fc145393b80 1 -- 172.21.15.31:0/1437347790 shutdown_connections 2024-04-26T17:55:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.523+0000 7fc145393b80 1 --2- 172.21.15.31:0/1437347790 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x55d70aeedf30 0x55d70aeef3a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.523+0000 7fc145393b80 1 --2- 172.21.15.31:0/1437347790 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed42d0 0x55d70aee0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.523+0000 7fc145393b80 1 --2- 172.21.15.31:0/1437347790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aeda3f0 0x55d70aedd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.523+0000 7fc145393b80 1 --2- 172.21.15.31:0/1437347790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aedd480 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.523+0000 7fc145393b80 1 -- 172.21.15.31:0/1437347790 >> 172.21.15.31:0/1437347790 conn(0x55d70ae00cf0 msgr2=0x55d70aeceda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.523+0000 7fc113fff700 0 client.14973 ms_handle_reset on v2:172.21.15.186:6838/1587952500 2024-04-26T17:55:40.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.523+0000 7fc145393b80 1 -- 172.21.15.31:0/1437347790 shutdown_connections 2024-04-26T17:55:40.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.523+0000 7fc145393b80 1 -- 172.21.15.31:0/1437347790 wait complete. 2024-04-26T17:55:40.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.523+0000 7fc145393b80 20 client.14973 trim_cache size 0 max 16384 2024-04-26T17:55:40.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.543+0000 7fc145393b80 1 Processor -- start 2024-04-26T17:55:40.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.543+0000 7fc145393b80 1 -- start start 2024-04-26T17:55:40.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc145393b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aed42d0 0x55d70aed3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc145393b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aedd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc145393b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aeda3f0 0x55d70aedd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc145393b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d70aeed6e0 con 0x55d70aed9ad0 2024-04-26T17:55:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc145393b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55d70aeeddb0 con 0x55d70aeda3f0 2024-04-26T17:55:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc145393b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d70aeee480 con 0x55d70aed42d0 2024-04-26T17:55:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc133fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aedd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc133fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aedd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52730/0 (socket says 172.21.15.31:52730) 2024-04-26T17:55:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc133fff700 1 -- 172.21.15.31:0/2042502579 learned_addr learned my addr 172.21.15.31:0/2042502579 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc13889d700 1 --2- 172.21.15.31:0/2042502579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aed42d0 0x55d70aed3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc133fff700 1 -- 172.21.15.31:0/2042502579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aed42d0 msgr2=0x55d70aed3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc133fff700 1 --2- 172.21.15.31:0/2042502579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aed42d0 0x55d70aed3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc133fff700 1 -- 172.21.15.31:0/2042502579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aeda3f0 msgr2=0x55d70aedd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc133fff700 1 --2- 172.21.15.31:0/2042502579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aeda3f0 0x55d70aedd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc133fff700 1 -- 172.21.15.31:0/2042502579 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d70aecf6f0 con 0x55d70aed9ad0 2024-04-26T17:55:40.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc13889d700 1 --2- 172.21.15.31:0/2042502579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aed42d0 0x55d70aed3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:55:40.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc133fff700 1 --2- 172.21.15.31:0/2042502579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aedd440 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc120099af0 tx=0x7fc120009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf732894c510cba4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:40.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc11e7fc700 1 -- 172.21.15.31:0/2042502579 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1200a1a40 con 0x55d70aed9ad0 2024-04-26T17:55:40.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc11e7fc700 1 -- 172.21.15.31:0/2042502579 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc120067070 con 0x55d70aed9ad0 2024-04-26T17:55:40.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.544+0000 7fc11e7fc700 1 -- 172.21.15.31:0/2042502579 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12009ba60 con 0x55d70aed9ad0 2024-04-26T17:55:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.545+0000 7fc145393b80 1 -- 172.21.15.31:0/2042502579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 msgr2=0x55d70aedd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.545+0000 7fc145393b80 1 --2- 172.21.15.31:0/2042502579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aedd440 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc120099af0 tx=0x7fc120009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.545+0000 7fc145393b80 1 -- 172.21.15.31:0/2042502579 shutdown_connections 2024-04-26T17:55:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.545+0000 7fc145393b80 1 --2- 172.21.15.31:0/2042502579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aeda3f0 0x55d70aedd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.545+0000 7fc145393b80 1 --2- 172.21.15.31:0/2042502579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aed42d0 0x55d70aed3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.545+0000 7fc145393b80 1 --2- 172.21.15.31:0/2042502579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aed9ad0 0x55d70aedd440 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.545+0000 7fc145393b80 1 -- 172.21.15.31:0/2042502579 >> 172.21.15.31:0/2042502579 conn(0x55d70ae00cf0 msgr2=0x55d70aeeccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.545+0000 7fc145393b80 1 -- 172.21.15.31:0/2042502579 shutdown_connections 2024-04-26T17:55:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.545+0000 7fc145393b80 1 -- 172.21.15.31:0/2042502579 wait complete. 2024-04-26T17:55:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.545+0000 7fc145393b80 1 -- start start 2024-04-26T17:55:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.545+0000 7fc145393b80 1 Processor -- start 2024-04-26T17:55:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc145393b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aed42d0 0x55d70aef8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc145393b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed9ad0 0x55d70aef8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc145393b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aeda3f0 0x55d70aef3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc145393b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d70aeef220 con 0x55d70aeda3f0 2024-04-26T17:55:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc145393b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55d70aeeeb50 con 0x55d70aed9ad0 2024-04-26T17:55:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc145393b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d70aeef8f0 con 0x55d70aed42d0 2024-04-26T17:55:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc13909e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aeda3f0 0x55d70aef3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc145393b80 10 client.0 _ll_register_callbacks cb 0x55d70adba530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:55:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc13909e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aeda3f0 0x55d70aef3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52736/0 (socket says 172.21.15.31:52736) 2024-04-26T17:55:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc13909e700 1 -- 172.21.15.31:0/3674359120 learned_addr learned my addr 172.21.15.31:0/3674359120 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc13889d700 1 --2- 172.21.15.31:0/3674359120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aed42d0 0x55d70aef8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc133fff700 1 --2- 172.21.15.31:0/3674359120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed9ad0 0x55d70aef8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc13909e700 1 -- 172.21.15.31:0/3674359120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aed42d0 msgr2=0x55d70aef8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc13909e700 1 --2- 172.21.15.31:0/3674359120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aed42d0 0x55d70aef8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc13909e700 1 -- 172.21.15.31:0/3674359120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed9ad0 msgr2=0x55d70aef8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc13909e700 1 --2- 172.21.15.31:0/3674359120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed9ad0 0x55d70aef8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc13909e700 1 -- 172.21.15.31:0/3674359120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d70aef4100 con 0x55d70aeda3f0 2024-04-26T17:55:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.546+0000 7fc13889d700 1 --2- 172.21.15.31:0/3674359120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aed42d0 0x55d70aef8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:55:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc13909e700 1 --2- 172.21.15.31:0/3674359120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aeda3f0 0x55d70aef3870 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0119e0 tx=0x7fc12c011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13bf96dd3f954a70 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc113fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:55:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc113fff700 1 -- 172.21.15.31:0/3674359120 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12c01b020 con 0x55d70aeda3f0 2024-04-26T17:55:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc113fff700 1 -- 172.21.15.31:0/3674359120 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fc12c00f6c0 con 0x55d70aeda3f0 2024-04-26T17:55:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc145393b80 1 -- 172.21.15.31:0/3674359120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc120067070 con 0x55d70aeda3f0 2024-04-26T17:55:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc113fff700 1 -- 172.21.15.31:0/3674359120 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc12c00f8c0 con 0x55d70aeda3f0 2024-04-26T17:55:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc145393b80 10 client.14979 fetch_fsmap learned FSMap version 25 2024-04-26T17:55:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc145393b80 1 -- 172.21.15.31:0/3674359120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d70af55bf0 con 0x55d70aeda3f0 2024-04-26T17:55:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc113fff700 1 -- 172.21.15.31:0/3674359120 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc120067070 con 0x55d70aeda3f0 2024-04-26T17:55:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc113fff700 1 -- 172.21.15.31:0/3674359120 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fc12c015430 con 0x55d70aeda3f0 2024-04-26T17:55:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc145393b80 10 client.14979 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T17:55:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc145393b80 10 client.14979 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T17:55:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc145393b80 20 client.14979 populate_metadata read hostname 'smithi031' 2024-04-26T17:55:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc145393b80 1 --2- 172.21.15.31:0/3674359120 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x55d70af5a810 0x55d70af59520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc145393b80 4 client.14979 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T17:55:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.547+0000 7fc145393b80 1 -- 172.21.15.31:0/3674359120 --> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] -- command(tid 0: {"prefix": "status"}) -- 0x55d70ae2e170 con 0x55d70af5a810 2024-04-26T17:55:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.548+0000 7fc13889d700 1 --2- 172.21.15.31:0/3674359120 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x55d70af5a810 0x55d70af59520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.548+0000 7fc13889d700 1 --2- 172.21.15.31:0/3674359120 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x55d70af5a810 0x55d70af59520 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7c026c0695fc9679 server_cookie=8e0b76296cc9c1be in_seq=0 out_seq=0 2024-04-26T17:55:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.548+0000 7fc113fff700 10 client.14979 ms_handle_connect on v2:172.21.15.186:6838/1587952500 2024-04-26T17:55:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.548+0000 7fc113fff700 1 -- 172.21.15.31:0/3674359120 <== mds.1 v2:172.21.15.186:6838/1587952500 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55d70ae2e170 con 0x55d70af5a810 2024-04-26T17:55:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.548+0000 7fc113fff700 10 client.14979 handle_command_reply: tid=0 2024-04-26T17:55:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.548+0000 7fc145393b80 1 client.14979 shutdown 2024-04-26T17:55:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.548+0000 7fc145393b80 1 -- 172.21.15.31:0/3674359120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aeda3f0 msgr2=0x55d70aef3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.548+0000 7fc145393b80 1 --2- 172.21.15.31:0/3674359120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aeda3f0 0x55d70aef3870 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0119e0 tx=0x7fc12c011cf0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.548+0000 7fc145393b80 1 -- 172.21.15.31:0/3674359120 shutdown_connections 2024-04-26T17:55:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.548+0000 7fc145393b80 1 --2- 172.21.15.31:0/3674359120 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x55d70af5a810 0x55d70af59520 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.549+0000 7fc145393b80 1 --2- 172.21.15.31:0/3674359120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d70aeda3f0 0x55d70aef3870 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.549+0000 7fc145393b80 1 --2- 172.21.15.31:0/3674359120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d70aed42d0 0x55d70aef8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.549+0000 7fc145393b80 1 --2- 172.21.15.31:0/3674359120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d70aed9ad0 0x55d70aef8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.549+0000 7fc145393b80 1 -- 172.21.15.31:0/3674359120 >> 172.21.15.31:0/3674359120 conn(0x55d70ae00cf0 msgr2=0x55d70aedd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.549+0000 7fc113fff700 0 client.14979 ms_handle_reset on v2:172.21.15.186:6838/1587952500 2024-04-26T17:55:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.549+0000 7fc145393b80 1 -- 172.21.15.31:0/3674359120 shutdown_connections 2024-04-26T17:55:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.549+0000 7fc145393b80 1 -- 172.21.15.31:0/3674359120 wait complete. 2024-04-26T17:55:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.549+0000 7fc145393b80 20 client.14979 trim_cache size 0 max 16384 2024-04-26T17:55:40.555 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T17:55:40.555 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "7e53e050-03f4-11ef-bc93-c7b262605968", 2024-04-26T17:55:40.555 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 1, 2024-04-26T17:55:40.555 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24457, 2024-04-26T17:55:40.555 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-26T17:55:40.555 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-26T17:55:40.555 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-26T17:55:40.555 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 141.87979855699999, 2024-04-26T17:55:40.555 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 25, 2024-04-26T17:55:40.555 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 90, 2024-04-26T17:55:40.556 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T17:55:40.556 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 180.643418048 2024-04-26T17:55:40.556 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T17:55:40.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.549+0000 7fc13ab01700 1 -- 172.21.15.31:0/3819578913 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fc1240a3c40 msgr2=0x7fc1240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:40.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.549+0000 7fc13ab01700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fc1240a3c40 0x7fc1240a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc13419aa80 tx=0x7fc12800b410 comp rx=0 tx=0).stop 2024-04-26T17:55:40.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.549+0000 7fc13ab01700 1 -- 172.21.15.31:0/3819578913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc134103e30 msgr2=0x7fc13419fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:40.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.549+0000 7fc13ab01700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc134103e30 0x7fc13419fcc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc1200057c0 tx=0x7fc1200057f0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.550+0000 7fc13ab01700 1 -- 172.21.15.31:0/3819578913 shutdown_connections 2024-04-26T17:55:40.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.550+0000 7fc13ab01700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fc1240a3c40 0x7fc1240a6100 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.550+0000 7fc13ab01700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc134103470 0x7fc13419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.550+0000 7fc13ab01700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1341094c0 0x7fc134199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.550+0000 7fc13ab01700 1 --2- 172.21.15.31:0/3819578913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc134103e30 0x7fc13419fcc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:40.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.550+0000 7fc13ab01700 1 -- 172.21.15.31:0/3819578913 >> 172.21.15.31:0/3819578913 conn(0x7fc13407a580 msgr2=0x7fc1341022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:40.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.550+0000 7fc13ab01700 1 -- 172.21.15.31:0/3819578913 shutdown_connections 2024-04-26T17:55:40.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:40.550+0000 7fc13ab01700 1 -- 172.21.15.31:0/3819578913 wait complete. 2024-04-26T17:55:41.389 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T17:55:41.648 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:42.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:41 smithi031 ceph-mon[29164]: pgmap v376: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:42.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:41 smithi097 ceph-mon[32419]: pgmap v376: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:42.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:41 smithi186 ceph-mon[32797]: pgmap v376: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:43.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.486+0000 7faf48e38700 1 -- 172.21.15.31:0/2201965461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf441021b0 msgr2=0x7faf44109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:43.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.486+0000 7faf48e38700 1 --2- 172.21.15.31:0/2201965461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf441021b0 0x7faf44109200 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7faf380099c0 tx=0x7faf38009cd0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.486+0000 7faf48e38700 1 -- 172.21.15.31:0/2201965461 shutdown_connections 2024-04-26T17:55:43.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.486+0000 7faf48e38700 1 --2- 172.21.15.31:0/2201965461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf44101800 0x7faf44101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.486+0000 7faf48e38700 1 --2- 172.21.15.31:0/2201965461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf441098e0 0x7faf44110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.486+0000 7faf48e38700 1 --2- 172.21.15.31:0/2201965461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf441021b0 0x7faf44109200 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.486+0000 7faf48e38700 1 -- 172.21.15.31:0/2201965461 >> 172.21.15.31:0/2201965461 conn(0x7faf44078520 msgr2=0x7faf440ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.486+0000 7faf48e38700 1 -- 172.21.15.31:0/2201965461 shutdown_connections 2024-04-26T17:55:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.486+0000 7faf48e38700 1 -- 172.21.15.31:0/2201965461 wait complete. 2024-04-26T17:55:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.487+0000 7faf48e38700 1 Processor -- start 2024-04-26T17:55:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.487+0000 7faf48e38700 1 -- start start 2024-04-26T17:55:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.487+0000 7faf48e38700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf44101800 0x7faf44199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.487+0000 7faf48e38700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf441021b0 0x7faf4419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.487+0000 7faf48e38700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf441098e0 0x7faf4419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.487+0000 7faf48e38700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faf441135b0 con 0x7faf441098e0 2024-04-26T17:55:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.487+0000 7faf48e38700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7faf44113430 con 0x7faf441021b0 2024-04-26T17:55:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.487+0000 7faf48e38700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faf44113730 con 0x7faf44101800 2024-04-26T17:55:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.487+0000 7faf42d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf441098e0 0x7faf4419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.487+0000 7faf41d9b700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf441021b0 0x7faf4419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.487+0000 7faf42d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf441098e0 0x7faf4419e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52750/0 (socket says 172.21.15.31:52750) 2024-04-26T17:55:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.487+0000 7faf42d9d700 1 -- 172.21.15.31:0/2239321564 learned_addr learned my addr 172.21.15.31:0/2239321564 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.487+0000 7faf4259c700 1 --2- 172.21.15.31:0/2239321564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf44101800 0x7faf44199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.488+0000 7faf42d9d700 1 -- 172.21.15.31:0/2239321564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf44101800 msgr2=0x7faf44199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.488+0000 7faf42d9d700 1 --2- 172.21.15.31:0/2239321564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf44101800 0x7faf44199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.488+0000 7faf42d9d700 1 -- 172.21.15.31:0/2239321564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf441021b0 msgr2=0x7faf4419a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.488+0000 7faf42d9d700 1 --2- 172.21.15.31:0/2239321564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf441021b0 0x7faf4419a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.488+0000 7faf42d9d700 1 -- 172.21.15.31:0/2239321564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf4419eb00 con 0x7faf441098e0 2024-04-26T17:55:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.488+0000 7faf41d9b700 1 --2- 172.21.15.31:0/2239321564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf441021b0 0x7faf4419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:55:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.488+0000 7faf42d9d700 1 --2- 172.21.15.31:0/2239321564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf441098e0 0x7faf4419e400 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7faf3c007d70 tx=0x7faf3c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.488+0000 7faf337fe700 1 -- 172.21.15.31:0/2239321564 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c016020 con 0x7faf441098e0 2024-04-26T17:55:43.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.488+0000 7faf337fe700 1 -- 172.21.15.31:0/2239321564 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf3c00ac90 con 0x7faf441098e0 2024-04-26T17:55:43.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.488+0000 7faf48e38700 1 -- 172.21.15.31:0/2239321564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf4419edf0 con 0x7faf441098e0 2024-04-26T17:55:43.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.488+0000 7faf337fe700 1 -- 172.21.15.31:0/2239321564 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c0115e0 con 0x7faf441098e0 2024-04-26T17:55:43.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.488+0000 7faf48e38700 1 -- 172.21.15.31:0/2239321564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf441a66b0 con 0x7faf441098e0 2024-04-26T17:55:43.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.491+0000 7faf337fe700 1 -- 172.21.15.31:0/2239321564 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7faf3c019020 con 0x7faf441098e0 2024-04-26T17:55:43.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.492+0000 7faf337fe700 1 --2- 172.21.15.31:0/2239321564 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7faf2c0a3c40 0x7faf2c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.492+0000 7faf4259c700 1 --2- 172.21.15.31:0/2239321564 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7faf2c0a3c40 0x7faf2c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.492+0000 7faf337fe700 1 -- 172.21.15.31:0/2239321564 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7faf3c0d0350 con 0x7faf441098e0 2024-04-26T17:55:43.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.492+0000 7faf4259c700 1 --2- 172.21.15.31:0/2239321564 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7faf2c0a3c40 0x7faf2c0a6100 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faf34009c60 tx=0x7faf34009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:43.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.492+0000 7faf536cab80 1 Processor -- start 2024-04-26T17:55:43.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf536cab80 1 -- start start 2024-04-26T17:55:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf536cab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c3068cc0 0x55b9c30690a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf536cab80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c30695e0 0x55b9c3063160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf536cab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3063780 0x55b9c3063be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf536cab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b9c2fb17e0 con 0x55b9c3068cc0 2024-04-26T17:55:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf536cab80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b9c2f178e0 con 0x55b9c30695e0 2024-04-26T17:55:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf536cab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b9c2fad7a0 con 0x55b9c3063780 2024-04-26T17:55:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf4259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c3068cc0 0x55b9c30690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf4259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c3068cc0 0x55b9c30690a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52756/0 (socket says 172.21.15.31:52756) 2024-04-26T17:55:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf4259c700 1 -- 172.21.15.31:0/76454169 learned_addr learned my addr 172.21.15.31:0/76454169 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf41d9b700 1 --2- 172.21.15.31:0/76454169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c30695e0 0x55b9c3063160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf42d9d700 1 --2- 172.21.15.31:0/76454169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3063780 0x55b9c3063be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf4259c700 1 -- 172.21.15.31:0/76454169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3063780 msgr2=0x55b9c3063be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf4259c700 1 --2- 172.21.15.31:0/76454169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3063780 0x55b9c3063be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf4259c700 1 -- 172.21.15.31:0/76454169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c30695e0 msgr2=0x55b9c3063160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf4259c700 1 --2- 172.21.15.31:0/76454169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c30695e0 0x55b9c3063160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.493+0000 7faf4259c700 1 -- 172.21.15.31:0/76454169 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9c2efe880 con 0x55b9c3068cc0 2024-04-26T17:55:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.494+0000 7faf4259c700 1 --2- 172.21.15.31:0/76454169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c3068cc0 0x55b9c30690a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7faf3400e4e0 tx=0x7faf34014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f458eb26a20c7967 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.494+0000 7faf30ff9700 1 -- 172.21.15.31:0/76454169 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3400ee00 con 0x55b9c3068cc0 2024-04-26T17:55:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.494+0000 7faf30ff9700 1 -- 172.21.15.31:0/76454169 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf3401c8c0 con 0x55b9c3068cc0 2024-04-26T17:55:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.494+0000 7faf30ff9700 1 -- 172.21.15.31:0/76454169 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3401d9c0 con 0x55b9c3068cc0 2024-04-26T17:55:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 -- 172.21.15.31:0/76454169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c3068cc0 msgr2=0x55b9c30690a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 --2- 172.21.15.31:0/76454169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c3068cc0 0x55b9c30690a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7faf3400e4e0 tx=0x7faf34014000 comp rx=0 tx=0).stop 2024-04-26T17:55:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 -- 172.21.15.31:0/76454169 shutdown_connections 2024-04-26T17:55:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 --2- 172.21.15.31:0/76454169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3063780 0x55b9c3063be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 --2- 172.21.15.31:0/76454169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c3068cc0 0x55b9c30690a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 --2- 172.21.15.31:0/76454169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c30695e0 0x55b9c3063160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 -- 172.21.15.31:0/76454169 >> 172.21.15.31:0/76454169 conn(0x55b9c2f8d570 msgr2=0x55b9c3059900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:43.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 -- 172.21.15.31:0/76454169 shutdown_connections 2024-04-26T17:55:43.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 -- 172.21.15.31:0/76454169 wait complete. 2024-04-26T17:55:43.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 -- start start 2024-04-26T17:55:43.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 Processor -- start 2024-04-26T17:55:43.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3063780 0x55b9c306f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c3068cc0 0x55b9c306c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c30695e0 0x55b9c306cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b9c30700c0 con 0x55b9c30695e0 2024-04-26T17:55:43.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b9c2f6ba30 con 0x55b9c3068cc0 2024-04-26T17:55:43.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.495+0000 7faf536cab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b9c3070240 con 0x55b9c3063780 2024-04-26T17:55:43.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf536cab80 10 client.0 _ll_register_callbacks cb 0x55b9c2f49900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:55:43.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf42d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c30695e0 0x55b9c306cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf42d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c30695e0 0x55b9c306cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52760/0 (socket says 172.21.15.31:52760) 2024-04-26T17:55:43.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf42d9d700 1 -- 172.21.15.31:0/2528725000 learned_addr learned my addr 172.21.15.31:0/2528725000 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:43.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf4259c700 1 --2- 172.21.15.31:0/2528725000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3063780 0x55b9c306f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf41d9b700 1 --2- 172.21.15.31:0/2528725000 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c3068cc0 0x55b9c306c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf42d9d700 1 -- 172.21.15.31:0/2528725000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3063780 msgr2=0x55b9c306f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:43.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf42d9d700 1 --2- 172.21.15.31:0/2528725000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3063780 0x55b9c306f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf42d9d700 1 -- 172.21.15.31:0/2528725000 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c3068cc0 msgr2=0x55b9c306c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:43.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf42d9d700 1 --2- 172.21.15.31:0/2528725000 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c3068cc0 0x55b9c306c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf42d9d700 1 -- 172.21.15.31:0/2528725000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9c2f6f430 con 0x55b9c30695e0 2024-04-26T17:55:43.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf42d9d700 1 --2- 172.21.15.31:0/2528725000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c30695e0 0x55b9c306cb80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7faf3c006130 tx=0x7faf3c09c030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f7cc008d0567c54 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:43.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf29ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:55:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf29ffb700 1 -- 172.21.15.31:0/2528725000 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c098040 con 0x55b9c30695e0 2024-04-26T17:55:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf29ffb700 1 -- 172.21.15.31:0/2528725000 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7faf3c0a4fe0 con 0x55b9c30695e0 2024-04-26T17:55:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.496+0000 7faf536cab80 1 -- 172.21.15.31:0/2528725000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf3401c8c0 con 0x55b9c30695e0 2024-04-26T17:55:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.497+0000 7faf29ffb700 1 -- 172.21.15.31:0/2528725000 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faf3c09cd00 con 0x55b9c30695e0 2024-04-26T17:55:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.497+0000 7faf536cab80 10 client.14991 fetch_fsmap learned FSMap version 25 2024-04-26T17:55:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.497+0000 7faf536cab80 1 -- 172.21.15.31:0/2528725000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b9c307a920 con 0x55b9c30695e0 2024-04-26T17:55:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.497+0000 7faf29ffb700 1 -- 172.21.15.31:0/2528725000 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf3401c8c0 con 0x55b9c30695e0 2024-04-26T17:55:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.497+0000 7faf29ffb700 1 -- 172.21.15.31:0/2528725000 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7faf3c085cb0 con 0x55b9c30695e0 2024-04-26T17:55:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.497+0000 7faf536cab80 10 client.14991 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T17:55:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.497+0000 7faf536cab80 10 client.14991 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T17:55:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.497+0000 7faf536cab80 20 client.14991 populate_metadata read hostname 'smithi031' 2024-04-26T17:55:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.497+0000 7faf536cab80 1 --2- 172.21.15.31:0/2528725000 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b9c307cea0 0x55b9c307e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.497+0000 7faf536cab80 4 client.14991 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:55:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.497+0000 7faf536cab80 1 -- 172.21.15.31:0/2528725000 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9c2fbf1c0 con 0x55b9c307cea0 2024-04-26T17:55:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.497+0000 7faf4259c700 1 --2- 172.21.15.31:0/2528725000 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b9c307cea0 0x55b9c307e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.498+0000 7faf4259c700 1 --2- 172.21.15.31:0/2528725000 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b9c307cea0 0x55b9c307e310 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a643bb9d38984a1e server_cookie=cba05c10eb7ab146 in_seq=0 out_seq=0 2024-04-26T17:55:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.498+0000 7faf29ffb700 10 client.14991 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:55:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.498+0000 7faf29ffb700 1 -- 172.21.15.31:0/2528725000 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b9c2fbf1c0 con 0x55b9c307cea0 2024-04-26T17:55:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.498+0000 7faf29ffb700 10 client.14991 handle_command_reply: tid=0 2024-04-26T17:55:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.498+0000 7faf536cab80 1 client.14991 shutdown 2024-04-26T17:55:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.499+0000 7faf536cab80 1 -- 172.21.15.31:0/2528725000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c30695e0 msgr2=0x55b9c306cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.499+0000 7faf536cab80 1 --2- 172.21.15.31:0/2528725000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c30695e0 0x55b9c306cb80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7faf3c006130 tx=0x7faf3c09c030 comp rx=0 tx=0).stop 2024-04-26T17:55:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.499+0000 7faf536cab80 1 -- 172.21.15.31:0/2528725000 shutdown_connections 2024-04-26T17:55:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.499+0000 7faf536cab80 1 --2- 172.21.15.31:0/2528725000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c30695e0 0x55b9c306cb80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.499+0000 7faf536cab80 1 --2- 172.21.15.31:0/2528725000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3063780 0x55b9c306f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.499+0000 7faf536cab80 1 --2- 172.21.15.31:0/2528725000 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b9c307cea0 0x55b9c307e310 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.499+0000 7faf536cab80 1 --2- 172.21.15.31:0/2528725000 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c3068cc0 0x55b9c306c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.499+0000 7faf536cab80 1 -- 172.21.15.31:0/2528725000 >> 172.21.15.31:0/2528725000 conn(0x55b9c2f8d570 msgr2=0x55b9c305dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:43.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.499+0000 7faf29ffb700 0 client.14991 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:55:43.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.499+0000 7faf536cab80 1 -- 172.21.15.31:0/2528725000 shutdown_connections 2024-04-26T17:55:43.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.499+0000 7faf536cab80 1 -- 172.21.15.31:0/2528725000 wait complete. 2024-04-26T17:55:43.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.499+0000 7faf536cab80 20 client.14991 trim_cache size 0 max 16384 2024-04-26T17:55:43.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.519+0000 7faf536cab80 1 Processor -- start 2024-04-26T17:55:43.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.519+0000 7faf536cab80 1 -- start start 2024-04-26T17:55:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf536cab80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c3063780 0x55b9c307c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf536cab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3068cc0 0x55b9c306cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf536cab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c30695e0 0x55b9c307bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf536cab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b9c307c650 con 0x55b9c30695e0 2024-04-26T17:55:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf536cab80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b9c307cd20 con 0x55b9c3063780 2024-04-26T17:55:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf536cab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b9c307d3f0 con 0x55b9c3068cc0 2024-04-26T17:55:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf42d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c30695e0 0x55b9c307bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf42d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c30695e0 0x55b9c307bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52776/0 (socket says 172.21.15.31:52776) 2024-04-26T17:55:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf42d9d700 1 -- 172.21.15.31:0/1694710106 learned_addr learned my addr 172.21.15.31:0/1694710106 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf4259c700 1 --2- 172.21.15.31:0/1694710106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c3063780 0x55b9c307c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf42d9d700 1 -- 172.21.15.31:0/1694710106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3068cc0 msgr2=0x55b9c306cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:43.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf42d9d700 1 --2- 172.21.15.31:0/1694710106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3068cc0 0x55b9c306cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf42d9d700 1 -- 172.21.15.31:0/1694710106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c3063780 msgr2=0x55b9c307c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:43.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf42d9d700 1 --2- 172.21.15.31:0/1694710106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c3063780 0x55b9c307c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf42d9d700 1 -- 172.21.15.31:0/1694710106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9c30561f0 con 0x55b9c30695e0 2024-04-26T17:55:43.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf4259c700 1 --2- 172.21.15.31:0/1694710106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c3063780 0x55b9c307c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:55:43.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf42d9d700 1 --2- 172.21.15.31:0/1694710106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c30695e0 0x55b9c307bde0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7faf3c09c590 tx=0x7faf3c09ab10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=601e0e502a04bdf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:43.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf2bfff700 1 -- 172.21.15.31:0/1694710106 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c0a09b0 con 0x55b9c30695e0 2024-04-26T17:55:43.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf2bfff700 1 -- 172.21.15.31:0/1694710106 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf3c0a5270 con 0x55b9c30695e0 2024-04-26T17:55:43.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.520+0000 7faf2bfff700 1 -- 172.21.15.31:0/1694710106 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c085df0 con 0x55b9c30695e0 2024-04-26T17:55:43.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.521+0000 7faf536cab80 1 -- 172.21.15.31:0/1694710106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c30695e0 msgr2=0x55b9c307bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:43.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.521+0000 7faf536cab80 1 --2- 172.21.15.31:0/1694710106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c30695e0 0x55b9c307bde0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7faf3c09c590 tx=0x7faf3c09ab10 comp rx=0 tx=0).stop 2024-04-26T17:55:43.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.521+0000 7faf536cab80 1 -- 172.21.15.31:0/1694710106 shutdown_connections 2024-04-26T17:55:43.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.521+0000 7faf536cab80 1 --2- 172.21.15.31:0/1694710106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c3063780 0x55b9c307c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.521+0000 7faf536cab80 1 --2- 172.21.15.31:0/1694710106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c30695e0 0x55b9c307bde0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.521+0000 7faf536cab80 1 --2- 172.21.15.31:0/1694710106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3068cc0 0x55b9c306cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.521+0000 7faf536cab80 1 -- 172.21.15.31:0/1694710106 >> 172.21.15.31:0/1694710106 conn(0x55b9c2f8d570 msgr2=0x55b9c3063b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.521+0000 7faf536cab80 1 -- 172.21.15.31:0/1694710106 shutdown_connections 2024-04-26T17:55:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.521+0000 7faf536cab80 1 -- 172.21.15.31:0/1694710106 wait complete. 2024-04-26T17:55:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.521+0000 7faf536cab80 1 -- start start 2024-04-26T17:55:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.521+0000 7faf536cab80 1 Processor -- start 2024-04-26T17:55:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf536cab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c3063780 0x55b9c3087790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf536cab80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3068cc0 0x55b9c3087cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf536cab80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c30695e0 0x55b9c3082740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf536cab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b9c307e190 con 0x55b9c3063780 2024-04-26T17:55:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf536cab80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b9c307dac0 con 0x55b9c30695e0 2024-04-26T17:55:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf536cab80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b9c307e860 con 0x55b9c3068cc0 2024-04-26T17:55:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf536cab80 10 client.0 _ll_register_callbacks cb 0x55b9c2f49900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:55:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf4259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c3063780 0x55b9c3087790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf4259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c3063780 0x55b9c3087790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52788/0 (socket says 172.21.15.31:52788) 2024-04-26T17:55:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf4259c700 1 -- 172.21.15.31:0/1414620086 learned_addr learned my addr 172.21.15.31:0/1414620086 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf41d9b700 1 --2- 172.21.15.31:0/1414620086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3068cc0 0x55b9c3087cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf42d9d700 1 --2- 172.21.15.31:0/1414620086 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c30695e0 0x55b9c3082740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf4259c700 1 -- 172.21.15.31:0/1414620086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3068cc0 msgr2=0x55b9c3087cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf4259c700 1 --2- 172.21.15.31:0/1414620086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3068cc0 0x55b9c3087cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf4259c700 1 -- 172.21.15.31:0/1414620086 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c30695e0 msgr2=0x55b9c3082740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:55:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf4259c700 1 --2- 172.21.15.31:0/1414620086 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c30695e0 0x55b9c3082740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf4259c700 1 -- 172.21.15.31:0/1414620086 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9c307f270 con 0x55b9c3063780 2024-04-26T17:55:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf4259c700 1 --2- 172.21.15.31:0/1414620086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c3063780 0x55b9c3087790 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7faf34005950 tx=0x7faf34005c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8df6829c2e464f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf29ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:55:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.522+0000 7faf29ffb700 1 -- 172.21.15.31:0/1414620086 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf34005040 con 0x55b9c3063780 2024-04-26T17:55:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.523+0000 7faf536cab80 1 -- 172.21.15.31:0/1414620086 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf3c0a5270 con 0x55b9c3063780 2024-04-26T17:55:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.523+0000 7faf29ffb700 1 -- 172.21.15.31:0/1414620086 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7faf340051e0 con 0x55b9c3063780 2024-04-26T17:55:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.523+0000 7faf29ffb700 1 -- 172.21.15.31:0/1414620086 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faf340053e0 con 0x55b9c3063780 2024-04-26T17:55:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.523+0000 7faf536cab80 10 client.14997 fetch_fsmap learned FSMap version 25 2024-04-26T17:55:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.523+0000 7faf536cab80 1 -- 172.21.15.31:0/1414620086 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b9c30e4740 con 0x55b9c3063780 2024-04-26T17:55:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.523+0000 7faf29ffb700 1 -- 172.21.15.31:0/1414620086 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf3c0a5270 con 0x55b9c3063780 2024-04-26T17:55:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.523+0000 7faf29ffb700 1 -- 172.21.15.31:0/1414620086 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7faf34003710 con 0x55b9c3063780 2024-04-26T17:55:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.523+0000 7faf536cab80 10 client.14997 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T17:55:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.523+0000 7faf536cab80 10 client.14997 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T17:55:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.523+0000 7faf536cab80 20 client.14997 populate_metadata read hostname 'smithi031' 2024-04-26T17:55:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.523+0000 7faf536cab80 1 --2- 172.21.15.31:0/1414620086 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b9c30e8ff0 0x55b9c30e7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.523+0000 7faf536cab80 4 client.14997 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T17:55:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.523+0000 7faf536cab80 1 -- 172.21.15.31:0/1414620086 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "status"}) -- 0x55b9c2fb9260 con 0x55b9c30e8ff0 2024-04-26T17:55:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf41d9b700 1 --2- 172.21.15.31:0/1414620086 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b9c30e8ff0 0x55b9c30e7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf41d9b700 1 --2- 172.21.15.31:0/1414620086 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b9c30e8ff0 0x55b9c30e7d00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44257208dc2e9c74 server_cookie=adf4c56489c63c2d in_seq=0 out_seq=0 2024-04-26T17:55:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf29ffb700 10 client.14997 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:55:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf29ffb700 1 -- 172.21.15.31:0/1414620086 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b9c2fb9260 con 0x55b9c30e8ff0 2024-04-26T17:55:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf29ffb700 10 client.14997 handle_command_reply: tid=0 2024-04-26T17:55:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf536cab80 1 client.14997 shutdown 2024-04-26T17:55:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf536cab80 1 -- 172.21.15.31:0/1414620086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c3063780 msgr2=0x55b9c3087790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf536cab80 1 --2- 172.21.15.31:0/1414620086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c3063780 0x55b9c3087790 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7faf34005950 tx=0x7faf34005c20 comp rx=0 tx=0).stop 2024-04-26T17:55:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf536cab80 1 -- 172.21.15.31:0/1414620086 shutdown_connections 2024-04-26T17:55:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf536cab80 1 --2- 172.21.15.31:0/1414620086 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b9c30e8ff0 0x55b9c30e7d00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf536cab80 1 --2- 172.21.15.31:0/1414620086 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b9c30695e0 0x55b9c3082740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf536cab80 1 --2- 172.21.15.31:0/1414620086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b9c3063780 0x55b9c3087790 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf29ffb700 0 client.14997 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:55:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf536cab80 1 --2- 172.21.15.31:0/1414620086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9c3068cc0 0x55b9c3087cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf536cab80 1 -- 172.21.15.31:0/1414620086 >> 172.21.15.31:0/1414620086 conn(0x55b9c2f8d570 msgr2=0x55b9c307bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:43.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf536cab80 1 -- 172.21.15.31:0/1414620086 shutdown_connections 2024-04-26T17:55:43.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.524+0000 7faf536cab80 1 -- 172.21.15.31:0/1414620086 wait complete. 2024-04-26T17:55:43.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.525+0000 7faf536cab80 20 client.14997 trim_cache size 0 max 16384 2024-04-26T17:55:43.530 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T17:55:43.530 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "7e53e050-03f4-11ef-bc93-c7b262605968", 2024-04-26T17:55:43.530 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-04-26T17:55:43.530 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24473, 2024-04-26T17:55:43.530 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-26T17:55:43.531 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-26T17:55:43.531 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-26T17:55:43.531 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 157.91124696200001, 2024-04-26T17:55:43.531 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 25, 2024-04-26T17:55:43.531 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 90, 2024-04-26T17:55:43.531 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T17:55:43.531 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 172.774953318 2024-04-26T17:55:43.531 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T17:55:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.525+0000 7faf48e38700 1 -- 172.21.15.31:0/2239321564 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7faf2c0a3c40 msgr2=0x7faf2c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.525+0000 7faf48e38700 1 --2- 172.21.15.31:0/2239321564 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7faf2c0a3c40 0x7faf2c0a6100 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faf34009c60 tx=0x7faf34009260 comp rx=0 tx=0).stop 2024-04-26T17:55:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.525+0000 7faf48e38700 1 -- 172.21.15.31:0/2239321564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf441098e0 msgr2=0x7faf4419e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.525+0000 7faf48e38700 1 --2- 172.21.15.31:0/2239321564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf441098e0 0x7faf4419e400 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7faf3c007d70 tx=0x7faf3c00a340 comp rx=0 tx=0).stop 2024-04-26T17:55:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.525+0000 7faf48e38700 1 -- 172.21.15.31:0/2239321564 shutdown_connections 2024-04-26T17:55:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.525+0000 7faf48e38700 1 --2- 172.21.15.31:0/2239321564 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7faf2c0a3c40 0x7faf2c0a6100 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.525+0000 7faf48e38700 1 --2- 172.21.15.31:0/2239321564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf441098e0 0x7faf4419e400 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.525+0000 7faf48e38700 1 --2- 172.21.15.31:0/2239321564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf44101800 0x7faf44199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.525+0000 7faf48e38700 1 --2- 172.21.15.31:0/2239321564 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf441021b0 0x7faf4419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.525+0000 7faf48e38700 1 -- 172.21.15.31:0/2239321564 >> 172.21.15.31:0/2239321564 conn(0x7faf44078520 msgr2=0x7faf44100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.525+0000 7faf48e38700 1 -- 172.21.15.31:0/2239321564 shutdown_connections 2024-04-26T17:55:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:43.525+0000 7faf48e38700 1 -- 172.21.15.31:0/2239321564 wait complete. 2024-04-26T17:55:44.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:43 smithi031 ceph-mon[29164]: pgmap v377: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:44.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:43 smithi097 ceph-mon[32419]: pgmap v377: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:44.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:43 smithi186 ceph-mon[32797]: pgmap v377: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:44.355 INFO:tasks.cephadm:Setup complete, yielding 2024-04-26T17:55:44.355 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T17:55:44.365 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-26T17:55:44.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-26T17:55:44.616 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:46.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:45 smithi031 ceph-mon[29164]: pgmap v378: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:46.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:45 smithi097 ceph-mon[32419]: pgmap v378: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:46.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:45 smithi186 ceph-mon[32797]: pgmap v378: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.318+0000 7f57192ef700 1 -- 172.21.15.31:0/1414295429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5714103f20 msgr2=0x7f571410a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.318+0000 7f57192ef700 1 --2- 172.21.15.31:0/1414295429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5714103f20 0x7f571410a7d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f570c00b0d0 tx=0x7f570c00b3e0 comp rx=0 tx=0).stop 2024-04-26T17:55:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.318+0000 7f57192ef700 1 -- 172.21.15.31:0/1414295429 shutdown_connections 2024-04-26T17:55:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.318+0000 7f57192ef700 1 --2- 172.21.15.31:0/1414295429 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5714102380 0x7f57141027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.318+0000 7f57192ef700 1 --2- 172.21.15.31:0/1414295429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5714103f20 0x7f571410a7d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.318+0000 7f57192ef700 1 --2- 172.21.15.31:0/1414295429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5714103580 0x7f57141039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.318+0000 7f57192ef700 1 -- 172.21.15.31:0/1414295429 >> 172.21.15.31:0/1414295429 conn(0x7f57140fd920 msgr2=0x7f57140ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.318+0000 7f57192ef700 1 -- 172.21.15.31:0/1414295429 shutdown_connections 2024-04-26T17:55:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.318+0000 7f57192ef700 1 -- 172.21.15.31:0/1414295429 wait complete. 2024-04-26T17:55:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.319+0000 7f57192ef700 1 Processor -- start 2024-04-26T17:55:46.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.319+0000 7f57192ef700 1 -- start start 2024-04-26T17:55:46.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.319+0000 7f57192ef700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5714102380 0x7f5714106310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:46.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.319+0000 7f57192ef700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5714103580 0x7f5714104960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:46.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.319+0000 7f57192ef700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5714103f20 0x7f5714104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:46.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57192ef700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f571410de70 con 0x7f5714103580 2024-04-26T17:55:46.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57192ef700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f571410dcf0 con 0x7f5714103f20 2024-04-26T17:55:46.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57192ef700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f571410dff0 con 0x7f5714102380 2024-04-26T17:55:46.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57127fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5714103580 0x7f5714104960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:46.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57127fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5714103580 0x7f5714104960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35738/0 (socket says 172.21.15.31:35738) 2024-04-26T17:55:46.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57127fc700 1 -- 172.21.15.31:0/3859666124 learned_addr learned my addr 172.21.15.31:0/3859666124 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:46.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f5712ffd700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5714102380 0x7f5714106310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:46.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57137fe700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5714103f20 0x7f5714104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:46.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57127fc700 1 -- 172.21.15.31:0/3859666124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5714102380 msgr2=0x7f5714106310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:46.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57127fc700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5714102380 0x7f5714106310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:46.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57127fc700 1 -- 172.21.15.31:0/3859666124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5714103f20 msgr2=0x7f5714104ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:46.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57127fc700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5714103f20 0x7f5714104ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:46.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57127fc700 1 -- 172.21.15.31:0/3859666124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5714105750 con 0x7f5714103580 2024-04-26T17:55:46.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57137fe700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5714103f20 0x7f5714104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:55:46.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57127fc700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5714103580 0x7f5714104960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f570800b530 tx=0x7f570800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:46.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f56fbfff700 1 -- 172.21.15.31:0/3859666124 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5708015020 con 0x7f5714103580 2024-04-26T17:55:46.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f56fbfff700 1 -- 172.21.15.31:0/3859666124 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5708011ce0 con 0x7f5714103580 2024-04-26T17:55:46.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57192ef700 1 -- 172.21.15.31:0/3859666124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57141a37c0 con 0x7f5714103580 2024-04-26T17:55:46.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f56fbfff700 1 -- 172.21.15.31:0/3859666124 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5708010e40 con 0x7f5714103580 2024-04-26T17:55:46.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.320+0000 7f57192ef700 1 -- 172.21.15.31:0/3859666124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57141a3d50 con 0x7f5714103580 2024-04-26T17:55:46.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.321+0000 7f57192ef700 1 -- 172.21.15.31:0/3859666124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f571406a110 con 0x7f5714103580 2024-04-26T17:55:46.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.322+0000 7f56fbfff700 1 -- 172.21.15.31:0/3859666124 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5708018020 con 0x7f5714103580 2024-04-26T17:55:46.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.323+0000 7f56fbfff700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f56fc0a3c40 0x7f56fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:46.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.323+0000 7f5712ffd700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f56fc0a3c40 0x7f56fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:46.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.323+0000 7f56fbfff700 1 -- 172.21.15.31:0/3859666124 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f57080d01a0 con 0x7f5714103580 2024-04-26T17:55:46.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.324+0000 7f5712ffd700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f56fc0a3c40 0x7f56fc0a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f57141033e0 tx=0x7f570400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:46.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.327+0000 7f56fbfff700 1 -- 172.21.15.31:0/3859666124 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f570801f020 con 0x7f5714103580 2024-04-26T17:55:46.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.553+0000 7f57192ef700 1 -- 172.21.15.31:0/3859666124 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f57141a4010 con 0x7f56fc0a3c40 2024-04-26T17:55:46.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.554+0000 7f56fbfff700 1 -- 172.21.15.31:0/3859666124 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f57141a4010 con 0x7f56fc0a3c40 2024-04-26T17:55:46.553 INFO:teuthology.orchestra.run.smithi031.stdout:Backend: cephadm 2024-04-26T17:55:46.554 INFO:teuthology.orchestra.run.smithi031.stdout:Available: Yes 2024-04-26T17:55:46.554 INFO:teuthology.orchestra.run.smithi031.stdout:Paused: No 2024-04-26T17:55:46.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.557+0000 7f57192ef700 1 -- 172.21.15.31:0/3859666124 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f56fc0a3c40 msgr2=0x7f56fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:46.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.557+0000 7f57192ef700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f56fc0a3c40 0x7f56fc0a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f57141033e0 tx=0x7f570400b410 comp rx=0 tx=0).stop 2024-04-26T17:55:46.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.557+0000 7f57192ef700 1 -- 172.21.15.31:0/3859666124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5714103580 msgr2=0x7f5714104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:46.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.557+0000 7f57192ef700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5714103580 0x7f5714104960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f570800b530 tx=0x7f570800b840 comp rx=0 tx=0).stop 2024-04-26T17:55:46.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.557+0000 7f57192ef700 1 -- 172.21.15.31:0/3859666124 shutdown_connections 2024-04-26T17:55:46.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.557+0000 7f57192ef700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f56fc0a3c40 0x7f56fc0a6100 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:46.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.557+0000 7f57192ef700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5714103f20 0x7f5714104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:46.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.557+0000 7f57192ef700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5714102380 0x7f5714106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:46.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.557+0000 7f57192ef700 1 --2- 172.21.15.31:0/3859666124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5714103580 0x7f5714104960 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:46.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.557+0000 7f57192ef700 1 -- 172.21.15.31:0/3859666124 >> 172.21.15.31:0/3859666124 conn(0x7f57140fd920 msgr2=0x7f57140ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:46.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.557+0000 7f57192ef700 1 -- 172.21.15.31:0/3859666124 shutdown_connections 2024-04-26T17:55:46.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:46.557+0000 7f57192ef700 1 -- 172.21.15.31:0/3859666124 wait complete. 2024-04-26T17:55:47.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-26T17:55:47.573 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:47.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:47 smithi031 ceph-mon[29164]: pgmap v379: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:47.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:47 smithi031 ceph-mon[29164]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:55:48.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:47 smithi097 ceph-mon[32419]: pgmap v379: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:48.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:47 smithi097 ceph-mon[32419]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:55:48.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:47 smithi186 ceph-mon[32797]: pgmap v379: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:48.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:47 smithi186 ceph-mon[32797]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:55:49.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.280+0000 7febaa2c0700 1 -- 172.21.15.31:0/1521887988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba4075e80 msgr2=0x7feba4076300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:49.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.280+0000 7febaa2c0700 1 --2- 172.21.15.31:0/1521887988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba4075e80 0x7feba4076300 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009990 tx=0x7feb8c009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:49.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.280+0000 7febaa2c0700 1 -- 172.21.15.31:0/1521887988 shutdown_connections 2024-04-26T17:55:49.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.280+0000 7febaa2c0700 1 --2- 172.21.15.31:0/1521887988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feba410fa60 0x7feba4111e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:49.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.280+0000 7febaa2c0700 1 --2- 172.21.15.31:0/1521887988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feba4077450 0x7feba40758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:49.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.280+0000 7febaa2c0700 1 --2- 172.21.15.31:0/1521887988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba4075e80 0x7feba4076300 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:49.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.280+0000 7febaa2c0700 1 -- 172.21.15.31:0/1521887988 >> 172.21.15.31:0/1521887988 conn(0x7feba40fea20 msgr2=0x7feba4100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.280+0000 7febaa2c0700 1 -- 172.21.15.31:0/1521887988 shutdown_connections 2024-04-26T17:55:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.280+0000 7febaa2c0700 1 -- 172.21.15.31:0/1521887988 wait complete. 2024-04-26T17:55:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.280+0000 7febaa2c0700 1 Processor -- start 2024-04-26T17:55:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.280+0000 7febaa2c0700 1 -- start start 2024-04-26T17:55:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7febaa2c0700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feba4075e80 0x7feba419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7febaa2c0700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba4077450 0x7feba41a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7febaa2c0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feba410fa60 0x7feba41a71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7febaa2c0700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feba4115590 con 0x7feba4077450 2024-04-26T17:55:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7febaa2c0700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7feba4115410 con 0x7feba4075e80 2024-04-26T17:55:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7febaa2c0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feba4115710 con 0x7feba410fa60 2024-04-26T17:55:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7feba37fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba4077450 0x7feba41a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7feba37fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba4077450 0x7feba41a01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35748/0 (socket says 172.21.15.31:35748) 2024-04-26T17:55:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7feba37fe700 1 -- 172.21.15.31:0/13486678 learned_addr learned my addr 172.21.15.31:0/13486678 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7feba3fff700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feba4075e80 0x7feba419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:49.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7feba885d700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feba410fa60 0x7feba41a71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:49.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7feba37fe700 1 -- 172.21.15.31:0/13486678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feba410fa60 msgr2=0x7feba41a71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:49.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7feba37fe700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feba410fa60 0x7feba41a71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:49.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7feba37fe700 1 -- 172.21.15.31:0/13486678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feba4075e80 msgr2=0x7feba419fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:49.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7feba37fe700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feba4075e80 0x7feba419fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:49.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7feba37fe700 1 -- 172.21.15.31:0/13486678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba41a78c0 con 0x7feba4077450 2024-04-26T17:55:49.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.281+0000 7feba885d700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feba410fa60 0x7feba41a71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:55:49.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.282+0000 7feba37fe700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba4077450 0x7feba41a01b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0036f0 tx=0x7feb8c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:49.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.282+0000 7feba17fa700 1 -- 172.21.15.31:0/13486678 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c027070 con 0x7feba4077450 2024-04-26T17:55:49.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.282+0000 7feba17fa700 1 -- 172.21.15.31:0/13486678 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb8c022470 con 0x7feba4077450 2024-04-26T17:55:49.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.282+0000 7febaa2c0700 1 -- 172.21.15.31:0/13486678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba41a7b50 con 0x7feba4077450 2024-04-26T17:55:49.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.282+0000 7feba17fa700 1 -- 172.21.15.31:0/13486678 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c00f6c0 con 0x7feba4077450 2024-04-26T17:55:49.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.282+0000 7febaa2c0700 1 -- 172.21.15.31:0/13486678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba41a8000 con 0x7feba4077450 2024-04-26T17:55:49.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.283+0000 7febaa2c0700 1 -- 172.21.15.31:0/13486678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feba406a150 con 0x7feba4077450 2024-04-26T17:55:49.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.284+0000 7feba17fa700 1 -- 172.21.15.31:0/13486678 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7feb8c02a070 con 0x7feba4077450 2024-04-26T17:55:49.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.285+0000 7feba17fa700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb900a3cc0 0x7feb900a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:49.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.285+0000 7feba3fff700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb900a3cc0 0x7feb900a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:49.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.285+0000 7feba17fa700 1 -- 172.21.15.31:0/13486678 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7feb8c0d1690 con 0x7feba4077450 2024-04-26T17:55:49.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.285+0000 7feba3fff700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb900a3cc0 0x7feb900a6180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7feba41a1150 tx=0x7feb94009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:49.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.289+0000 7feba17fa700 1 -- 172.21.15.31:0/13486678 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feb8c031080 con 0x7feba4077450 2024-04-26T17:55:49.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.506+0000 7febaa2c0700 1 -- 172.21.15.31:0/13486678 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7feba4063860 con 0x7feb900a3cc0 2024-04-26T17:55:49.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.520+0000 7feba17fa700 1 -- 172.21.15.31:0/13486678 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7feba4063860 con 0x7feb900a3cc0 2024-04-26T17:55:49.520 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-26T17:55:49.520 INFO:teuthology.orchestra.run.smithi031.stdout:mds.a smithi031 running (3m) 33s ago 3m 14.3M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 1647cb60c974 2024-04-26T17:55:49.520 INFO:teuthology.orchestra.run.smithi031.stdout:mds.b smithi097 running (2m) 43s ago 2m 28.0M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 6ab9bc8bb9b0 2024-04-26T17:55:49.520 INFO:teuthology.orchestra.run.smithi031.stdout:mds.c smithi186 running (3m) 43s ago 3m 20.3M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 84410eca4b78 2024-04-26T17:55:49.520 INFO:teuthology.orchestra.run.smithi031.stdout:mds.d smithi031 running (3m) 33s ago 3m 13.7M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b d63a664ed7d0 2024-04-26T17:55:49.521 INFO:teuthology.orchestra.run.smithi031.stdout:mds.e smithi097 running (3m) 43s ago 3m 18.3M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b c07ebece9e02 2024-04-26T17:55:49.521 INFO:teuthology.orchestra.run.smithi031.stdout:mds.f smithi186 running (3m) 43s ago 3m 19.4M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b ad71e0cc5520 2024-04-26T17:55:49.521 INFO:teuthology.orchestra.run.smithi031.stdout:mds.g smithi031 running (3m) 33s ago 3m 17.0M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b c4605ee34e42 2024-04-26T17:55:49.521 INFO:teuthology.orchestra.run.smithi031.stdout:mds.h smithi097 running (3m) 43s ago 3m 19.2M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 46498fc53496 2024-04-26T17:55:49.521 INFO:teuthology.orchestra.run.smithi031.stdout:mds.i smithi186 running (3m) 43s ago 3m 22.2M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 44a624468434 2024-04-26T17:55:49.521 INFO:teuthology.orchestra.run.smithi031.stdout:mds.j smithi031 running (3m) 33s ago 3m 16.9M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b e44088701a82 2024-04-26T17:55:49.522 INFO:teuthology.orchestra.run.smithi031.stdout:mds.k smithi097 running (3m) 43s ago 3m 14.7M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 5f9a7c1d519c 2024-04-26T17:55:49.522 INFO:teuthology.orchestra.run.smithi031.stdout:mds.l smithi186 running (3m) 43s ago 3m 14.4M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 883a1e6b36df 2024-04-26T17:55:49.522 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.x smithi031 *:9283,8765 running (10m) 33s ago 10m 515M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b 7f2efebd461e 2024-04-26T17:55:49.522 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.y smithi097 *:8443,8765 running (8m) 43s ago 8m 453M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b d1c415b7a384 2024-04-26T17:55:49.522 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.z smithi186 *:8443,8765 running (8m) 43s ago 8m 453M - 19.0.0-3267-gbc94a2a9 3a6bde214c5b e4e8d064514a 2024-04-26T17:55:49.522 INFO:teuthology.orchestra.run.smithi031.stdout:mon.a smithi031 running (10m) 33s ago 11m 51.4M 2048M 19.0.0-3267-gbc94a2a9 3a6bde214c5b bd136c7b0b4e 2024-04-26T17:55:49.523 INFO:teuthology.orchestra.run.smithi031.stdout:mon.b smithi097 running (9m) 43s ago 9m 40.6M 2048M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 81484a0a13bd 2024-04-26T17:55:49.523 INFO:teuthology.orchestra.run.smithi031.stdout:mon.c smithi186 running (9m) 43s ago 9m 39.7M 2048M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 7c71567ce1a4 2024-04-26T17:55:49.523 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (8m) 33s ago 8m 46.4M 4273M 19.0.0-3267-gbc94a2a9 3a6bde214c5b facefcfadbae 2024-04-26T17:55:49.523 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi097 running (8m) 43s ago 8m 57.7M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b b6f8eea343ed 2024-04-26T17:55:49.523 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi186 running (7m) 43s ago 7m 62.2M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 7eb21129363b 2024-04-26T17:55:49.523 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (7m) 33s ago 7m 55.8M 4273M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 56fb33f9971f 2024-04-26T17:55:49.524 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi097 running (6m) 43s ago 6m 46.0M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b d399f4a5e5ea 2024-04-26T17:55:49.524 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi186 running (6m) 43s ago 6m 56.0M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 8f8387fbd454 2024-04-26T17:55:49.524 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi031 running (6m) 33s ago 6m 53.0M 4273M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 38a36a9ef15a 2024-04-26T17:55:49.524 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi097 running (5m) 43s ago 5m 45.3M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 8bfc575ef7ff 2024-04-26T17:55:49.524 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8 smithi186 running (5m) 43s ago 5m 44.1M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b eb37c8feddbd 2024-04-26T17:55:49.524 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9 smithi031 running (4m) 33s ago 4m 52.3M 4273M 19.0.0-3267-gbc94a2a9 3a6bde214c5b b73b20d2bdb0 2024-04-26T17:55:49.525 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10 smithi097 running (4m) 43s ago 4m 44.0M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b cedae32b8a0e 2024-04-26T17:55:49.525 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11 smithi186 running (3m) 43s ago 3m 42.9M 4272M 19.0.0-3267-gbc94a2a9 3a6bde214c5b 5413abcf1060 2024-04-26T17:55:49.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.523+0000 7febaa2c0700 1 -- 172.21.15.31:0/13486678 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb900a3cc0 msgr2=0x7feb900a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:49.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.523+0000 7febaa2c0700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb900a3cc0 0x7feb900a6180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7feba41a1150 tx=0x7feb94009190 comp rx=0 tx=0).stop 2024-04-26T17:55:49.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.523+0000 7febaa2c0700 1 -- 172.21.15.31:0/13486678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba4077450 msgr2=0x7feba41a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:49.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.523+0000 7febaa2c0700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba4077450 0x7feba41a01b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0036f0 tx=0x7feb8c0057c0 comp rx=0 tx=0).stop 2024-04-26T17:55:49.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.523+0000 7febaa2c0700 1 -- 172.21.15.31:0/13486678 shutdown_connections 2024-04-26T17:55:49.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.523+0000 7febaa2c0700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feb900a3cc0 0x7feb900a6180 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:49.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.523+0000 7febaa2c0700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feba4075e80 0x7feba419fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:49.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.523+0000 7febaa2c0700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feba410fa60 0x7feba41a71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:49.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.523+0000 7febaa2c0700 1 --2- 172.21.15.31:0/13486678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba4077450 0x7feba41a01b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:49.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.523+0000 7febaa2c0700 1 -- 172.21.15.31:0/13486678 >> 172.21.15.31:0/13486678 conn(0x7feba40fea20 msgr2=0x7feba4102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:49.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.523+0000 7febaa2c0700 1 -- 172.21.15.31:0/13486678 shutdown_connections 2024-04-26T17:55:49.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:49.523+0000 7febaa2c0700 1 -- 172.21.15.31:0/13486678 wait complete. 2024-04-26T17:55:50.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:49 smithi097 ceph-mon[32419]: pgmap v380: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:50.219 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:49 smithi031 ceph-mon[29164]: pgmap v380: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:50.248 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-26T17:55:50.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:49 smithi186 ceph-mon[32797]: pgmap v380: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:50.496 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:51.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:50 smithi097 ceph-mon[32419]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:55:51.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:50 smithi186 ceph-mon[32797]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:55:51.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:50 smithi031 ceph-mon[29164]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:55:52.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:51 smithi097 ceph-mon[32419]: pgmap v381: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:52.149 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:51 smithi031 ceph-mon[29164]: pgmap v381: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:52.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.157+0000 7f9f1e190700 1 -- 172.21.15.31:0/2553998216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f18102380 msgr2=0x7f9f181027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:52.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.157+0000 7f9f1e190700 1 --2- 172.21.15.31:0/2553998216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f18102380 0x7f9f181027a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9f080099c0 tx=0x7f9f08009cd0 comp rx=0 tx=0).stop 2024-04-26T17:55:52.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.157+0000 7f9f1e190700 1 -- 172.21.15.31:0/2553998216 shutdown_connections 2024-04-26T17:55:52.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.157+0000 7f9f1e190700 1 --2- 172.21.15.31:0/2553998216 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f18103f20 0x7f9f1810a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:52.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.157+0000 7f9f1e190700 1 --2- 172.21.15.31:0/2553998216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f18102380 0x7f9f181027a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:52.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.157+0000 7f9f1e190700 1 --2- 172.21.15.31:0/2553998216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f18103580 0x7f9f181039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:52.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.157+0000 7f9f1e190700 1 -- 172.21.15.31:0/2553998216 >> 172.21.15.31:0/2553998216 conn(0x7f9f180fd920 msgr2=0x7f9f180ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:52.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.157+0000 7f9f1e190700 1 -- 172.21.15.31:0/2553998216 shutdown_connections 2024-04-26T17:55:52.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.157+0000 7f9f1e190700 1 -- 172.21.15.31:0/2553998216 wait complete. 2024-04-26T17:55:52.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.158+0000 7f9f1e190700 1 Processor -- start 2024-04-26T17:55:52.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.158+0000 7f9f1e190700 1 -- start start 2024-04-26T17:55:52.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.158+0000 7f9f1e190700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f18102380 0x7f9f181985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:52.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.158+0000 7f9f1e190700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f18103580 0x7f9f18198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:52.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.158+0000 7f9f1e190700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f18103f20 0x7f9f1819fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:52.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f1e190700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1810de70 con 0x7f9f18103580 2024-04-26T17:55:52.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f1e190700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1810dcf0 con 0x7f9f18102380 2024-04-26T17:55:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f1e190700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1810dff0 con 0x7f9f18103f20 2024-04-26T17:55:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f0effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f18103580 0x7f9f18198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f0effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f18103580 0x7f9f18198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35758/0 (socket says 172.21.15.31:35758) 2024-04-26T17:55:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f0effd700 1 -- 172.21.15.31:0/3333195772 learned_addr learned my addr 172.21.15.31:0/3333195772 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f17fff700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f18103f20 0x7f9f1819fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f177fe700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f18102380 0x7f9f181985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f0effd700 1 -- 172.21.15.31:0/3333195772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f18103f20 msgr2=0x7f9f1819fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f0effd700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f18103f20 0x7f9f1819fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f0effd700 1 -- 172.21.15.31:0/3333195772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f18102380 msgr2=0x7f9f181985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f0effd700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f18102380 0x7f9f181985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f0effd700 1 -- 172.21.15.31:0/3333195772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f181a0200 con 0x7f9f18103580 2024-04-26T17:55:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f177fe700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f18102380 0x7f9f181985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:55:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f0effd700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f18103580 0x7f9f18198ae0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000b560 tx=0x7f9f0000b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:52.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f157fa700 1 -- 172.21.15.31:0/3333195772 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f00015020 con 0x7f9f18103580 2024-04-26T17:55:52.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f157fa700 1 -- 172.21.15.31:0/3333195772 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f00011ce0 con 0x7f9f18103580 2024-04-26T17:55:52.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f1e190700 1 -- 172.21.15.31:0/3333195772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f181a04f0 con 0x7f9f18103580 2024-04-26T17:55:52.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f157fa700 1 -- 172.21.15.31:0/3333195772 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f00010da0 con 0x7f9f18103580 2024-04-26T17:55:52.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.159+0000 7f9f1e190700 1 -- 172.21.15.31:0/3333195772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f181a0a80 con 0x7f9f18103580 2024-04-26T17:55:52.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.161+0000 7f9f1e190700 1 -- 172.21.15.31:0/3333195772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f1806a110 con 0x7f9f18103580 2024-04-26T17:55:52.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.162+0000 7f9f157fa700 1 -- 172.21.15.31:0/3333195772 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9f00018020 con 0x7f9f18103580 2024-04-26T17:55:52.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.162+0000 7f9f157fa700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9f040a3cc0 0x7f9f040a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:52.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.162+0000 7f9f177fe700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9f040a3cc0 0x7f9f040a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:52.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.162+0000 7f9f157fa700 1 -- 172.21.15.31:0/3333195772 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f9f000cfb70 con 0x7f9f18103580 2024-04-26T17:55:52.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.163+0000 7f9f177fe700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9f040a3cc0 0x7f9f040a6180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9f080038c0 tx=0x7f9f08009f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:52.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.167+0000 7f9f157fa700 1 -- 172.21.15.31:0/3333195772 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9f0001f020 con 0x7f9f18103580 2024-04-26T17:55:52.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:51 smithi186 ceph-mon[32797]: pgmap v381: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:55:52.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.383+0000 7f9f1e190700 1 -- 172.21.15.31:0/3333195772 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f9f18063860 con 0x7f9f040a3cc0 2024-04-26T17:55:52.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.389+0000 7f9f157fa700 1 -- 172.21.15.31:0/3333195772 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f9f18063860 con 0x7f9f040a3cc0 2024-04-26T17:55:52.388 INFO:teuthology.orchestra.run.smithi031.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-26T17:55:52.388 INFO:teuthology.orchestra.run.smithi031.stdout:mds.all 12/12 46s ago 3m smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi097=b;smithi097=e;smithi097=h;smithi097=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-26T17:55:52.389 INFO:teuthology.orchestra.run.smithi031.stdout:mgr 3/3 46s ago 8m smithi031=x;smithi097=y;smithi186=z;count:3 2024-04-26T17:55:52.389 INFO:teuthology.orchestra.run.smithi031.stdout:mon 3/3 46s ago 9m smithi031:172.21.15.31=a;smithi097:172.21.15.97=b;smithi186:172.21.15.186=c;count:3 2024-04-26T17:55:52.389 INFO:teuthology.orchestra.run.smithi031.stdout:osd 12 46s ago - 2024-04-26T17:55:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.391+0000 7f9f1e190700 1 -- 172.21.15.31:0/3333195772 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9f040a3cc0 msgr2=0x7f9f040a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.391+0000 7f9f1e190700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9f040a3cc0 0x7f9f040a6180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9f080038c0 tx=0x7f9f08009f10 comp rx=0 tx=0).stop 2024-04-26T17:55:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.391+0000 7f9f1e190700 1 -- 172.21.15.31:0/3333195772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f18103580 msgr2=0x7f9f18198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.391+0000 7f9f1e190700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f18103580 0x7f9f18198ae0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000b560 tx=0x7f9f0000b920 comp rx=0 tx=0).stop 2024-04-26T17:55:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.391+0000 7f9f1e190700 1 -- 172.21.15.31:0/3333195772 shutdown_connections 2024-04-26T17:55:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.391+0000 7f9f1e190700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f9f040a3cc0 0x7f9f040a6180 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.391+0000 7f9f1e190700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f18102380 0x7f9f181985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.391+0000 7f9f1e190700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f18103f20 0x7f9f1819fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.391+0000 7f9f1e190700 1 --2- 172.21.15.31:0/3333195772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f18103580 0x7f9f18198ae0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.391+0000 7f9f1e190700 1 -- 172.21.15.31:0/3333195772 >> 172.21.15.31:0/3333195772 conn(0x7f9f180fd920 msgr2=0x7f9f181069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:52.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.391+0000 7f9f1e190700 1 -- 172.21.15.31:0/3333195772 shutdown_connections 2024-04-26T17:55:52.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:52.391+0000 7f9f1e190700 1 -- 172.21.15.31:0/3333195772 wait complete. 2024-04-26T17:55:53.184 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-26T17:55:53.434 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:54.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:53 smithi031 ceph-mon[29164]: pgmap v382: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:54.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:53 smithi031 ceph-mon[29164]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:55:54.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:53 smithi186 ceph-mon[32797]: pgmap v382: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:54.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:53 smithi186 ceph-mon[32797]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:55:54.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:53 smithi097 ceph-mon[32419]: pgmap v382: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:55:54.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:53 smithi097 ceph-mon[32419]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:55:55.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.258+0000 7f09fde67700 1 -- 172.21.15.31:0/2306067523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f8103580 msgr2=0x7f09f81039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:55.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.258+0000 7f09fde67700 1 --2- 172.21.15.31:0/2306067523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f8103580 0x7f09f81039e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f09ec0099c0 tx=0x7f09ec009cd0 comp rx=0 tx=0).stop 2024-04-26T17:55:55.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.258+0000 7f09fde67700 1 -- 172.21.15.31:0/2306067523 shutdown_connections 2024-04-26T17:55:55.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.258+0000 7f09fde67700 1 --2- 172.21.15.31:0/2306067523 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09f8103f20 0x7f09f810a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:55.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.258+0000 7f09fde67700 1 --2- 172.21.15.31:0/2306067523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f8102380 0x7f09f81027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:55.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.258+0000 7f09fde67700 1 --2- 172.21.15.31:0/2306067523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f8103580 0x7f09f81039e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:55.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.258+0000 7f09fde67700 1 -- 172.21.15.31:0/2306067523 >> 172.21.15.31:0/2306067523 conn(0x7f09f80fd920 msgr2=0x7f09f80ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:55.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.258+0000 7f09fde67700 1 -- 172.21.15.31:0/2306067523 shutdown_connections 2024-04-26T17:55:55.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.258+0000 7f09fde67700 1 -- 172.21.15.31:0/2306067523 wait complete. 2024-04-26T17:55:55.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.259+0000 7f09fde67700 1 Processor -- start 2024-04-26T17:55:55.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.259+0000 7f09fde67700 1 -- start start 2024-04-26T17:55:55.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.259+0000 7f09fde67700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09f8102380 0x7f09f8199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:55.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.259+0000 7f09fde67700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f8103580 0x7f09f8199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:55.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.259+0000 7f09fde67700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f8103f20 0x7f09f81a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:55.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.259+0000 7f09fde67700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f09f810de90 con 0x7f09f8103f20 2024-04-26T17:55:55.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.259+0000 7f09fde67700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f09f810dd10 con 0x7f09f8102380 2024-04-26T17:55:55.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.259+0000 7f09fde67700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09f810e010 con 0x7f09f8103580 2024-04-26T17:55:55.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f8103f20 0x7f09f81a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:55.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f8103f20 0x7f09f81a0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34074/0 (socket says 172.21.15.31:34074) 2024-04-26T17:55:55.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f77fe700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09f8102380 0x7f09f8199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:55.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f7fff700 1 -- 172.21.15.31:0/2595677571 learned_addr learned my addr 172.21.15.31:0/2595677571 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:55.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f6ffd700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f8103580 0x7f09f8199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f7fff700 1 -- 172.21.15.31:0/2595677571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f8103580 msgr2=0x7f09f8199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f7fff700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f8103580 0x7f09f8199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f7fff700 1 -- 172.21.15.31:0/2595677571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09f8102380 msgr2=0x7f09f8199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f7fff700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09f8102380 0x7f09f8199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f7fff700 1 -- 172.21.15.31:0/2595677571 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f81a12e0 con 0x7f09f8103f20 2024-04-26T17:55:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f6ffd700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f8103580 0x7f09f8199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:55:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f77fe700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09f8102380 0x7f09f8199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:55:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f7fff700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f8103f20 0x7f09f81a0c40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f09f000ef60 tx=0x7f09f000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f4ff9700 1 -- 172.21.15.31:0/2595677571 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f00092e0 con 0x7f09f8103f20 2024-04-26T17:55:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09fde67700 1 -- 172.21.15.31:0/2595677571 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f81a1570 con 0x7f09f8103f20 2024-04-26T17:55:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f4ff9700 1 -- 172.21.15.31:0/2595677571 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09f0004bb0 con 0x7f09f8103f20 2024-04-26T17:55:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09f4ff9700 1 -- 172.21.15.31:0/2595677571 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f0004280 con 0x7f09f8103f20 2024-04-26T17:55:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.260+0000 7f09fde67700 1 -- 172.21.15.31:0/2595677571 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f81a1b30 con 0x7f09f8103f20 2024-04-26T17:55:55.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.261+0000 7f09fde67700 1 -- 172.21.15.31:0/2595677571 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f806a110 con 0x7f09f8103f20 2024-04-26T17:55:55.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.263+0000 7f09f4ff9700 1 -- 172.21.15.31:0/2595677571 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f09f001b030 con 0x7f09f8103f20 2024-04-26T17:55:55.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.263+0000 7f09f4ff9700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f09e00a3c40 0x7f09e00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:55.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.263+0000 7f09f77fe700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f09e00a3c40 0x7f09e00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:55.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.263+0000 7f09f4ff9700 1 -- 172.21.15.31:0/2595677571 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f09f00cfee0 con 0x7f09f8103f20 2024-04-26T17:55:55.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.264+0000 7f09f77fe700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f09e00a3c40 0x7f09e00a6100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f09e8006fd0 tx=0x7f09e8009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:55.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.268+0000 7f09f4ff9700 1 -- 172.21.15.31:0/2595677571 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f09f0022020 con 0x7f09f8103f20 2024-04-26T17:55:55.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.480+0000 7f09fde67700 1 -- 172.21.15.31:0/2595677571 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f09f80ffb60 con 0x7f09e00a3c40 2024-04-26T17:55:55.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.481+0000 7f09f4ff9700 1 -- 172.21.15.31:0/2595677571 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f09f80ffb60 con 0x7f09e00a3c40 2024-04-26T17:55:55.481 INFO:teuthology.orchestra.run.smithi031.stdout:HOST ADDR LABELS STATUS 2024-04-26T17:55:55.481 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031 172.21.15.31 2024-04-26T17:55:55.481 INFO:teuthology.orchestra.run.smithi031.stdout:smithi097 172.21.15.97 2024-04-26T17:55:55.481 INFO:teuthology.orchestra.run.smithi031.stdout:smithi186 172.21.15.186 2024-04-26T17:55:55.481 INFO:teuthology.orchestra.run.smithi031.stdout:3 hosts in cluster 2024-04-26T17:55:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.483+0000 7f09fde67700 1 -- 172.21.15.31:0/2595677571 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f09e00a3c40 msgr2=0x7f09e00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.483+0000 7f09fde67700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f09e00a3c40 0x7f09e00a6100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f09e8006fd0 tx=0x7f09e8009190 comp rx=0 tx=0).stop 2024-04-26T17:55:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.483+0000 7f09fde67700 1 -- 172.21.15.31:0/2595677571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f8103f20 msgr2=0x7f09f81a0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.483+0000 7f09fde67700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f8103f20 0x7f09f81a0c40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f09f000ef60 tx=0x7f09f000ef90 comp rx=0 tx=0).stop 2024-04-26T17:55:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.483+0000 7f09fde67700 1 -- 172.21.15.31:0/2595677571 shutdown_connections 2024-04-26T17:55:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.483+0000 7f09fde67700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f09e00a3c40 0x7f09e00a6100 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.483+0000 7f09fde67700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f09f8102380 0x7f09f8199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.483+0000 7f09fde67700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09f8103f20 0x7f09f81a0c40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.483+0000 7f09fde67700 1 --2- 172.21.15.31:0/2595677571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09f8103580 0x7f09f8199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.483+0000 7f09fde67700 1 -- 172.21.15.31:0/2595677571 >> 172.21.15.31:0/2595677571 conn(0x7f09f80fd920 msgr2=0x7f09f80ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.483+0000 7f09fde67700 1 -- 172.21.15.31:0/2595677571 shutdown_connections 2024-04-26T17:55:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:55.483+0000 7f09fde67700 1 -- 172.21.15.31:0/2595677571 wait complete. 2024-04-26T17:55:56.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:55 smithi031 ceph-mon[29164]: pgmap v383: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:56.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:55 smithi186 ceph-mon[32797]: pgmap v383: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:56.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-26T17:55:56.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:55 smithi097 ceph-mon[32419]: pgmap v383: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:55:56.529 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:55:56.910 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:56 smithi031 ceph-mon[29164]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:55:57.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:56 smithi186 ceph-mon[32797]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:55:57.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:56 smithi097 ceph-mon[32419]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:55:58.052 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:57 smithi031 ceph-mon[29164]: pgmap v384: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:58.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:57 smithi186 ceph-mon[32797]: pgmap v384: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:58.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.286+0000 7f38286ab700 1 -- 172.21.15.31:0/1290255349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820101820 msgr2=0x7f3820101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:58.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.286+0000 7f38286ab700 1 --2- 172.21.15.31:0/1290255349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820101820 0x7f3820101c40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f381c009990 tx=0x7f381c009ca0 comp rx=0 tx=0).stop 2024-04-26T17:55:58.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.286+0000 7f38286ab700 1 -- 172.21.15.31:0/1290255349 shutdown_connections 2024-04-26T17:55:58.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.286+0000 7f38286ab700 1 --2- 172.21.15.31:0/1290255349 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f38201051a0 0x7f382010ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.286+0000 7f38286ab700 1 --2- 172.21.15.31:0/1290255349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820101820 0x7f3820101c40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.286+0000 7f38286ab700 1 --2- 172.21.15.31:0/1290255349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3820104800 0x7f3820104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.286+0000 7f38286ab700 1 -- 172.21.15.31:0/1290255349 >> 172.21.15.31:0/1290255349 conn(0x7f38200fd180 msgr2=0x7f38200ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.286+0000 7f38286ab700 1 -- 172.21.15.31:0/1290255349 shutdown_connections 2024-04-26T17:55:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.286+0000 7f38286ab700 1 -- 172.21.15.31:0/1290255349 wait complete. 2024-04-26T17:55:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.287+0000 7f38286ab700 1 Processor -- start 2024-04-26T17:55:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.287+0000 7f38286ab700 1 -- start start 2024-04-26T17:55:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.287+0000 7f38286ab700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3820101820 0x7f38201996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.287+0000 7f38286ab700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820104800 0x7f3820199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.287+0000 7f38286ab700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38201051a0 0x7f38201a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.287+0000 7f38286ab700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f382010f070 con 0x7f3820104800 2024-04-26T17:55:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.287+0000 7f38286ab700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f382010eef0 con 0x7f3820101820 2024-04-26T17:55:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.287+0000 7f38286ab700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f382010f1f0 con 0x7f38201051a0 2024-04-26T17:55:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.287+0000 7f3825c46700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820104800 0x7f3820199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.287+0000 7f3825c46700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820104800 0x7f3820199c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34090/0 (socket says 172.21.15.31:34090) 2024-04-26T17:55:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.287+0000 7f3825c46700 1 -- 172.21.15.31:0/298407502 learned_addr learned my addr 172.21.15.31:0/298407502 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:55:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.287+0000 7f3826447700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3820101820 0x7f38201996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.288+0000 7f3826c48700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38201051a0 0x7f38201a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.288+0000 7f3825c46700 1 -- 172.21.15.31:0/298407502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38201051a0 msgr2=0x7f38201a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.288+0000 7f3825c46700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38201051a0 0x7f38201a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.288+0000 7f3825c46700 1 -- 172.21.15.31:0/298407502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3820101820 msgr2=0x7f38201996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.288+0000 7f3825c46700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3820101820 0x7f38201996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.288+0000 7f3825c46700 1 -- 172.21.15.31:0/298407502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38201a1320 con 0x7f3820104800 2024-04-26T17:55:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.288+0000 7f3826c48700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38201051a0 0x7f38201a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:55:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.288+0000 7f3825c46700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820104800 0x7f3820199c00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f381000d9b0 tx=0x7f381000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.288+0000 7f38177fe700 1 -- 172.21.15.31:0/298407502 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3810017070 con 0x7f3820104800 2024-04-26T17:55:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.288+0000 7f38177fe700 1 -- 172.21.15.31:0/298407502 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3810005020 con 0x7f3820104800 2024-04-26T17:55:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.288+0000 7f38177fe700 1 -- 172.21.15.31:0/298407502 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3810009b50 con 0x7f3820104800 2024-04-26T17:55:58.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.288+0000 7f38286ab700 1 -- 172.21.15.31:0/298407502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38201a1610 con 0x7f3820104800 2024-04-26T17:55:58.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.288+0000 7f38286ab700 1 -- 172.21.15.31:0/298407502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38201a1b00 con 0x7f3820104800 2024-04-26T17:55:58.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.289+0000 7f38286ab700 1 -- 172.21.15.31:0/298407502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f382006a110 con 0x7f3820104800 2024-04-26T17:55:58.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.290+0000 7f38177fe700 1 -- 172.21.15.31:0/298407502 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f38100051d0 con 0x7f3820104800 2024-04-26T17:55:58.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.291+0000 7f38177fe700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f380c0a3c40 0x7f380c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:55:58.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.291+0000 7f3826447700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f380c0a3c40 0x7f380c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:55:58.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.291+0000 7f38177fe700 1 -- 172.21.15.31:0/298407502 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f38100d06d0 con 0x7f3820104800 2024-04-26T17:55:58.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.292+0000 7f3826447700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f380c0a3c40 0x7f380c0a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f381c0038a0 tx=0x7f381c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:55:58.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.295+0000 7f38177fe700 1 -- 172.21.15.31:0/298407502 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3810020020 con 0x7f3820104800 2024-04-26T17:55:58.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:57 smithi097 ceph-mon[32419]: pgmap v384: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:55:58.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.514+0000 7f38286ab700 1 -- 172.21.15.31:0/298407502 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f3820063860 con 0x7f380c0a3c40 2024-04-26T17:55:58.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.516+0000 7f38177fe700 1 -- 172.21.15.31:0/298407502 <== mgr.14150 v2:172.21.15.31:6800/2410008001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f3820063860 con 0x7f380c0a3c40 2024-04-26T17:55:58.516 INFO:teuthology.orchestra.run.smithi031.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-26T17:55:58.516 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53310008400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-26T17:55:58.516 INFO:teuthology.orchestra.run.smithi031.stdout:smithi097 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204013T400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-26T17:55:58.516 INFO:teuthology.orchestra.run.smithi031.stdout:smithi186 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CB400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-26T17:55:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.518+0000 7f38286ab700 1 -- 172.21.15.31:0/298407502 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f380c0a3c40 msgr2=0x7f380c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.518+0000 7f38286ab700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f380c0a3c40 0x7f380c0a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f381c0038a0 tx=0x7f381c005fb0 comp rx=0 tx=0).stop 2024-04-26T17:55:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.518+0000 7f38286ab700 1 -- 172.21.15.31:0/298407502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820104800 msgr2=0x7f3820199c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:55:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.518+0000 7f38286ab700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820104800 0x7f3820199c00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f381000d9b0 tx=0x7f381000dcc0 comp rx=0 tx=0).stop 2024-04-26T17:55:58.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.518+0000 7f38286ab700 1 -- 172.21.15.31:0/298407502 shutdown_connections 2024-04-26T17:55:58.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.518+0000 7f38286ab700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f380c0a3c40 0x7f380c0a6100 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:58.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.518+0000 7f38286ab700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3820101820 0x7f38201996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:58.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.518+0000 7f38286ab700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38201051a0 0x7f38201a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:58.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.518+0000 7f38286ab700 1 --2- 172.21.15.31:0/298407502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820104800 0x7f3820199c00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:55:58.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.518+0000 7f38286ab700 1 -- 172.21.15.31:0/298407502 >> 172.21.15.31:0/298407502 conn(0x7f38200fd180 msgr2=0x7f3820107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:55:58.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.518+0000 7f38286ab700 1 -- 172.21.15.31:0/298407502 shutdown_connections 2024-04-26T17:55:58.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:55:58.518+0000 7f38286ab700 1 -- 172.21.15.31:0/298407502 wait complete. 2024-04-26T17:55:59.260 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T17:55:59.270 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-26T17:55:59.270 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-26T17:55:59.525 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:00.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:59 smithi186 ceph-mon[32797]: pgmap v385: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:00.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:55:59 smithi186 ceph-mon[32797]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:00.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:59 smithi031 ceph-mon[29164]: pgmap v385: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:00.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:55:59 smithi031 ceph-mon[29164]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:00.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:59 smithi097 ceph-mon[32419]: pgmap v385: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:00.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:55:59 smithi097 ceph-mon[32419]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:01.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.134+0000 7febc792d700 1 -- 172.21.15.31:0/4027370459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc01046a0 msgr2=0x7febc0104b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:01.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.134+0000 7febc792d700 1 --2- 172.21.15.31:0/4027370459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc01046a0 0x7febc0104b00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7febb40099c0 tx=0x7febb4009cd0 comp rx=0 tx=0).stop 2024-04-26T17:56:01.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.134+0000 7febc792d700 1 -- 172.21.15.31:0/4027370459 shutdown_connections 2024-04-26T17:56:01.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.134+0000 7febc792d700 1 --2- 172.21.15.31:0/4027370459 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7febc0105040 0x7febc010b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.134+0000 7febc792d700 1 --2- 172.21.15.31:0/4027370459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febc01034a0 0x7febc01038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.134+0000 7febc792d700 1 --2- 172.21.15.31:0/4027370459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc01046a0 0x7febc0104b00 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.134+0000 7febc792d700 1 -- 172.21.15.31:0/4027370459 >> 172.21.15.31:0/4027370459 conn(0x7febc00fea40 msgr2=0x7febc0100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.134+0000 7febc792d700 1 -- 172.21.15.31:0/4027370459 shutdown_connections 2024-04-26T17:56:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.134+0000 7febc792d700 1 -- 172.21.15.31:0/4027370459 wait complete. 2024-04-26T17:56:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc792d700 1 Processor -- start 2024-04-26T17:56:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc792d700 1 -- start start 2024-04-26T17:56:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc792d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc01034a0 0x7febc0199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc792d700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7febc01046a0 0x7febc0199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc792d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febc0105040 0x7febc01a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc792d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7febc010ef40 con 0x7febc01034a0 2024-04-26T17:56:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc792d700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7febc010edc0 con 0x7febc01046a0 2024-04-26T17:56:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc792d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7febc010f0c0 con 0x7febc0105040 2024-04-26T17:56:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc56c9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc01034a0 0x7febc0199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc56c9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc01034a0 0x7febc0199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34108/0 (socket says 172.21.15.31:34108) 2024-04-26T17:56:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc56c9700 1 -- 172.21.15.31:0/2109294094 learned_addr learned my addr 172.21.15.31:0/2109294094 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc4ec8700 1 --2- 172.21.15.31:0/2109294094 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7febc01046a0 0x7febc0199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc5eca700 1 --2- 172.21.15.31:0/2109294094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febc0105040 0x7febc01a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc56c9700 1 -- 172.21.15.31:0/2109294094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febc0105040 msgr2=0x7febc01a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc56c9700 1 --2- 172.21.15.31:0/2109294094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febc0105040 0x7febc01a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc56c9700 1 -- 172.21.15.31:0/2109294094 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7febc01046a0 msgr2=0x7febc0199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc56c9700 1 --2- 172.21.15.31:0/2109294094 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7febc01046a0 0x7febc0199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.135+0000 7febc56c9700 1 -- 172.21.15.31:0/2109294094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc01a11f0 con 0x7febc01034a0 2024-04-26T17:56:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.136+0000 7febc56c9700 1 --2- 172.21.15.31:0/2109294094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc01034a0 0x7febc0199590 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7febb800add0 tx=0x7febb800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.136+0000 7febbe7fc700 1 -- 172.21.15.31:0/2109294094 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb8017070 con 0x7febc01034a0 2024-04-26T17:56:01.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.136+0000 7febbe7fc700 1 -- 172.21.15.31:0/2109294094 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febb8004bb0 con 0x7febc01034a0 2024-04-26T17:56:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.136+0000 7febc792d700 1 -- 172.21.15.31:0/2109294094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febc01a14e0 con 0x7febc01034a0 2024-04-26T17:56:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.136+0000 7febbe7fc700 1 -- 172.21.15.31:0/2109294094 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb800ce20 con 0x7febc01034a0 2024-04-26T17:56:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.136+0000 7febc792d700 1 -- 172.21.15.31:0/2109294094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febc01a1a20 con 0x7febc01034a0 2024-04-26T17:56:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.137+0000 7febc792d700 1 -- 172.21.15.31:0/2109294094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febc006a110 con 0x7febc01034a0 2024-04-26T17:56:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.138+0000 7febbe7fc700 1 -- 172.21.15.31:0/2109294094 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7febb801a070 con 0x7febc01034a0 2024-04-26T17:56:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.139+0000 7febbe7fc700 1 --2- 172.21.15.31:0/2109294094 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feba80a3cc0 0x7feba80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.139+0000 7febc4ec8700 1 --2- 172.21.15.31:0/2109294094 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feba80a3cc0 0x7feba80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.139+0000 7febbe7fc700 1 -- 172.21.15.31:0/2109294094 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7febb80cf640 con 0x7febc01034a0 2024-04-26T17:56:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.139+0000 7febc4ec8700 1 --2- 172.21.15.31:0/2109294094 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feba80a3cc0 0x7feba80a6180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7febb4009990 tx=0x7febb4005c20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:01.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.142+0000 7febbe7fc700 1 -- 172.21.15.31:0/2109294094 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7febb809c050 con 0x7febc01034a0 2024-04-26T17:56:01.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.401+0000 7febc792d700 1 -- 172.21.15.31:0/2109294094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7febc0064030 con 0x7febc01034a0 2024-04-26T17:56:01.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.401+0000 7febbe7fc700 1 -- 172.21.15.31:0/2109294094 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) ==== 76+0+3050 (secure 0 0 0) 0x7febb809b950 con 0x7febc01034a0 2024-04-26T17:56:01.401 INFO:teuthology.orchestra.run.smithi031.stdout:e25 2024-04-26T17:56:01.402 INFO:teuthology.orchestra.run.smithi031.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T17:56:01.402 INFO:teuthology.orchestra.run.smithi031.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T17:56:01.402 INFO:teuthology.orchestra.run.smithi031.stdout:legacy client fscid: 1 2024-04-26T17:56:01.402 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:01.402 INFO:teuthology.orchestra.run.smithi031.stdout:Filesystem 'cephfs' (1) 2024-04-26T17:56:01.402 INFO:teuthology.orchestra.run.smithi031.stdout:fs_name cephfs 2024-04-26T17:56:01.402 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 25 2024-04-26T17:56:01.402 INFO:teuthology.orchestra.run.smithi031.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-26T17:56:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-04-26T17:53:05.425897+0000 2024-04-26T17:56:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:modified 2024-04-26T17:53:25.754649+0000 2024-04-26T17:56:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:tableserver 0 2024-04-26T17:56:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:root 0 2024-04-26T17:56:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:session_timeout 300 2024-04-26T17:56:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:session_autoclose 300 2024-04-26T17:56:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:max_file_size 1099511627776 2024-04-26T17:56:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:max_xattr_size 65536 2024-04-26T17:56:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:required_client_features {} 2024-04-26T17:56:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:last_failure 0 2024-04-26T17:56:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:last_failure_osd_epoch 0 2024-04-26T17:56:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T17:56:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:max_mds 3 2024-04-26T17:56:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:in 0,1,2 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:up {0=24473,1=24457,2=24446} 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:failed 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:damaged 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:stopped 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:data_pools [3] 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:metadata_pool 2 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:inline_data disabled 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:balancer 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:bal_rank_mask -1 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:standby_count_wanted 0 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:qdb_cluster leader: 24473 members: 24457,24473,24446 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.b{0:24473} state up:active seq 5 addr [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.c{0:24445} state up:standby-replay seq 1 addr [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.i{1:24457} state up:active seq 11 addr [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T17:56:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.f{1:24439} state up:standby-replay seq 1 addr [v2:172.21.15.186:6834/580433615,v1:172.21.15.186:6835/580433615] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T17:56:01.405 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.h{2:24446} state up:active seq 12 addr [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T17:56:01.405 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.e{2:24428} state up:standby-replay seq 1 addr [v2:172.21.15.97:6834/1721162755,v1:172.21.15.97:6835/1721162755] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T17:56:01.405 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:01.405 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:01.405 INFO:teuthology.orchestra.run.smithi031.stdout:Standby daemons: 2024-04-26T17:56:01.405 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:01.405 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.j{-1:14508} state up:standby seq 1 addr [v2:172.21.15.31:6834/3371658851,v1:172.21.15.31:6835/3371658851] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T17:56:01.405 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.d{-1:14520} state up:standby seq 1 addr [v2:172.21.15.31:6836/1425597980,v1:172.21.15.31:6837/1425597980] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T17:56:01.405 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.g{-1:14565} state up:standby seq 1 addr [v2:172.21.15.31:6838/2783481787,v1:172.21.15.31:6839/2783481787] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T17:56:01.405 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.a{-1:14577} state up:standby seq 1 addr [v2:172.21.15.31:6840/2984904862,v1:172.21.15.31:6841/2984904862] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T17:56:01.405 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.k{-1:24404} state up:standby seq 1 addr [v2:172.21.15.97:6832/2883739522,v1:172.21.15.97:6833/2883739522] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T17:56:01.405 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.l{-1:24412} state up:standby seq 1 addr [v2:172.21.15.186:6832/1597494912,v1:172.21.15.186:6833/1597494912] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T17:56:01.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.403+0000 7febc792d700 1 -- 172.21.15.31:0/2109294094 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feba80a3cc0 msgr2=0x7feba80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:01.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.403+0000 7febc792d700 1 --2- 172.21.15.31:0/2109294094 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feba80a3cc0 0x7feba80a6180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7febb4009990 tx=0x7febb4005c20 comp rx=0 tx=0).stop 2024-04-26T17:56:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.403+0000 7febc792d700 1 -- 172.21.15.31:0/2109294094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc01034a0 msgr2=0x7febc0199590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.403+0000 7febc792d700 1 --2- 172.21.15.31:0/2109294094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc01034a0 0x7febc0199590 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7febb800add0 tx=0x7febb800c330 comp rx=0 tx=0).stop 2024-04-26T17:56:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.403+0000 7febc792d700 1 -- 172.21.15.31:0/2109294094 shutdown_connections 2024-04-26T17:56:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.403+0000 7febc792d700 1 --2- 172.21.15.31:0/2109294094 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7feba80a3cc0 0x7feba80a6180 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.403+0000 7febc792d700 1 --2- 172.21.15.31:0/2109294094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febc0105040 0x7febc01a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.403+0000 7febc792d700 1 --2- 172.21.15.31:0/2109294094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc01034a0 0x7febc0199590 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.403+0000 7febc792d700 1 --2- 172.21.15.31:0/2109294094 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7febc01046a0 0x7febc0199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.403+0000 7febc792d700 1 -- 172.21.15.31:0/2109294094 >> 172.21.15.31:0/2109294094 conn(0x7febc00fea40 msgr2=0x7febc0107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.403+0000 7febc792d700 1 -- 172.21.15.31:0/2109294094 shutdown_connections 2024-04-26T17:56:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:01.403+0000 7febc792d700 1 -- 172.21.15.31:0/2109294094 wait complete. 2024-04-26T17:56:01.408 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-04-26T17:56:02.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:01 smithi031 ceph-mon[29164]: pgmap v386: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:02.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:01 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2109294094' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T17:56:02.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-26T17:56:02.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:01 smithi186 ceph-mon[32797]: pgmap v386: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:02.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:01 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2109294094' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T17:56:02.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:01 smithi097 ceph-mon[32419]: pgmap v386: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:02.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:01 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2109294094' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T17:56:02.428 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.031+0000 7f757b807700 1 -- 172.21.15.31:0/2741533202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7574104720 msgr2=0x7f7574104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.031+0000 7f757b807700 1 --2- 172.21.15.31:0/2741533202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7574104720 0x7f7574104b80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7564009990 tx=0x7f7564009ca0 comp rx=0 tx=0).stop 2024-04-26T17:56:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.031+0000 7f757b807700 1 -- 172.21.15.31:0/2741533202 shutdown_connections 2024-04-26T17:56:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.031+0000 7f757b807700 1 --2- 172.21.15.31:0/2741533202 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7574101740 0x7f7574101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.031+0000 7f757b807700 1 --2- 172.21.15.31:0/2741533202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75741050c0 0x7f757410b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.031+0000 7f757b807700 1 --2- 172.21.15.31:0/2741533202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7574104720 0x7f7574104b80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.031+0000 7f757b807700 1 -- 172.21.15.31:0/2741533202 >> 172.21.15.31:0/2741533202 conn(0x7f75740fd080 msgr2=0x7f75740ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.031+0000 7f757b807700 1 -- 172.21.15.31:0/2741533202 shutdown_connections 2024-04-26T17:56:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.031+0000 7f757b807700 1 -- 172.21.15.31:0/2741533202 wait complete. 2024-04-26T17:56:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.031+0000 7f757b807700 1 Processor -- start 2024-04-26T17:56:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.031+0000 7f757b807700 1 -- start start 2024-04-26T17:56:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f757b807700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7574101740 0x7f7574199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f757b807700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7574104720 0x7f7574199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f757b807700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75741050c0 0x7f75741a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f757b807700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f757410ef10 con 0x7f7574101740 2024-04-26T17:56:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f757b807700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f757410ed90 con 0x7f7574104720 2024-04-26T17:56:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f757b807700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f757410f090 con 0x7f75741050c0 2024-04-26T17:56:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f75795a3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7574101740 0x7f7574199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f75795a3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7574101740 0x7f7574199560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34126/0 (socket says 172.21.15.31:34126) 2024-04-26T17:56:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f75795a3700 1 -- 172.21.15.31:0/2301052964 learned_addr learned my addr 172.21.15.31:0/2301052964 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f7578da2700 1 --2- 172.21.15.31:0/2301052964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7574104720 0x7f7574199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f7579da4700 1 --2- 172.21.15.31:0/2301052964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75741050c0 0x7f75741a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f75795a3700 1 -- 172.21.15.31:0/2301052964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75741050c0 msgr2=0x7f75741a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f75795a3700 1 --2- 172.21.15.31:0/2301052964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75741050c0 0x7f75741a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f75795a3700 1 -- 172.21.15.31:0/2301052964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7574104720 msgr2=0x7f7574199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f75795a3700 1 --2- 172.21.15.31:0/2301052964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7574104720 0x7f7574199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f75795a3700 1 -- 172.21.15.31:0/2301052964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75741a11c0 con 0x7f7574101740 2024-04-26T17:56:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f75795a3700 1 --2- 172.21.15.31:0/2301052964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7574101740 0x7f7574199560 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f757000d9b0 tx=0x7f757000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f756a7fc700 1 -- 172.21.15.31:0/2301052964 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7570016070 con 0x7f7574101740 2024-04-26T17:56:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f757b807700 1 -- 172.21.15.31:0/2301052964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75741a14b0 con 0x7f7574101740 2024-04-26T17:56:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.032+0000 7f756a7fc700 1 -- 172.21.15.31:0/2301052964 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f757000be00 con 0x7f7574101740 2024-04-26T17:56:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.033+0000 7f756a7fc700 1 -- 172.21.15.31:0/2301052964 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7570010e30 con 0x7f7574101740 2024-04-26T17:56:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.033+0000 7f757b807700 1 -- 172.21.15.31:0/2301052964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75741a19f0 con 0x7f7574101740 2024-04-26T17:56:04.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.033+0000 7f757b807700 1 -- 172.21.15.31:0/2301052964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f757406a110 con 0x7f7574101740 2024-04-26T17:56:04.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.035+0000 7f756a7fc700 1 -- 172.21.15.31:0/2301052964 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7570019070 con 0x7f7574101740 2024-04-26T17:56:04.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.035+0000 7f756a7fc700 1 --2- 172.21.15.31:0/2301052964 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f75600a3c40 0x7f75600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:04.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.036+0000 7f7578da2700 1 --2- 172.21.15.31:0/2301052964 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f75600a3c40 0x7f75600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:04.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.036+0000 7f756a7fc700 1 -- 172.21.15.31:0/2301052964 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f75700d0800 con 0x7f7574101740 2024-04-26T17:56:04.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.036+0000 7f7578da2700 1 --2- 172.21.15.31:0/2301052964 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f75600a3c40 0x7f75600a6100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f756400b5c0 tx=0x7f7564005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:04.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.039+0000 7f756a7fc700 1 -- 172.21.15.31:0/2301052964 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7570020030 con 0x7f7574101740 2024-04-26T17:56:04.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:03 smithi031 ceph-mon[29164]: pgmap v387: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:56:04.186 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:03 smithi097 ceph-mon[32419]: pgmap v387: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:56:04.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.247+0000 7f757b807700 1 -- 172.21.15.31:0/2301052964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f7574064030 con 0x7f7574101740 2024-04-26T17:56:04.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.248+0000 7f756a7fc700 1 -- 172.21.15.31:0/2301052964 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+5245 (secure 0 0 0) 0x7f757009d210 con 0x7f7574101740 2024-04-26T17:56:04.249 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 90 2024-04-26T17:56:04.249 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T17:56:04.249 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-04-26T17:44:48.618250+0000 2024-04-26T17:56:04.249 INFO:teuthology.orchestra.run.smithi031.stdout:modified 2024-04-26T17:53:49.754504+0000 2024-04-26T17:56:04.249 INFO:teuthology.orchestra.run.smithi031.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-26T17:56:04.249 INFO:teuthology.orchestra.run.smithi031.stdout:crush_version 26 2024-04-26T17:56:04.249 INFO:teuthology.orchestra.run.smithi031.stdout:full_ratio 0.95 2024-04-26T17:56:04.249 INFO:teuthology.orchestra.run.smithi031.stdout:backfillfull_ratio 0.9 2024-04-26T17:56:04.249 INFO:teuthology.orchestra.run.smithi031.stdout:nearfull_ratio 0.85 2024-04-26T17:56:04.249 INFO:teuthology.orchestra.run.smithi031.stdout:require_min_compat_client luminous 2024-04-26T17:56:04.249 INFO:teuthology.orchestra.run.smithi031.stdout:min_compat_client luminous 2024-04-26T17:56:04.249 INFO:teuthology.orchestra.run.smithi031.stdout:require_osd_release squid 2024-04-26T17:56:04.250 INFO:teuthology.orchestra.run.smithi031.stdout:stretch_mode_enabled false 2024-04-26T17:56:04.250 INFO:teuthology.orchestra.run.smithi031.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-26T17:56:04.250 INFO:teuthology.orchestra.run.smithi031.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 88 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-26T17:56:04.250 INFO:teuthology.orchestra.run.smithi031.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 88 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-26T17:56:04.250 INFO:teuthology.orchestra.run.smithi031.stdout:max_osd 12 2024-04-26T17:56:04.250 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 up in weight 1 up_from 8 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6802/1957398328,v1:172.21.15.31:6803/1957398328] [v2:172.21.15.31:6804/1957398328,v1:172.21.15.31:6805/1957398328] exists,up 6f684ee5-eeee-454a-a00b-e5d6d2901572 2024-04-26T17:56:04.250 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 up in weight 1 up_from 13 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6800/4209866427,v1:172.21.15.97:6801/4209866427] [v2:172.21.15.97:6802/4209866427,v1:172.21.15.97:6803/4209866427] exists,up 08be6b26-cb60-41f7-be8b-01a0c93ebafd 2024-04-26T17:56:04.250 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 up in weight 1 up_from 18 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6800/223535147,v1:172.21.15.186:6801/223535147] [v2:172.21.15.186:6802/223535147,v1:172.21.15.186:6803/223535147] exists,up 498216b1-a0ac-4923-ade8-ce50fb97d472 2024-04-26T17:56:04.250 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 up in weight 1 up_from 25 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6810/2640112341,v1:172.21.15.31:6811/2640112341] [v2:172.21.15.31:6812/2640112341,v1:172.21.15.31:6813/2640112341] exists,up 568a352d-a430-4f0f-9a49-6f992e0ceae1 2024-04-26T17:56:04.250 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 up in weight 1 up_from 33 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6808/1359263143,v1:172.21.15.97:6809/1359263143] [v2:172.21.15.97:6810/1359263143,v1:172.21.15.97:6811/1359263143] exists,up 112ccbf7-24e8-473c-9852-2e7167dd3108 2024-04-26T17:56:04.250 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 up in weight 1 up_from 38 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6808/2006134626,v1:172.21.15.186:6809/2006134626] [v2:172.21.15.186:6810/2006134626,v1:172.21.15.186:6811/2006134626] exists,up eb17759e-1ad8-4bda-9f61-54d2a9781535 2024-04-26T17:56:04.250 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 up in weight 1 up_from 46 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6818/2955759395,v1:172.21.15.31:6819/2955759395] [v2:172.21.15.31:6820/2955759395,v1:172.21.15.31:6821/2955759395] exists,up f1d11e38-40e9-4db4-868f-145f1ffc0d30 2024-04-26T17:56:04.250 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 up in weight 1 up_from 51 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6816/3940083075,v1:172.21.15.97:6817/3940083075] [v2:172.21.15.97:6818/3940083075,v1:172.21.15.97:6819/3940083075] exists,up 4e95f728-23ff-4bca-9b94-afde2662a5bf 2024-04-26T17:56:04.251 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8 up in weight 1 up_from 59 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6816/3193558170,v1:172.21.15.186:6817/3193558170] [v2:172.21.15.186:6818/3193558170,v1:172.21.15.186:6819/3193558170] exists,up 5f91c915-2568-4d20-b645-00dad2efe952 2024-04-26T17:56:04.251 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9 up in weight 1 up_from 67 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6826/3660801905,v1:172.21.15.31:6827/3660801905] [v2:172.21.15.31:6828/3660801905,v1:172.21.15.31:6829/3660801905] exists,up 12fda87d-830a-461f-a6cb-4699c52dc5e5 2024-04-26T17:56:04.251 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10 up in weight 1 up_from 72 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6824/4097344903,v1:172.21.15.97:6825/4097344903] [v2:172.21.15.97:6826/4097344903,v1:172.21.15.97:6827/4097344903] exists,up 92a639b3-918b-485c-bd30-258b4224993e 2024-04-26T17:56:04.251 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11 up in weight 1 up_from 80 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6824/1063973352,v1:172.21.15.186:6825/1063973352] [v2:172.21.15.186:6826/1063973352,v1:172.21.15.186:6827/1063973352] exists,up 842aa599-d21d-4e4a-b872-e9ac20d8c0ff 2024-04-26T17:56:04.251 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 2.4 [2,11] 2024-04-26T17:56:04.251 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 2.1d [8,11] 2024-04-26T17:56:04.251 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.3 [4,9] 2024-04-26T17:56:04.251 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.1e [3,9] 2024-04-26T17:56:04.251 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.23 [4,7] 2024-04-26T17:56:04.251 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.27 [4,8] 2024-04-26T17:56:04.251 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.33 [5,9] 2024-04-26T17:56:04.251 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/718966776 expires 2024-04-27T17:45:48.218816+0000 2024-04-26T17:56:04.252 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/2035551671 expires 2024-04-27T17:45:48.218816+0000 2024-04-26T17:56:04.252 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6801/153341213 expires 2024-04-27T17:45:48.218816+0000 2024-04-26T17:56:04.252 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/810461143 expires 2024-04-27T17:45:21.129062+0000 2024-04-26T17:56:04.252 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/3148082955 expires 2024-04-27T17:45:21.129062+0000 2024-04-26T17:56:04.252 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6800/545710545 expires 2024-04-27T17:45:21.129062+0000 2024-04-26T17:56:04.252 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6801/545710545 expires 2024-04-27T17:45:21.129062+0000 2024-04-26T17:56:04.252 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/288292084 expires 2024-04-27T17:45:48.218816+0000 2024-04-26T17:56:04.252 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/3101596726 expires 2024-04-27T17:45:21.129062+0000 2024-04-26T17:56:04.252 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6800/153341213 expires 2024-04-27T17:45:48.218816+0000 2024-04-26T17:56:04.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.251+0000 7f757b807700 1 -- 172.21.15.31:0/2301052964 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f75600a3c40 msgr2=0x7f75600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:04.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.251+0000 7f757b807700 1 --2- 172.21.15.31:0/2301052964 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f75600a3c40 0x7f75600a6100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f756400b5c0 tx=0x7f7564005fb0 comp rx=0 tx=0).stop 2024-04-26T17:56:04.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.251+0000 7f757b807700 1 -- 172.21.15.31:0/2301052964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7574101740 msgr2=0x7f7574199560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:04.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.251+0000 7f757b807700 1 --2- 172.21.15.31:0/2301052964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7574101740 0x7f7574199560 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f757000d9b0 tx=0x7f757000dcc0 comp rx=0 tx=0).stop 2024-04-26T17:56:04.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.251+0000 7f757b807700 1 -- 172.21.15.31:0/2301052964 shutdown_connections 2024-04-26T17:56:04.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.251+0000 7f757b807700 1 --2- 172.21.15.31:0/2301052964 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f75600a3c40 0x7f75600a6100 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:04.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.251+0000 7f757b807700 1 --2- 172.21.15.31:0/2301052964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75741050c0 0x7f75741a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:04.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.251+0000 7f757b807700 1 --2- 172.21.15.31:0/2301052964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7574101740 0x7f7574199560 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.251+0000 7f757b807700 1 --2- 172.21.15.31:0/2301052964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7574104720 0x7f7574199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.251+0000 7f757b807700 1 -- 172.21.15.31:0/2301052964 >> 172.21.15.31:0/2301052964 conn(0x7f75740fd080 msgr2=0x7f7574107aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.251+0000 7f757b807700 1 -- 172.21.15.31:0/2301052964 shutdown_connections 2024-04-26T17:56:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:04.251+0000 7f757b807700 1 -- 172.21.15.31:0/2301052964 wait complete. 2024-04-26T17:56:04.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:03 smithi186 ceph-mon[32797]: pgmap v387: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:56:04.967 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-26T17:56:04.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:56:05.235 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:05.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:04 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2301052964' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T17:56:05.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:04 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2301052964' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T17:56:05.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:04 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2301052964' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T17:56:06.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:05 smithi186 ceph-mon[32797]: pgmap v388: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:06.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:05 smithi031 ceph-mon[29164]: pgmap v388: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:06.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:05 smithi097 ceph-mon[32419]: pgmap v388: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:07.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.006+0000 7fb884712700 1 -- 172.21.15.31:0/1850606513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb87c1094a0 msgr2=0x7fb87c109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:07.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.006+0000 7fb884712700 1 --2- 172.21.15.31:0/1850606513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb87c1094a0 0x7fb87c109880 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb878009990 tx=0x7fb878009ca0 comp rx=0 tx=0).stop 2024-04-26T17:56:07.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.006+0000 7fb884712700 1 -- 172.21.15.31:0/1850606513 shutdown_connections 2024-04-26T17:56:07.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.006+0000 7fb884712700 1 --2- 172.21.15.31:0/1850606513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb87c06c510 0x7fb87c1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:07.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.006+0000 7fb884712700 1 --2- 172.21.15.31:0/1850606513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb87c1094a0 0x7fb87c109880 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:07.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.006+0000 7fb884712700 1 --2- 172.21.15.31:0/1850606513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb87c06bb70 0x7fb87c06bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:07.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.006+0000 7fb884712700 1 -- 172.21.15.31:0/1850606513 >> 172.21.15.31:0/1850606513 conn(0x7fb87c0792f0 msgr2=0x7fb87c079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:07.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.006+0000 7fb884712700 1 -- 172.21.15.31:0/1850606513 shutdown_connections 2024-04-26T17:56:07.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.006+0000 7fb884712700 1 -- 172.21.15.31:0/1850606513 wait complete. 2024-04-26T17:56:07.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.006+0000 7fb884712700 1 Processor -- start 2024-04-26T17:56:07.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.006+0000 7fb884712700 1 -- start start 2024-04-26T17:56:07.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb884712700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb87c06bb70 0x7fb87c19f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:07.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb884712700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb87c06c510 0x7fb87c19fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:07.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb884712700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb87c1094a0 0x7fb87c1998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:07.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb884712700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c113590 con 0x7fb87c1094a0 2024-04-26T17:56:07.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb884712700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c113410 con 0x7fb87c06bb70 2024-04-26T17:56:07.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb884712700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c113710 con 0x7fb87c06c510 2024-04-26T17:56:07.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb882caf700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb87c1094a0 0x7fb87c1998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb882caf700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb87c1094a0 0x7fb87c1998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45660/0 (socket says 172.21.15.31:45660) 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb882caf700 1 -- 172.21.15.31:0/2448786890 learned_addr learned my addr 172.21.15.31:0/2448786890 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb881cad700 1 --2- 172.21.15.31:0/2448786890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb87c06c510 0x7fb87c19fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb8824ae700 1 --2- 172.21.15.31:0/2448786890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb87c06bb70 0x7fb87c19f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb882caf700 1 -- 172.21.15.31:0/2448786890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb87c06c510 msgr2=0x7fb87c19fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb882caf700 1 --2- 172.21.15.31:0/2448786890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb87c06c510 0x7fb87c19fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb882caf700 1 -- 172.21.15.31:0/2448786890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb87c06bb70 msgr2=0x7fb87c19f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb882caf700 1 --2- 172.21.15.31:0/2448786890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb87c06bb70 0x7fb87c19f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb882caf700 1 -- 172.21.15.31:0/2448786890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87c19a030 con 0x7fb87c1094a0 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb882caf700 1 --2- 172.21.15.31:0/2448786890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb87c1094a0 0x7fb87c1998d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb874007ee0 tx=0x7fb87400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.007+0000 7fb8737fe700 1 -- 172.21.15.31:0/2448786890 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb874004bb0 con 0x7fb87c1094a0 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.008+0000 7fb884712700 1 -- 172.21.15.31:0/2448786890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb87c19a320 con 0x7fb87c1094a0 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.008+0000 7fb8737fe700 1 -- 172.21.15.31:0/2448786890 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb874004d50 con 0x7fb87c1094a0 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.008+0000 7fb8737fe700 1 -- 172.21.15.31:0/2448786890 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb874005920 con 0x7fb87c1094a0 2024-04-26T17:56:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.008+0000 7fb884712700 1 -- 172.21.15.31:0/2448786890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb87c1a65e0 con 0x7fb87c1094a0 2024-04-26T17:56:07.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.008+0000 7fb884712700 1 -- 172.21.15.31:0/2448786890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb87c06a110 con 0x7fb87c1094a0 2024-04-26T17:56:07.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.010+0000 7fb8737fe700 1 -- 172.21.15.31:0/2448786890 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb87401a020 con 0x7fb87c1094a0 2024-04-26T17:56:07.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.010+0000 7fb8737fe700 1 --2- 172.21.15.31:0/2448786890 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb8680a3c40 0x7fb8680a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:07.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.010+0000 7fb8824ae700 1 --2- 172.21.15.31:0/2448786890 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb8680a3c40 0x7fb8680a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:07.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.010+0000 7fb8737fe700 1 -- 172.21.15.31:0/2448786890 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fb8740cff80 con 0x7fb87c1094a0 2024-04-26T17:56:07.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.011+0000 7fb8824ae700 1 --2- 172.21.15.31:0/2448786890 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb8680a3c40 0x7fb8680a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb87800b5c0 tx=0x7fb878005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:07.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.014+0000 7fb8737fe700 1 -- 172.21.15.31:0/2448786890 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb874010070 con 0x7fb87c1094a0 2024-04-26T17:56:07.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.268+0000 7fb884712700 1 -- 172.21.15.31:0/2448786890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb87c19ae00 con 0x7fb87c1094a0 2024-04-26T17:56:07.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.268+0000 7fb8737fe700 1 -- 172.21.15.31:0/2448786890 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11871 (secure 0 0 0) 0x7fb87409ca30 con 0x7fb87c1094a0 2024-04-26T17:56:07.269 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:07.269 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T17:53:25.754649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24457,24473,24446]},"id":1}]} 2024-04-26T17:56:07.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.271+0000 7fb884712700 1 -- 172.21.15.31:0/2448786890 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb8680a3c40 msgr2=0x7fb8680a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:07.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.271+0000 7fb884712700 1 --2- 172.21.15.31:0/2448786890 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb8680a3c40 0x7fb8680a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb87800b5c0 tx=0x7fb878005960 comp rx=0 tx=0).stop 2024-04-26T17:56:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.271+0000 7fb884712700 1 -- 172.21.15.31:0/2448786890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb87c1094a0 msgr2=0x7fb87c1998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.271+0000 7fb884712700 1 --2- 172.21.15.31:0/2448786890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb87c1094a0 0x7fb87c1998d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb874007ee0 tx=0x7fb87400d3b0 comp rx=0 tx=0).stop 2024-04-26T17:56:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.271+0000 7fb884712700 1 -- 172.21.15.31:0/2448786890 shutdown_connections 2024-04-26T17:56:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.271+0000 7fb884712700 1 --2- 172.21.15.31:0/2448786890 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fb8680a3c40 0x7fb8680a6100 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.271+0000 7fb884712700 1 --2- 172.21.15.31:0/2448786890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb87c06bb70 0x7fb87c19f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.271+0000 7fb884712700 1 --2- 172.21.15.31:0/2448786890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb87c1094a0 0x7fb87c1998d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.271+0000 7fb884712700 1 --2- 172.21.15.31:0/2448786890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb87c06c510 0x7fb87c19fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.271+0000 7fb884712700 1 -- 172.21.15.31:0/2448786890 >> 172.21.15.31:0/2448786890 conn(0x7fb87c0792f0 msgr2=0x7fb87c100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.271+0000 7fb884712700 1 -- 172.21.15.31:0/2448786890 shutdown_connections 2024-04-26T17:56:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:07.271+0000 7fb884712700 1 -- 172.21.15.31:0/2448786890 wait complete. 2024-04-26T17:56:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-04-26T17:56:08.086 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:56:08.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:07 smithi186 ceph-mon[32797]: pgmap v389: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:56:08.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:07 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2448786890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:08.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:07 smithi031 ceph-mon[29164]: pgmap v389: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:56:08.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:07 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2448786890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:08.338 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:08.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:07 smithi097 ceph-mon[32419]: pgmap v389: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:56:08.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:07 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2448786890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:10.107 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:09 smithi031 ceph-mon[29164]: pgmap v390: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:10.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.106+0000 7fd886db4700 1 -- 172.21.15.31:0/744353262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd880102310 msgr2=0x7fd880102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:10.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.106+0000 7fd886db4700 1 --2- 172.21.15.31:0/744353262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd880102310 0x7fd880102790 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009990 tx=0x7fd86c009ca0 comp rx=0 tx=0).stop 2024-04-26T17:56:10.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.106+0000 7fd886db4700 1 -- 172.21.15.31:0/744353262 shutdown_connections 2024-04-26T17:56:10.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.106+0000 7fd886db4700 1 --2- 172.21.15.31:0/744353262 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd880102cd0 0x7fd88010f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:10.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.106+0000 7fd886db4700 1 --2- 172.21.15.31:0/744353262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd880108380 0x7fd880108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:10.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.106+0000 7fd886db4700 1 --2- 172.21.15.31:0/744353262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd880102310 0x7fd880102790 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:10.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.106+0000 7fd886db4700 1 -- 172.21.15.31:0/744353262 >> 172.21.15.31:0/744353262 conn(0x7fd880079290 msgr2=0x7fd8800796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:10.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.106+0000 7fd886db4700 1 -- 172.21.15.31:0/744353262 shutdown_connections 2024-04-26T17:56:10.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.106+0000 7fd886db4700 1 -- 172.21.15.31:0/744353262 wait complete. 2024-04-26T17:56:10.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.107+0000 7fd886db4700 1 Processor -- start 2024-04-26T17:56:10.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.107+0000 7fd886db4700 1 -- start start 2024-04-26T17:56:10.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.107+0000 7fd886db4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd880102310 0x7fd88019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:10.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.107+0000 7fd886db4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd880102cd0 0x7fd88019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:10.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.107+0000 7fd886db4700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd880108380 0x7fd8801987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:10.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.107+0000 7fd886db4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd8801124a0 con 0x7fd880102cd0 2024-04-26T17:56:10.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.107+0000 7fd886db4700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd880112320 con 0x7fd880108380 2024-04-26T17:56:10.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.107+0000 7fd886db4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd880112620 con 0x7fd880102310 2024-04-26T17:56:10.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.108+0000 7fd884b50700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd880102310 0x7fd88019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:10.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.108+0000 7fd884b50700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd880102310 0x7fd88019e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54680/0 (socket says 172.21.15.31:54680) 2024-04-26T17:56:10.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.108+0000 7fd884b50700 1 -- 172.21.15.31:0/56595108 learned_addr learned my addr 172.21.15.31:0/56595108 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:10.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.108+0000 7fd87ffff700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd880102cd0 0x7fd88019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:10.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.108+0000 7fd885351700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd880108380 0x7fd8801987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:10.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.108+0000 7fd884b50700 1 -- 172.21.15.31:0/56595108 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd880108380 msgr2=0x7fd8801987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:10.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.108+0000 7fd884b50700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd880108380 0x7fd8801987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:10.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.108+0000 7fd884b50700 1 -- 172.21.15.31:0/56595108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd880102cd0 msgr2=0x7fd88019eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:10.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.108+0000 7fd884b50700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd880102cd0 0x7fd88019eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:10.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.108+0000 7fd884b50700 1 -- 172.21.15.31:0/56595108 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd880198fc0 con 0x7fd880102310 2024-04-26T17:56:10.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.108+0000 7fd885351700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd880108380 0x7fd8801987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:56:10.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.108+0000 7fd884b50700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd880102310 0x7fd88019e660 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd874009970 tx=0x7fd874009c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:10.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.108+0000 7fd87dffb700 1 -- 172.21.15.31:0/56595108 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87401c070 con 0x7fd880102310 2024-04-26T17:56:10.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.109+0000 7fd87dffb700 1 -- 172.21.15.31:0/56595108 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd87400d260 con 0x7fd880102310 2024-04-26T17:56:10.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.109+0000 7fd87dffb700 1 -- 172.21.15.31:0/56595108 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd874017a60 con 0x7fd880102310 2024-04-26T17:56:10.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.109+0000 7fd886db4700 1 -- 172.21.15.31:0/56595108 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8801992b0 con 0x7fd880102310 2024-04-26T17:56:10.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.109+0000 7fd886db4700 1 -- 172.21.15.31:0/56595108 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd88006b990 con 0x7fd880102310 2024-04-26T17:56:10.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.110+0000 7fd886db4700 1 -- 172.21.15.31:0/56595108 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd880103a30 con 0x7fd880102310 2024-04-26T17:56:10.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.111+0000 7fd87dffb700 1 -- 172.21.15.31:0/56595108 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd87401f070 con 0x7fd880102310 2024-04-26T17:56:10.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.112+0000 7fd87dffb700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd8700a3c40 0x7fd8700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:10.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.112+0000 7fd87ffff700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd8700a3c40 0x7fd8700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:10.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.112+0000 7fd87dffb700 1 -- 172.21.15.31:0/56595108 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fd8740d18e0 con 0x7fd880102310 2024-04-26T17:56:10.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.112+0000 7fd87ffff700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd8700a3c40 0x7fd8700a6100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00b2e0 tx=0x7fd86c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:10.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.116+0000 7fd87dffb700 1 -- 172.21.15.31:0/56595108 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd87409e290 con 0x7fd880102310 2024-04-26T17:56:10.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:09 smithi186 ceph-mon[32797]: pgmap v390: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:10.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.375+0000 7fd886db4700 1 -- 172.21.15.31:0/56595108 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd88006a110 con 0x7fd880102310 2024-04-26T17:56:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.376+0000 7fd87dffb700 1 -- 172.21.15.31:0/56595108 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11871 (secure 0 0 0) 0x7fd87409db90 con 0x7fd880102310 2024-04-26T17:56:10.376 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:10.377 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T17:53:25.754649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24457,24473,24446]},"id":1}]} 2024-04-26T17:56:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.379+0000 7fd886db4700 1 -- 172.21.15.31:0/56595108 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd8700a3c40 msgr2=0x7fd8700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.379+0000 7fd886db4700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd8700a3c40 0x7fd8700a6100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00b2e0 tx=0x7fd86c005960 comp rx=0 tx=0).stop 2024-04-26T17:56:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.379+0000 7fd886db4700 1 -- 172.21.15.31:0/56595108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd880102310 msgr2=0x7fd88019e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.379+0000 7fd886db4700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd880102310 0x7fd88019e660 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd874009970 tx=0x7fd874009c80 comp rx=0 tx=0).stop 2024-04-26T17:56:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.379+0000 7fd886db4700 1 -- 172.21.15.31:0/56595108 shutdown_connections 2024-04-26T17:56:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.379+0000 7fd886db4700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fd8700a3c40 0x7fd8700a6100 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.379+0000 7fd886db4700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd880108380 0x7fd8801987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.379+0000 7fd886db4700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd880102310 0x7fd88019e660 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.379+0000 7fd886db4700 1 --2- 172.21.15.31:0/56595108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd880102cd0 0x7fd88019eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.379+0000 7fd886db4700 1 -- 172.21.15.31:0/56595108 >> 172.21.15.31:0/56595108 conn(0x7fd880079290 msgr2=0x7fd880101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.379+0000 7fd886db4700 1 -- 172.21.15.31:0/56595108 shutdown_connections 2024-04-26T17:56:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:10.379+0000 7fd886db4700 1 -- 172.21.15.31:0/56595108 wait complete. 2024-04-26T17:56:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-04-26T17:56:10.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:09 smithi097 ceph-mon[32419]: pgmap v390: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:11.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:10 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/56595108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:11.090 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T17:56:11.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:10 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/56595108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:11.340 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:11.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:10 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/56595108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:12.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:11 smithi031 ceph-mon[29164]: pgmap v391: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:12.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:11 smithi186 ceph-mon[32797]: pgmap v391: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:12.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:11 smithi097 ceph-mon[32419]: pgmap v391: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:13.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.005+0000 7f6b9d262700 1 -- 172.21.15.31:0/501393441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b98075650 msgr2=0x7f6b98075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:13.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.005+0000 7f6b9d262700 1 --2- 172.21.15.31:0/501393441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b98075650 0x7f6b98075a30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009990 tx=0x7f6b88009ca0 comp rx=0 tx=0).stop 2024-04-26T17:56:13.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.005+0000 7f6b9d262700 1 -- 172.21.15.31:0/501393441 shutdown_connections 2024-04-26T17:56:13.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.005+0000 7f6b9d262700 1 --2- 172.21.15.31:0/501393441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b98114620 0x7f6b98116a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:13.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.005+0000 7f6b9d262700 1 --2- 172.21.15.31:0/501393441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b98075650 0x7f6b98075a30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:13.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.005+0000 7f6b9d262700 1 --2- 172.21.15.31:0/501393441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98075f70 0x7f6b981140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:13.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.005+0000 7f6b9d262700 1 -- 172.21.15.31:0/501393441 >> 172.21.15.31:0/501393441 conn(0x7f6b9807a590 msgr2=0x7f6b9807a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:13.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.005+0000 7f6b9d262700 1 -- 172.21.15.31:0/501393441 shutdown_connections 2024-04-26T17:56:13.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.005+0000 7f6b9d262700 1 -- 172.21.15.31:0/501393441 wait complete. 2024-04-26T17:56:13.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.005+0000 7f6b9d262700 1 Processor -- start 2024-04-26T17:56:13.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.005+0000 7f6b9d262700 1 -- start start 2024-04-26T17:56:13.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b9d262700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b98075650 0x7f6b981a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:13.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b9d262700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98075f70 0x7f6b981a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:13.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b9d262700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b98114620 0x7f6b981a49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:13.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b9d262700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6b98119c20 con 0x7f6b98114620 2024-04-26T17:56:13.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b9d262700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6b98119aa0 con 0x7f6b98075650 2024-04-26T17:56:13.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b9d262700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6b98119da0 con 0x7f6b98075f70 2024-04-26T17:56:13.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b977fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b98114620 0x7f6b981a49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:13.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b96ffd700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b98075650 0x7f6b981a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:13.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b977fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b98114620 0x7f6b981a49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45712/0 (socket says 172.21.15.31:45712) 2024-04-26T17:56:13.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b977fe700 1 -- 172.21.15.31:0/1901348151 learned_addr learned my addr 172.21.15.31:0/1901348151 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:13.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b967fc700 1 --2- 172.21.15.31:0/1901348151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98075f70 0x7f6b981a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:13.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b977fe700 1 -- 172.21.15.31:0/1901348151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98075f70 msgr2=0x7f6b981a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:13.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b977fe700 1 --2- 172.21.15.31:0/1901348151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98075f70 0x7f6b981a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:13.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b977fe700 1 -- 172.21.15.31:0/1901348151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b98075650 msgr2=0x7f6b981a00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:13.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b977fe700 1 --2- 172.21.15.31:0/1901348151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b98075650 0x7f6b981a00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:13.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b977fe700 1 -- 172.21.15.31:0/1901348151 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b981a50e0 con 0x7f6b98114620 2024-04-26T17:56:13.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.006+0000 7f6b977fe700 1 --2- 172.21.15.31:0/1901348151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b98114620 0x7f6b981a49a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6b90007d40 tx=0x7f6b9000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:13.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.007+0000 7f6b7ffff700 1 -- 172.21.15.31:0/1901348151 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b90016020 con 0x7f6b98114620 2024-04-26T17:56:13.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.007+0000 7f6b9d262700 1 -- 172.21.15.31:0/1901348151 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b981a53d0 con 0x7f6b98114620 2024-04-26T17:56:13.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.007+0000 7f6b7ffff700 1 -- 172.21.15.31:0/1901348151 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b9000ac90 con 0x7f6b98114620 2024-04-26T17:56:13.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.007+0000 7f6b7ffff700 1 -- 172.21.15.31:0/1901348151 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b900115e0 con 0x7f6b98114620 2024-04-26T17:56:13.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.007+0000 7f6b9d262700 1 -- 172.21.15.31:0/1901348151 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b981acc50 con 0x7f6b98114620 2024-04-26T17:56:13.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.008+0000 7f6b9d262700 1 -- 172.21.15.31:0/1901348151 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b9806a110 con 0x7f6b98114620 2024-04-26T17:56:13.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.009+0000 7f6b7ffff700 1 -- 172.21.15.31:0/1901348151 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6b90019020 con 0x7f6b98114620 2024-04-26T17:56:13.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.010+0000 7f6b7ffff700 1 --2- 172.21.15.31:0/1901348151 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6b800a3c40 0x7f6b800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:13.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.010+0000 7f6b96ffd700 1 --2- 172.21.15.31:0/1901348151 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6b800a3c40 0x7f6b800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:13.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.010+0000 7f6b7ffff700 1 -- 172.21.15.31:0/1901348151 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f6b900d0230 con 0x7f6b98114620 2024-04-26T17:56:13.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.010+0000 7f6b96ffd700 1 --2- 172.21.15.31:0/1901348151 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6b800a3c40 0x7f6b800a6100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800b5c0 tx=0x7f6b88005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:13.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.015+0000 7f6b7ffff700 1 -- 172.21.15.31:0/1901348151 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b90020020 con 0x7f6b98114620 2024-04-26T17:56:13.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.238+0000 7f6b9d262700 1 -- 172.21.15.31:0/1901348151 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b981acee0 con 0x7f6b98114620 2024-04-26T17:56:13.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.239+0000 7f6b7ffff700 1 -- 172.21.15.31:0/1901348151 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22577 (secure 0 0 0) 0x7f6b9009cbe0 con 0x7f6b98114620 2024-04-26T17:56:13.239 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","created":"2024-04-26T17:44:48.618250+0000","modified":"2024-04-26T17:53:49.754504+0000","last_up_change":"2024-04-26T17:52:00.798886+0000","last_in_change":"2024-04-26T17:51:40.613593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T17:48:12.307250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T17:52:58.650338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T17:53:02.488144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f684ee5-eeee-454a-a00b-e5d6d2901572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6803","nonce":1957398328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6805","nonce":1957398328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6809","nonce":1957398328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6807","nonce":1957398328}]},"public_addr":"172.21.15.31:6803/1957398328","cluster_addr":"172.21.15.31:6805/1957398328","heartbeat_back_addr":"172.21.15.31:6809/1957398328","heartbeat_front_addr":"172.21.15.31:6807/1957398328","state":["exists","up"]},{"osd":1,"uuid":"08be6b26-cb60-41f7-be8b-01a0c93ebafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6801","nonce":4209866427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6803","nonce":4209866427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6807","nonce":4209866427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6805","nonce":4209866427}]},"public_addr":"172.21.15.97:6801/4209866427","cluster_addr":"172.21.15.97:6803/4209866427","heartbeat_back_addr":"172.21.15.97:6807/4209866427","heartbeat_front_addr":"172.21.15.97:6805/4209866427","state":["exists","up"]},{"osd":2,"uuid":"498216b1-a0ac-4923-ade8-ce50fb97d472","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6801","nonce":223535147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6803","nonce":223535147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6807","nonce":223535147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6805","nonce":223535147}]},"public_addr":"172.21.15.186:6801/223535147","cluster_addr":"172.21.15.186:6803/223535147","heartbeat_back_addr":"172.21.15.186:6807/223535147","heartbeat_front_addr":"172.21.15.186:6805/223535147","state":["exists","up"]},{"osd":3,"uuid":"568a352d-a430-4f0f-9a49-6f992e0ceae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6811","nonce":2640112341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6813","nonce":2640112341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6817","nonce":2640112341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6815","nonce":2640112341}]},"public_addr":"172.21.15.31:6811/2640112341","cluster_addr":"172.21.15.31:6813/2640112341","heartbeat_back_addr":"172.21.15.31:6817/2640112341","heartbeat_front_addr":"172.21.15.31:6815/2640112341","state":["exists","up"]},{"osd":4,"uuid":"112ccbf7-24e8-473c-9852-2e7167dd3108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6809","nonce":1359263143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6811","nonce":1359263143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6815","nonce":1359263143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6813","nonce":1359263143}]},"public_addr":"172.21.15.97:6809/1359263143","cluster_addr":"172.21.15.97:6811/1359263143","heartbeat_back_addr":"172.21.15.97:6815/1359263143","heartbeat_front_addr":"172.21.15.97:6813/1359263143","state":["exists","up"]},{"osd":5,"uuid":"eb17759e-1ad8-4bda-9f61-54d2a9781535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6809","nonce":2006134626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6811","nonce":2006134626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6815","nonce":2006134626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6813","nonce":2006134626}]},"public_addr":"172.21.15.186:6809/2006134626","cluster_addr":"172.21.15.186:6811/2006134626","heartbeat_back_addr":"172.21.15.186:6815/2006134626","heartbeat_front_addr":"172.21.15.186:6813/2006134626","state":["exists","up"]},{"osd":6,"uuid":"f1d11e38-40e9-4db4-868f-145f1ffc0d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6819","nonce":2955759395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6821","nonce":2955759395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6825","nonce":2955759395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6823","nonce":2955759395}]},"public_addr":"172.21.15.31:6819/2955759395","cluster_addr":"172.21.15.31:6821/2955759395","heartbeat_back_addr":"172.21.15.31:6825/2955759395","heartbeat_front_addr":"172.21.15.31:6823/2955759395","state":["exists","up"]},{"osd":7,"uuid":"4e95f728-23ff-4bca-9b94-afde2662a5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6817","nonce":3940083075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6819","nonce":3940083075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6823","nonce":3940083075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6821","nonce":3940083075}]},"public_addr":"172.21.15.97:6817/3940083075","cluster_addr":"172.21.15.97:6819/3940083075","heartbeat_back_addr":"172.21.15.97:6823/3940083075","heartbeat_front_addr":"172.21.15.97:6821/3940083075","state":["exists","up"]},{"osd":8,"uuid":"5f91c915-2568-4d20-b645-00dad2efe952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6817","nonce":3193558170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6819","nonce":3193558170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6823","nonce":3193558170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6821","nonce":3193558170}]},"public_addr":"172.21.15.186:6817/3193558170","cluster_addr":"172.21.15.186:6819/3193558170","heartbeat_back_addr":"172.21.15.186:6823/3193558170","heartbeat_front_addr":"172.21.15.186:6821/3193558170","state":["exists","up"]},{"osd":9,"uuid":"12fda87d-830a-461f-a6cb-4699c52dc5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6827","nonce":3660801905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6829","nonce":3660801905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6833","nonce":3660801905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6831","nonce":3660801905}]},"public_addr":"172.21.15.31:6827/3660801905","cluster_addr":"172.21.15.31:6829/3660801905","heartbeat_back_addr":"172.21.15.31:6833/3660801905","heartbeat_front_addr":"172.21.15.31:6831/3660801905","state":["exists","up"]},{"osd":10,"uuid":"92a639b3-918b-485c-bd30-258b4224993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6825","nonce":4097344903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6827","nonce":4097344903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6831","nonce":4097344903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6829","nonce":4097344903}]},"public_addr":"172.21.15.97:6825/4097344903","cluster_addr":"172.21.15.97:6827/4097344903","heartbeat_back_addr":"172.21.15.97:6831/4097344903","heartbeat_front_addr":"172.21.15.97:6829/4097344903","state":["exists","up"]},{"osd":11,"uuid":"842aa599-d21d-4e4a-b872-e9ac20d8c0ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6825","nonce":1063973352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6827","nonce":1063973352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6831","nonce":1063973352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6829","nonce":1063973352}]},"public_addr":"172.21.15.186:6825/1063973352","cluster_addr":"172.21.15.186:6827/1063973352","heartbeat_back_addr":"172.21.15.186:6831/1063973352","heartbeat_front_addr":"172.21.15.186:6829/1063973352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:21.305331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:44.877932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:08.784665+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:34.381762+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:58.999358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:23.620237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:49.961719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:15.363702+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:40.599703+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:07.391645+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:32.800851+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:59.384148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/718966776":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/2035551671":"2024-04-27T17:45:48.218816+0000","172.21.15.31:6801/153341213":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/810461143":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/3148082955":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6801/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/288292084":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/3101596726":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/153341213":"2024-04-27T17:45:48.218816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T17:56:13.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.241+0000 7f6b9d262700 1 -- 172.21.15.31:0/1901348151 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6b800a3c40 msgr2=0x7f6b800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:13.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.241+0000 7f6b9d262700 1 --2- 172.21.15.31:0/1901348151 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6b800a3c40 0x7f6b800a6100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800b5c0 tx=0x7f6b88005960 comp rx=0 tx=0).stop 2024-04-26T17:56:13.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.241+0000 7f6b9d262700 1 -- 172.21.15.31:0/1901348151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b98114620 msgr2=0x7f6b981a49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:13.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.241+0000 7f6b9d262700 1 --2- 172.21.15.31:0/1901348151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b98114620 0x7f6b981a49a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6b90007d40 tx=0x7f6b9000a340 comp rx=0 tx=0).stop 2024-04-26T17:56:13.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.241+0000 7f6b9d262700 1 -- 172.21.15.31:0/1901348151 shutdown_connections 2024-04-26T17:56:13.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.241+0000 7f6b9d262700 1 --2- 172.21.15.31:0/1901348151 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f6b800a3c40 0x7f6b800a6100 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:13.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.241+0000 7f6b9d262700 1 --2- 172.21.15.31:0/1901348151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6b98075650 0x7f6b981a00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:13.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.241+0000 7f6b9d262700 1 --2- 172.21.15.31:0/1901348151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b98114620 0x7f6b981a49a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:13.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.241+0000 7f6b9d262700 1 --2- 172.21.15.31:0/1901348151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98075f70 0x7f6b981a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:13.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.242+0000 7f6b9d262700 1 -- 172.21.15.31:0/1901348151 >> 172.21.15.31:0/1901348151 conn(0x7f6b9807a590 msgr2=0x7f6b981133e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:13.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.242+0000 7f6b9d262700 1 -- 172.21.15.31:0/1901348151 shutdown_connections 2024-04-26T17:56:13.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:13.242+0000 7f6b9d262700 1 -- 172.21.15.31:0/1901348151 wait complete. 2024-04-26T17:56:13.876 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T17:53:05.425897+0000', 'modified': '2024-04-26T17:53:25.754649+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24473, 'mds_1': 24457, 'mds_2': 24446}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24428': {'gid': 24428, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.97:6835/1721162755', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 1721162755}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 1721162755}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24439': {'gid': 24439, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6835/580433615', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6834', 'nonce': 580433615}, {'type': 'v1', 'addr': '172.21.15.186:6835', 'nonce': 580433615}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6837/1510454985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6836', 'nonce': 1510454985}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 1510454985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24446': {'gid': 24446, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.97:6837/3334702590', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 3334702590}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3334702590}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.186:6839/1587952500', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 1587952500}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 1587952500}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.97:6839/3765855684', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6838', 'nonce': 3765855684}, {'type': 'v1', 'addr': '172.21.15.97:6839', 'nonce': 3765855684}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24473, 'qdb_cluster': [24457, 24473, 24446]} 2024-04-26T17:56:13.876 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T17:56:13.876 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell mds.h status 2024-04-26T17:56:14.130 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:14.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:13 smithi031 ceph-mon[29164]: pgmap v392: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:56:14.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:13 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1901348151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:56:14.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:13 smithi186 ceph-mon[32797]: pgmap v392: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:56:14.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:13 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1901348151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:56:14.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:13 smithi097 ceph-mon[32419]: pgmap v392: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T17:56:14.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:13 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1901348151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:56:15.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.782+0000 7fa836392700 1 -- 172.21.15.31:0/2978898305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa830107640 msgr2=0x7fa83010f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:15.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.782+0000 7fa836392700 1 --2- 172.21.15.31:0/2978898305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa830107640 0x7fa83010f210 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa82800b0d0 tx=0x7fa82800b3e0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.782+0000 7fa836392700 1 -- 172.21.15.31:0/2978898305 shutdown_connections 2024-04-26T17:56:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.782+0000 7fa836392700 1 --2- 172.21.15.31:0/2978898305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa830107640 0x7fa83010f210 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.782+0000 7fa836392700 1 --2- 172.21.15.31:0/2978898305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8301025b0 0x7fa830102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.782+0000 7fa836392700 1 --2- 172.21.15.31:0/2978898305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa830102f60 0x7fa830106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.782+0000 7fa836392700 1 -- 172.21.15.31:0/2978898305 >> 172.21.15.31:0/2978898305 conn(0x7fa8300792d0 msgr2=0x7fa8300796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.782+0000 7fa836392700 1 -- 172.21.15.31:0/2978898305 shutdown_connections 2024-04-26T17:56:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.782+0000 7fa836392700 1 -- 172.21.15.31:0/2978898305 wait complete. 2024-04-26T17:56:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.783+0000 7fa836392700 1 Processor -- start 2024-04-26T17:56:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.783+0000 7fa836392700 1 -- start start 2024-04-26T17:56:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.783+0000 7fa836392700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8301025b0 0x7fa830104000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.783+0000 7fa836392700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa830102f60 0x7fa830104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.783+0000 7fa836392700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa830107640 0x7fa830104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.783+0000 7fa836392700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa830112570 con 0x7fa830102f60 2024-04-26T17:56:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.783+0000 7fa836392700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa8301123f0 con 0x7fa830107640 2024-04-26T17:56:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.783+0000 7fa836392700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa8301126f0 con 0x7fa8301025b0 2024-04-26T17:56:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.783+0000 7fa82f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa830102f60 0x7fa830104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.783+0000 7fa82f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa830102f60 0x7fa830104540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53408/0 (socket says 172.21.15.31:53408) 2024-04-26T17:56:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.783+0000 7fa82f7fe700 1 -- 172.21.15.31:0/52726556 learned_addr learned my addr 172.21.15.31:0/52726556 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.783+0000 7fa82ffff700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8301025b0 0x7fa830104000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.783+0000 7fa83492f700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa830107640 0x7fa830104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.784+0000 7fa82f7fe700 1 -- 172.21.15.31:0/52726556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8301025b0 msgr2=0x7fa830104000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.784+0000 7fa82f7fe700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8301025b0 0x7fa830104000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.784+0000 7fa82f7fe700 1 -- 172.21.15.31:0/52726556 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa830107640 msgr2=0x7fa830104a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.784+0000 7fa82f7fe700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa830107640 0x7fa830104a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.784+0000 7fa82f7fe700 1 -- 172.21.15.31:0/52726556 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8301a80e0 con 0x7fa830102f60 2024-04-26T17:56:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.784+0000 7fa83492f700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa830107640 0x7fa830104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:56:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.784+0000 7fa82f7fe700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa830102f60 0x7fa830104540 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa81800b500 tx=0x7fa81800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.784+0000 7fa82d7fa700 1 -- 172.21.15.31:0/52726556 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818015020 con 0x7fa830102f60 2024-04-26T17:56:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.784+0000 7fa82d7fa700 1 -- 172.21.15.31:0/52726556 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa818011ce0 con 0x7fa830102f60 2024-04-26T17:56:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.784+0000 7fa836392700 1 -- 172.21.15.31:0/52726556 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8301a83d0 con 0x7fa830102f60 2024-04-26T17:56:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.784+0000 7fa82d7fa700 1 -- 172.21.15.31:0/52726556 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818010d40 con 0x7fa830102f60 2024-04-26T17:56:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.784+0000 7fa836392700 1 -- 172.21.15.31:0/52726556 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8301a8960 con 0x7fa830102f60 2024-04-26T17:56:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.786+0000 7fa82d7fa700 1 -- 172.21.15.31:0/52726556 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa818018020 con 0x7fa830102f60 2024-04-26T17:56:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.787+0000 7fa82d7fa700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa81c0a3cc0 0x7fa81c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.787+0000 7fa82ffff700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa81c0a3cc0 0x7fa81c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.787+0000 7fa82d7fa700 1 -- 172.21.15.31:0/52726556 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fa8180cfa70 con 0x7fa830102f60 2024-04-26T17:56:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.787+0000 7fa82ffff700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa81c0a3cc0 0x7fa81c0a6180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa820005950 tx=0x7fa82000a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa840c24b80 1 Processor -- start 2024-04-26T17:56:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa840c24b80 1 -- start start 2024-04-26T17:56:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa840c24b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 0x55edf1c83eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa840c24b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c843f0 0x55edf1c7dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa840c24b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c7e2d0 0x55edf1c7e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa840c24b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55edf1bd06f0 con 0x55edf1c7e2d0 2024-04-26T17:56:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa840c24b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55edf1b6f590 con 0x55edf1c843f0 2024-04-26T17:56:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa840c24b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55edf1bcc6b0 con 0x55edf1c83ad0 2024-04-26T17:56:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa83492f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c7e2d0 0x55edf1c7e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa83492f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c7e2d0 0x55edf1c7e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53410/0 (socket says 172.21.15.31:53410) 2024-04-26T17:56:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa83492f700 1 -- 172.21.15.31:0/2506384069 learned_addr learned my addr 172.21.15.31:0/2506384069 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa82f7fe700 1 --2- 172.21.15.31:0/2506384069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c843f0 0x55edf1c7dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa82ffff700 1 --2- 172.21.15.31:0/2506384069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 0x55edf1c83eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa83492f700 1 -- 172.21.15.31:0/2506384069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 msgr2=0x55edf1c83eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa83492f700 1 --2- 172.21.15.31:0/2506384069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 0x55edf1c83eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa83492f700 1 -- 172.21.15.31:0/2506384069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c843f0 msgr2=0x55edf1c7dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa83492f700 1 --2- 172.21.15.31:0/2506384069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c843f0 0x55edf1c7dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa83492f700 1 -- 172.21.15.31:0/2506384069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edf1b1d880 con 0x55edf1c7e2d0 2024-04-26T17:56:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa82f7fe700 1 --2- 172.21.15.31:0/2506384069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c843f0 0x55edf1c7dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:56:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.788+0000 7fa83492f700 1 --2- 172.21.15.31:0/2506384069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c7e2d0 0x55edf1c7e710 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa828009dc0 tx=0x7fa828009df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e7a660d719ac951 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.789+0000 7fa8267fc700 1 -- 172.21.15.31:0/2506384069 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa828020030 con 0x55edf1c7e2d0 2024-04-26T17:56:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.789+0000 7fa8267fc700 1 -- 172.21.15.31:0/2506384069 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa828009250 con 0x55edf1c7e2d0 2024-04-26T17:56:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.789+0000 7fa8267fc700 1 -- 172.21.15.31:0/2506384069 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa828007dd0 con 0x55edf1c7e2d0 2024-04-26T17:56:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.789+0000 7fa840c24b80 1 -- 172.21.15.31:0/2506384069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c7e2d0 msgr2=0x55edf1c7e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.789+0000 7fa840c24b80 1 --2- 172.21.15.31:0/2506384069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c7e2d0 0x55edf1c7e710 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa828009dc0 tx=0x7fa828009df0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.789+0000 7fa840c24b80 1 -- 172.21.15.31:0/2506384069 shutdown_connections 2024-04-26T17:56:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.789+0000 7fa840c24b80 1 --2- 172.21.15.31:0/2506384069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c7e2d0 0x55edf1c7e710 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.789+0000 7fa840c24b80 1 --2- 172.21.15.31:0/2506384069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 0x55edf1c83eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.789+0000 7fa840c24b80 1 --2- 172.21.15.31:0/2506384069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c843f0 0x55edf1c7dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.789+0000 7fa840c24b80 1 -- 172.21.15.31:0/2506384069 >> 172.21.15.31:0/2506384069 conn(0x55edf1baacf0 msgr2=0x55edf1c791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.790+0000 7fa840c24b80 1 -- 172.21.15.31:0/2506384069 shutdown_connections 2024-04-26T17:56:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.790+0000 7fa840c24b80 1 -- 172.21.15.31:0/2506384069 wait complete. 2024-04-26T17:56:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.790+0000 7fa840c24b80 1 -- start start 2024-04-26T17:56:15.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.790+0000 7fa840c24b80 1 Processor -- start 2024-04-26T17:56:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.790+0000 7fa840c24b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c7e2d0 0x55edf1c8a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.790+0000 7fa840c24b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 0x55edf1c87420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.790+0000 7fa840c24b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c843f0 0x55edf1c87960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.790+0000 7fa840c24b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55edf1c8aeb0 con 0x55edf1c843f0 2024-04-26T17:56:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.790+0000 7fa840c24b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55edf1b36900 con 0x55edf1c7e2d0 2024-04-26T17:56:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.790+0000 7fa840c24b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55edf1c8b030 con 0x55edf1c83ad0 2024-04-26T17:56:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.790+0000 7fa840c24b80 10 client.0 _ll_register_callbacks cb 0x55edf1b64530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:56:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.790+0000 7fa83492f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c843f0 0x55edf1c87960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.790+0000 7fa83492f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c843f0 0x55edf1c87960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53412/0 (socket says 172.21.15.31:53412) 2024-04-26T17:56:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa83492f700 1 -- 172.21.15.31:0/3888504322 learned_addr learned my addr 172.21.15.31:0/3888504322 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa82ffff700 1 --2- 172.21.15.31:0/3888504322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c7e2d0 0x55edf1c8a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa82f7fe700 1 --2- 172.21.15.31:0/3888504322 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 0x55edf1c87420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa83492f700 1 -- 172.21.15.31:0/3888504322 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 msgr2=0x55edf1c87420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa83492f700 1 --2- 172.21.15.31:0/3888504322 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 0x55edf1c87420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa83492f700 1 -- 172.21.15.31:0/3888504322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c7e2d0 msgr2=0x55edf1c8a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa83492f700 1 --2- 172.21.15.31:0/3888504322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c7e2d0 0x55edf1c8a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa83492f700 1 -- 172.21.15.31:0/3888504322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edf1c87f70 con 0x55edf1c843f0 2024-04-26T17:56:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa83492f700 1 --2- 172.21.15.31:0/3888504322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c843f0 0x55edf1c87960 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa8280062a0 tx=0x7fa828007cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d880f16690da26b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa80f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:56:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa80f7fe700 1 -- 172.21.15.31:0/3888504322 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa828029030 con 0x55edf1c843f0 2024-04-26T17:56:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa80f7fe700 1 -- 172.21.15.31:0/3888504322 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fa82800f040 con 0x55edf1c843f0 2024-04-26T17:56:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa840c24b80 1 -- 172.21.15.31:0/3888504322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa828009250 con 0x55edf1c843f0 2024-04-26T17:56:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa80f7fe700 1 -- 172.21.15.31:0/3888504322 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa828020040 con 0x55edf1c843f0 2024-04-26T17:56:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa840c24b80 10 client.15069 fetch_fsmap learned FSMap version 25 2024-04-26T17:56:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.791+0000 7fa840c24b80 1 -- 172.21.15.31:0/3888504322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55edf1c958c0 con 0x55edf1c843f0 2024-04-26T17:56:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.792+0000 7fa80f7fe700 1 -- 172.21.15.31:0/3888504322 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa828009250 con 0x55edf1c843f0 2024-04-26T17:56:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.792+0000 7fa80f7fe700 1 -- 172.21.15.31:0/3888504322 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fa828011070 con 0x55edf1c843f0 2024-04-26T17:56:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.792+0000 7fa840c24b80 10 client.15069 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T17:56:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.792+0000 7fa840c24b80 10 client.15069 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T17:56:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.792+0000 7fa840c24b80 20 client.15069 populate_metadata read hostname 'smithi031' 2024-04-26T17:56:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.792+0000 7fa840c24b80 1 --2- 172.21.15.31:0/3888504322 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55edf1c96c10 0x55edf1c98d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.792+0000 7fa840c24b80 4 client.15069 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:56:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.792+0000 7fa840c24b80 1 -- 172.21.15.31:0/3888504322 --> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edf1bde0d0 con 0x55edf1c96c10 2024-04-26T17:56:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.792+0000 7fa82ffff700 1 --2- 172.21.15.31:0/3888504322 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55edf1c96c10 0x55edf1c98d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.793+0000 7fa80f7fe700 10 client.15069 ms_handle_connect on v2:172.21.15.97:6836/3334702590 2024-04-26T17:56:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.793+0000 7fa82ffff700 1 --2- 172.21.15.31:0/3888504322 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55edf1c96c10 0x55edf1c98d90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dae928d0974c3074 server_cookie=4d00b35a1dfc3b99 in_seq=0 out_seq=0 2024-04-26T17:56:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.793+0000 7fa80f7fe700 1 -- 172.21.15.31:0/3888504322 <== mds.2 v2:172.21.15.97:6836/3334702590 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55edf1bde0d0 con 0x55edf1c96c10 2024-04-26T17:56:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.793+0000 7fa80f7fe700 10 client.15069 handle_command_reply: tid=0 2024-04-26T17:56:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.793+0000 7fa840c24b80 1 client.15069 shutdown 2024-04-26T17:56:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.794+0000 7fa840c24b80 1 -- 172.21.15.31:0/3888504322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c843f0 msgr2=0x55edf1c87960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.794+0000 7fa840c24b80 1 --2- 172.21.15.31:0/3888504322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c843f0 0x55edf1c87960 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa8280062a0 tx=0x7fa828007cc0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.794+0000 7fa840c24b80 1 -- 172.21.15.31:0/3888504322 shutdown_connections 2024-04-26T17:56:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.794+0000 7fa840c24b80 1 --2- 172.21.15.31:0/3888504322 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55edf1c96c10 0x55edf1c98d90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.794+0000 7fa840c24b80 1 --2- 172.21.15.31:0/3888504322 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c7e2d0 0x55edf1c8a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.794+0000 7fa840c24b80 1 --2- 172.21.15.31:0/3888504322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c843f0 0x55edf1c87960 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.794+0000 7fa840c24b80 1 --2- 172.21.15.31:0/3888504322 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 0x55edf1c87420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.794+0000 7fa840c24b80 1 -- 172.21.15.31:0/3888504322 >> 172.21.15.31:0/3888504322 conn(0x55edf1baacf0 msgr2=0x55edf1c78da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.794+0000 7fa80f7fe700 0 client.15069 ms_handle_reset on v2:172.21.15.97:6836/3334702590 2024-04-26T17:56:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.794+0000 7fa840c24b80 1 -- 172.21.15.31:0/3888504322 shutdown_connections 2024-04-26T17:56:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.794+0000 7fa840c24b80 1 -- 172.21.15.31:0/3888504322 wait complete. 2024-04-26T17:56:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.794+0000 7fa840c24b80 20 client.15069 trim_cache size 0 max 16384 2024-04-26T17:56:15.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.815+0000 7fa840c24b80 1 Processor -- start 2024-04-26T17:56:15.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.815+0000 7fa840c24b80 1 -- start start 2024-04-26T17:56:15.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa840c24b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c7e2d0 0x55edf1c7dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa840c24b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 0x55edf1c97250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa840c24b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c843f0 0x55edf1c78ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa840c24b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55edf1c970d0 con 0x55edf1c7e2d0 2024-04-26T17:56:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa840c24b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55edf1c977a0 con 0x55edf1c843f0 2024-04-26T17:56:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa840c24b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55edf1c97e70 con 0x55edf1c83ad0 2024-04-26T17:56:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa82ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c7e2d0 0x55edf1c7dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa82ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c7e2d0 0x55edf1c7dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53428/0 (socket says 172.21.15.31:53428) 2024-04-26T17:56:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa82ffff700 1 -- 172.21.15.31:0/3227914293 learned_addr learned my addr 172.21.15.31:0/3227914293 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa82f7fe700 1 --2- 172.21.15.31:0/3227914293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 0x55edf1c97250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa82ffff700 1 -- 172.21.15.31:0/3227914293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 msgr2=0x55edf1c97250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:15.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa83492f700 1 --2- 172.21.15.31:0/3227914293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c843f0 0x55edf1c78ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa82ffff700 1 --2- 172.21.15.31:0/3227914293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 0x55edf1c97250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa82ffff700 1 -- 172.21.15.31:0/3227914293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c843f0 msgr2=0x55edf1c78ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:15.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa82ffff700 1 --2- 172.21.15.31:0/3227914293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c843f0 0x55edf1c78ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa82ffff700 1 -- 172.21.15.31:0/3227914293 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edf1c7ed20 con 0x55edf1c7e2d0 2024-04-26T17:56:15.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa82ffff700 1 --2- 172.21.15.31:0/3227914293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c7e2d0 0x55edf1c7dcb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa82000fdf0 tx=0x7fa820012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d067f9937571e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:15.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa825ffb700 1 -- 172.21.15.31:0/3227914293 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa820016070 con 0x55edf1c7e2d0 2024-04-26T17:56:15.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa825ffb700 1 -- 172.21.15.31:0/3227914293 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa820026340 con 0x55edf1c7e2d0 2024-04-26T17:56:15.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.816+0000 7fa825ffb700 1 -- 172.21.15.31:0/3227914293 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa820004bb0 con 0x55edf1c7e2d0 2024-04-26T17:56:15.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.817+0000 7fa840c24b80 1 -- 172.21.15.31:0/3227914293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c7e2d0 msgr2=0x55edf1c7dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:15.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.817+0000 7fa840c24b80 1 --2- 172.21.15.31:0/3227914293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c7e2d0 0x55edf1c7dcb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa82000fdf0 tx=0x7fa820012000 comp rx=0 tx=0).stop 2024-04-26T17:56:15.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.817+0000 7fa840c24b80 1 -- 172.21.15.31:0/3227914293 shutdown_connections 2024-04-26T17:56:15.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.817+0000 7fa840c24b80 1 --2- 172.21.15.31:0/3227914293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c843f0 0x55edf1c78ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.817+0000 7fa840c24b80 1 --2- 172.21.15.31:0/3227914293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c7e2d0 0x55edf1c7dcb0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.817+0000 7fa840c24b80 1 --2- 172.21.15.31:0/3227914293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c83ad0 0x55edf1c97250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.817+0000 7fa840c24b80 1 -- 172.21.15.31:0/3227914293 >> 172.21.15.31:0/3227914293 conn(0x55edf1baacf0 msgr2=0x55edf1c89d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.817+0000 7fa840c24b80 1 -- 172.21.15.31:0/3227914293 shutdown_connections 2024-04-26T17:56:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.817+0000 7fa840c24b80 1 -- 172.21.15.31:0/3227914293 wait complete. 2024-04-26T17:56:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.817+0000 7fa840c24b80 1 -- start start 2024-04-26T17:56:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa840c24b80 1 Processor -- start 2024-04-26T17:56:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa840c24b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c7e2d0 0x55edf1c88e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa840c24b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c83ad0 0x55edf1c89370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa840c24b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c843f0 0x55edf1ca1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa840c24b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55edf1c98c10 con 0x55edf1c843f0 2024-04-26T17:56:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa840c24b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55edf1c98540 con 0x55edf1c83ad0 2024-04-26T17:56:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa840c24b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55edf1c992e0 con 0x55edf1c7e2d0 2024-04-26T17:56:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa840c24b80 10 client.0 _ll_register_callbacks cb 0x55edf1b64530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:56:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa83492f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c843f0 0x55edf1ca1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa83492f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c843f0 0x55edf1ca1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53432/0 (socket says 172.21.15.31:53432) 2024-04-26T17:56:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa83492f700 1 -- 172.21.15.31:0/672063649 learned_addr learned my addr 172.21.15.31:0/672063649 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa82f7fe700 1 --2- 172.21.15.31:0/672063649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c83ad0 0x55edf1c89370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa82ffff700 1 --2- 172.21.15.31:0/672063649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c7e2d0 0x55edf1c88e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa83492f700 1 -- 172.21.15.31:0/672063649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c7e2d0 msgr2=0x55edf1c88e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa83492f700 1 --2- 172.21.15.31:0/672063649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c7e2d0 0x55edf1c88e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa83492f700 1 -- 172.21.15.31:0/672063649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c83ad0 msgr2=0x55edf1c89370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa83492f700 1 --2- 172.21.15.31:0/672063649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c83ad0 0x55edf1c89370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa83492f700 1 -- 172.21.15.31:0/672063649 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edf1ca1740 con 0x55edf1c843f0 2024-04-26T17:56:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.818+0000 7fa82ffff700 1 --2- 172.21.15.31:0/672063649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c7e2d0 0x55edf1c88e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:56:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa83492f700 1 --2- 172.21.15.31:0/672063649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c843f0 0x55edf1ca1200 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa828004da0 tx=0x7fa82800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292bdc34d85d3234 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa80f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:56:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa80f7fe700 1 -- 172.21.15.31:0/672063649 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa828029070 con 0x55edf1c843f0 2024-04-26T17:56:15.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa80f7fe700 1 -- 172.21.15.31:0/672063649 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fa8280074a0 con 0x55edf1c843f0 2024-04-26T17:56:15.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa840c24b80 1 -- 172.21.15.31:0/672063649 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa820026340 con 0x55edf1c843f0 2024-04-26T17:56:15.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa80f7fe700 1 -- 172.21.15.31:0/672063649 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa8280076a0 con 0x55edf1c843f0 2024-04-26T17:56:15.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa840c24b80 10 client.15075 fetch_fsmap learned FSMap version 25 2024-04-26T17:56:15.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa840c24b80 1 -- 172.21.15.31:0/672063649 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55edf1ca2a20 con 0x55edf1c843f0 2024-04-26T17:56:15.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa80f7fe700 1 -- 172.21.15.31:0/672063649 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa820026340 con 0x55edf1c843f0 2024-04-26T17:56:15.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa80f7fe700 1 -- 172.21.15.31:0/672063649 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fa828011040 con 0x55edf1c843f0 2024-04-26T17:56:15.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa840c24b80 10 client.15075 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T17:56:15.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa840c24b80 10 client.15075 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T17:56:15.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa840c24b80 20 client.15075 populate_metadata read hostname 'smithi031' 2024-04-26T17:56:15.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa840c24b80 1 --2- 172.21.15.31:0/672063649 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55edf1d030b0 0x55edf1d01dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:15.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa840c24b80 4 client.15075 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T17:56:15.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.819+0000 7fa840c24b80 1 -- 172.21.15.31:0/672063649 --> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] -- command(tid 0: {"prefix": "status"}) -- 0x55edf1bd8170 con 0x55edf1d030b0 2024-04-26T17:56:15.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.820+0000 7fa82ffff700 1 --2- 172.21.15.31:0/672063649 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55edf1d030b0 0x55edf1d01dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.820+0000 7fa82ffff700 1 --2- 172.21.15.31:0/672063649 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55edf1d030b0 0x55edf1d01dc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f70735b71895fb6c server_cookie=af0cde782f92776f in_seq=0 out_seq=0 2024-04-26T17:56:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.820+0000 7fa80f7fe700 10 client.15075 ms_handle_connect on v2:172.21.15.97:6836/3334702590 2024-04-26T17:56:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.820+0000 7fa80f7fe700 1 -- 172.21.15.31:0/672063649 <== mds.2 v2:172.21.15.97:6836/3334702590 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55edf1bd8170 con 0x55edf1d030b0 2024-04-26T17:56:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.820+0000 7fa80f7fe700 10 client.15075 handle_command_reply: tid=0 2024-04-26T17:56:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.820+0000 7fa840c24b80 1 client.15075 shutdown 2024-04-26T17:56:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.820+0000 7fa840c24b80 1 -- 172.21.15.31:0/672063649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c843f0 msgr2=0x55edf1ca1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.820+0000 7fa840c24b80 1 --2- 172.21.15.31:0/672063649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c843f0 0x55edf1ca1200 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa828004da0 tx=0x7fa82800b890 comp rx=0 tx=0).stop 2024-04-26T17:56:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa840c24b80 1 -- 172.21.15.31:0/672063649 shutdown_connections 2024-04-26T17:56:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa840c24b80 1 --2- 172.21.15.31:0/672063649 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55edf1d030b0 0x55edf1d01dc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa840c24b80 1 --2- 172.21.15.31:0/672063649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edf1c843f0 0x55edf1ca1200 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa840c24b80 1 --2- 172.21.15.31:0/672063649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55edf1c7e2d0 0x55edf1c88e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa840c24b80 1 --2- 172.21.15.31:0/672063649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55edf1c83ad0 0x55edf1c89370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa80f7fe700 0 client.15075 ms_handle_reset on v2:172.21.15.97:6836/3334702590 2024-04-26T17:56:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa840c24b80 1 -- 172.21.15.31:0/672063649 >> 172.21.15.31:0/672063649 conn(0x55edf1baacf0 msgr2=0x55edf1c81620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa840c24b80 1 -- 172.21.15.31:0/672063649 shutdown_connections 2024-04-26T17:56:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa840c24b80 1 -- 172.21.15.31:0/672063649 wait complete. 2024-04-26T17:56:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa840c24b80 20 client.15075 trim_cache size 0 max 16384 2024-04-26T17:56:15.827 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T17:56:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "7e53e050-03f4-11ef-bc93-c7b262605968", 2024-04-26T17:56:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 2, 2024-04-26T17:56:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24446, 2024-04-26T17:56:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-26T17:56:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-26T17:56:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-26T17:56:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 176.14346668300001, 2024-04-26T17:56:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 25, 2024-04-26T17:56:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 90, 2024-04-26T17:56:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T17:56:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 218.986323016 2024-04-26T17:56:15.828 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T17:56:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa836392700 1 -- 172.21.15.31:0/52726556 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa81c0a3cc0 msgr2=0x7fa81c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa836392700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa81c0a3cc0 0x7fa81c0a6180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa820005950 tx=0x7fa82000a300 comp rx=0 tx=0).stop 2024-04-26T17:56:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa836392700 1 -- 172.21.15.31:0/52726556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa830102f60 msgr2=0x7fa830104540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa836392700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa830102f60 0x7fa830104540 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa81800b500 tx=0x7fa81800b810 comp rx=0 tx=0).stop 2024-04-26T17:56:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa836392700 1 -- 172.21.15.31:0/52726556 shutdown_connections 2024-04-26T17:56:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa836392700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fa81c0a3cc0 0x7fa81c0a6180 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa836392700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa830107640 0x7fa830104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa836392700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8301025b0 0x7fa830104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa836392700 1 --2- 172.21.15.31:0/52726556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa830102f60 0x7fa830104540 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.821+0000 7fa836392700 1 -- 172.21.15.31:0/52726556 >> 172.21.15.31:0/52726556 conn(0x7fa8300792d0 msgr2=0x7fa83006beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.822+0000 7fa836392700 1 -- 172.21.15.31:0/52726556 shutdown_connections 2024-04-26T17:56:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:15.822+0000 7fa836392700 1 -- 172.21.15.31:0/52726556 wait complete. 2024-04-26T17:56:16.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:15 smithi186 ceph-mon[32797]: pgmap v393: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:16.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:15 smithi031 ceph-mon[29164]: pgmap v393: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:16.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:15 smithi097 ceph-mon[32419]: pgmap v393: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:16.493 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T17:56:16.756 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:17.131 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:16 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:56:17.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:16 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:56:17.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:16 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:56:18.214 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:18 smithi031 ceph-mon[29164]: pgmap v394: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:56:18.214 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:18 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:18.214 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:18 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:18.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:18 smithi097 ceph-mon[32419]: pgmap v394: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:56:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:18 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:18 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:18.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.441+0000 7f7740586700 1 -- 172.21.15.31:0/759472684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f773806bad0 msgr2=0x7f77381102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:18.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.441+0000 7f7740586700 1 --2- 172.21.15.31:0/759472684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f773806bad0 0x7f77381102e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f773000b0a0 tx=0x7f773000b3b0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.441+0000 7f7740586700 1 -- 172.21.15.31:0/759472684 shutdown_connections 2024-04-26T17:56:18.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.441+0000 7f7740586700 1 --2- 172.21.15.31:0/759472684 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77381094a0 0x7f7738109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.441+0000 7f7740586700 1 --2- 172.21.15.31:0/759472684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f773806bad0 0x7f77381102e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.441+0000 7f7740586700 1 --2- 172.21.15.31:0/759472684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f773806b130 0x7f773806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.441+0000 7f7740586700 1 -- 172.21.15.31:0/759472684 >> 172.21.15.31:0/759472684 conn(0x7f77380792d0 msgr2=0x7f77380796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.441+0000 7f7740586700 1 -- 172.21.15.31:0/759472684 shutdown_connections 2024-04-26T17:56:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.441+0000 7f7740586700 1 -- 172.21.15.31:0/759472684 wait complete. 2024-04-26T17:56:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.442+0000 7f7740586700 1 Processor -- start 2024-04-26T17:56:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.442+0000 7f7740586700 1 -- start start 2024-04-26T17:56:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.442+0000 7f7740586700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773806b130 0x7f77381023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.442+0000 7f7740586700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f773806bad0 0x7f7738102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.442+0000 7f7740586700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77381094a0 0x7f7738102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.442+0000 7f7740586700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7738113650 con 0x7f77381094a0 2024-04-26T17:56:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.442+0000 7f7740586700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f77381134d0 con 0x7f773806b130 2024-04-26T17:56:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.442+0000 7f7740586700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f77381137d0 con 0x7f773806bad0 2024-04-26T17:56:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.442+0000 7f773eb23700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77381094a0 0x7f7738102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.442+0000 7f773eb23700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77381094a0 0x7f7738102e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53450/0 (socket says 172.21.15.31:53450) 2024-04-26T17:56:18.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.442+0000 7f773eb23700 1 -- 172.21.15.31:0/3871718459 learned_addr learned my addr 172.21.15.31:0/3871718459 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.442+0000 7f773e322700 1 --2- 172.21.15.31:0/3871718459 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773806b130 0x7f77381023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.442+0000 7f773db21700 1 --2- 172.21.15.31:0/3871718459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f773806bad0 0x7f7738102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.443+0000 7f773eb23700 1 -- 172.21.15.31:0/3871718459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f773806bad0 msgr2=0x7f7738102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.443+0000 7f773eb23700 1 --2- 172.21.15.31:0/3871718459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f773806bad0 0x7f7738102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.443+0000 7f773eb23700 1 -- 172.21.15.31:0/3871718459 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773806b130 msgr2=0x7f77381023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.443+0000 7f773eb23700 1 --2- 172.21.15.31:0/3871718459 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773806b130 0x7f77381023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.443+0000 7f773eb23700 1 -- 172.21.15.31:0/3871718459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77381a8070 con 0x7f77381094a0 2024-04-26T17:56:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.443+0000 7f773eb23700 1 --2- 172.21.15.31:0/3871718459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77381094a0 0x7f7738102e70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f773000b830 tx=0x7f7730004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.443+0000 7f772f7fe700 1 -- 172.21.15.31:0/3871718459 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7730021070 con 0x7f77381094a0 2024-04-26T17:56:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.443+0000 7f772f7fe700 1 -- 172.21.15.31:0/3871718459 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7730007e30 con 0x7f77381094a0 2024-04-26T17:56:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.443+0000 7f772f7fe700 1 -- 172.21.15.31:0/3871718459 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7730004700 con 0x7f77381094a0 2024-04-26T17:56:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.443+0000 7f7740586700 1 -- 172.21.15.31:0/3871718459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77381a8300 con 0x7f77381094a0 2024-04-26T17:56:18.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.443+0000 7f7740586700 1 -- 172.21.15.31:0/3871718459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77381a8760 con 0x7f77381094a0 2024-04-26T17:56:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.445+0000 7f772f7fe700 1 -- 172.21.15.31:0/3871718459 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7730007980 con 0x7f77381094a0 2024-04-26T17:56:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.446+0000 7f772f7fe700 1 --2- 172.21.15.31:0/3871718459 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f77240a3cc0 0x7f77240a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.446+0000 7f773e322700 1 --2- 172.21.15.31:0/3871718459 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f77240a3cc0 0x7f77240a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.446+0000 7f772f7fe700 1 -- 172.21.15.31:0/3871718459 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f77300d1360 con 0x7f77381094a0 2024-04-26T17:56:18.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.447+0000 7f774ae18b80 1 Processor -- start 2024-04-26T17:56:18.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.447+0000 7f773e322700 1 --2- 172.21.15.31:0/3871718459 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f77240a3cc0 0x7f77240a6180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7734005950 tx=0x7f773400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:18.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.447+0000 7f774ae18b80 1 -- start start 2024-04-26T17:56:18.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.447+0000 7f774ae18b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e3dad0 0x558b28e3deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.447+0000 7f774ae18b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3e3f0 0x558b28e37cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.447+0000 7f774ae18b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e382d0 0x558b28e38710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.447+0000 7f774ae18b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x558b28d8a6f0 con 0x558b28e382d0 2024-04-26T17:56:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.447+0000 7f774ae18b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x558b28d29590 con 0x558b28e3e3f0 2024-04-26T17:56:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.447+0000 7f774ae18b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558b28d866b0 con 0x558b28e3dad0 2024-04-26T17:56:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.447+0000 7f773eb23700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e382d0 0x558b28e38710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.447+0000 7f773eb23700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e382d0 0x558b28e38710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53452/0 (socket says 172.21.15.31:53452) 2024-04-26T17:56:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.447+0000 7f773eb23700 1 -- 172.21.15.31:0/614747087 learned_addr learned my addr 172.21.15.31:0/614747087 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.447+0000 7f773e322700 1 --2- 172.21.15.31:0/614747087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e3dad0 0x558b28e3deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.448+0000 7f773db21700 1 --2- 172.21.15.31:0/614747087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3e3f0 0x558b28e37cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.448+0000 7f773eb23700 1 -- 172.21.15.31:0/614747087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e3dad0 msgr2=0x558b28e3deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.448+0000 7f773eb23700 1 --2- 172.21.15.31:0/614747087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e3dad0 0x558b28e3deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.448+0000 7f773eb23700 1 -- 172.21.15.31:0/614747087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3e3f0 msgr2=0x558b28e37cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:18.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.448+0000 7f773eb23700 1 --2- 172.21.15.31:0/614747087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3e3f0 0x558b28e37cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.448+0000 7f773eb23700 1 -- 172.21.15.31:0/614747087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b28cd7880 con 0x558b28e382d0 2024-04-26T17:56:18.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.448+0000 7f773e322700 1 --2- 172.21.15.31:0/614747087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e3dad0 0x558b28e3deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:56:18.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.448+0000 7f773eb23700 1 --2- 172.21.15.31:0/614747087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e382d0 0x558b28e38710 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f773009d630 tx=0x7f773000b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d5d6481478235a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:18.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.448+0000 7f772cff9700 1 -- 172.21.15.31:0/614747087 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773002a070 con 0x558b28e382d0 2024-04-26T17:56:18.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.448+0000 7f772cff9700 1 -- 172.21.15.31:0/614747087 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7730007c70 con 0x558b28e382d0 2024-04-26T17:56:18.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.448+0000 7f772cff9700 1 -- 172.21.15.31:0/614747087 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773009b9a0 con 0x558b28e382d0 2024-04-26T17:56:18.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.449+0000 7f774ae18b80 1 -- 172.21.15.31:0/614747087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e382d0 msgr2=0x558b28e38710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:18.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.449+0000 7f774ae18b80 1 --2- 172.21.15.31:0/614747087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e382d0 0x558b28e38710 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f773009d630 tx=0x7f773000b3b0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.449+0000 7f774ae18b80 1 -- 172.21.15.31:0/614747087 shutdown_connections 2024-04-26T17:56:18.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.449+0000 7f774ae18b80 1 --2- 172.21.15.31:0/614747087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e382d0 0x558b28e38710 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.449+0000 7f774ae18b80 1 --2- 172.21.15.31:0/614747087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e3dad0 0x558b28e3deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.449+0000 7f774ae18b80 1 --2- 172.21.15.31:0/614747087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3e3f0 0x558b28e37cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.449+0000 7f774ae18b80 1 -- 172.21.15.31:0/614747087 >> 172.21.15.31:0/614747087 conn(0x558b28d64cf0 msgr2=0x558b28e331b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:18.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.449+0000 7f774ae18b80 1 -- 172.21.15.31:0/614747087 shutdown_connections 2024-04-26T17:56:18.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.449+0000 7f774ae18b80 1 -- 172.21.15.31:0/614747087 wait complete. 2024-04-26T17:56:18.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f774ae18b80 1 -- start start 2024-04-26T17:56:18.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f774ae18b80 1 Processor -- start 2024-04-26T17:56:18.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f774ae18b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e382d0 0x558b28e441c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f774ae18b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e414c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f774ae18b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e3e3f0 0x558b28e41a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f774ae18b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x558b28e44cd0 con 0x558b28e3e3f0 2024-04-26T17:56:18.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f774ae18b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x558b28cf0900 con 0x558b28e3dad0 2024-04-26T17:56:18.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f774ae18b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558b28e44e50 con 0x558b28e382d0 2024-04-26T17:56:18.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f774ae18b80 10 client.0 _ll_register_callbacks cb 0x558b28d1e530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:56:18.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f773db21700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f773eb23700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e3e3f0 0x558b28e41a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f773db21700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e414c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:56322/0 (socket says 172.21.15.31:56322) 2024-04-26T17:56:18.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f773db21700 1 -- 172.21.15.31:0/445519513 learned_addr learned my addr 172.21.15.31:0/445519513 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:18.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f773e322700 1 --2- 172.21.15.31:0/445519513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e382d0 0x558b28e441c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f773db21700 1 -- 172.21.15.31:0/445519513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e382d0 msgr2=0x558b28e441c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f773db21700 1 --2- 172.21.15.31:0/445519513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e382d0 0x558b28e441c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f773db21700 1 -- 172.21.15.31:0/445519513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e3e3f0 msgr2=0x558b28e41a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f773db21700 1 --2- 172.21.15.31:0/445519513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e3e3f0 0x558b28e41a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.450+0000 7f773db21700 1 -- 172.21.15.31:0/445519513 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b28e420d0 con 0x558b28e3dad0 2024-04-26T17:56:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.451+0000 7f773db21700 1 --2- 172.21.15.31:0/445519513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e414c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f77280115e0 tx=0x7f77280118f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2391fa09fb4f97c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.451+0000 7f7721ffb700 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T17:56:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.451+0000 7f7721ffb700 1 -- 172.21.15.31:0/445519513 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728014650 con 0x558b28e3dad0 2024-04-26T17:56:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.451+0000 7f7721ffb700 1 -- 172.21.15.31:0/445519513 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f77280147f0 con 0x558b28e3dad0 2024-04-26T17:56:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.451+0000 7f774ae18b80 1 -- 172.21.15.31:0/445519513 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558b28d980d0 con 0x558b28e3dad0 2024-04-26T17:56:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.451+0000 7f7721ffb700 1 -- 172.21.15.31:0/445519513 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7728024070 con 0x558b28e3dad0 2024-04-26T17:56:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.451+0000 7f774ae18b80 10 client.24836 fetch_fsmap learned FSMap version 25 2024-04-26T17:56:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.451+0000 7f774ae18b80 1 -- 172.21.15.31:0/445519513 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558b28e4f720 con 0x558b28e3dad0 2024-04-26T17:56:18.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.451+0000 7f7721ffb700 1 -- 172.21.15.31:0/445519513 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558b28d980d0 con 0x558b28e3dad0 2024-04-26T17:56:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.452+0000 7f7721ffb700 1 -- 172.21.15.31:0/445519513 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f772801daf0 con 0x558b28e3dad0 2024-04-26T17:56:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.452+0000 7f774ae18b80 10 client.24836 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T17:56:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.452+0000 7f774ae18b80 10 client.24836 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T17:56:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.452+0000 7f774ae18b80 20 client.24836 populate_metadata read hostname 'smithi031' 2024-04-26T17:56:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.452+0000 7f774ae18b80 1 --2- 172.21.15.31:0/445519513 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x558b28e51d30 0x558b28e531a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.452+0000 7f774ae18b80 4 client.24836 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:56:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.452+0000 7f774ae18b80 1 -- 172.21.15.31:0/445519513 --> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b28d92170 con 0x558b28e51d30 2024-04-26T17:56:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.452+0000 7f773e322700 1 --2- 172.21.15.31:0/445519513 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x558b28e51d30 0x558b28e531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.452+0000 7f773e322700 1 --2- 172.21.15.31:0/445519513 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x558b28e51d30 0x558b28e531a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=96bede8a3f0f99b5 server_cookie=719670a6d123c208 in_seq=0 out_seq=0 2024-04-26T17:56:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.452+0000 7f7721ffb700 10 client.24836 ms_handle_connect on v2:172.21.15.186:6838/1587952500 2024-04-26T17:56:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.453+0000 7f7721ffb700 1 -- 172.21.15.31:0/445519513 <== mds.1 v2:172.21.15.186:6838/1587952500 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b28d92170 con 0x558b28e51d30 2024-04-26T17:56:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.453+0000 7f7721ffb700 10 client.24836 handle_command_reply: tid=0 2024-04-26T17:56:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.453+0000 7f774ae18b80 1 client.24836 shutdown 2024-04-26T17:56:18.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.453+0000 7f774ae18b80 1 -- 172.21.15.31:0/445519513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 msgr2=0x558b28e414c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:18.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.453+0000 7f774ae18b80 1 --2- 172.21.15.31:0/445519513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e414c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f77280115e0 tx=0x7f77280118f0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.453+0000 7f774ae18b80 1 -- 172.21.15.31:0/445519513 shutdown_connections 2024-04-26T17:56:18.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.453+0000 7f774ae18b80 1 --2- 172.21.15.31:0/445519513 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x558b28e51d30 0x558b28e531a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.453+0000 7f774ae18b80 1 --2- 172.21.15.31:0/445519513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e3e3f0 0x558b28e41a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.453+0000 7f774ae18b80 1 --2- 172.21.15.31:0/445519513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e382d0 0x558b28e441c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.453+0000 7f774ae18b80 1 --2- 172.21.15.31:0/445519513 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e414c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.453+0000 7f774ae18b80 1 -- 172.21.15.31:0/445519513 >> 172.21.15.31:0/445519513 conn(0x558b28d64cf0 msgr2=0x558b28e32da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:18.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.453+0000 7f7721ffb700 0 client.24836 ms_handle_reset on v2:172.21.15.186:6838/1587952500 2024-04-26T17:56:18.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.454+0000 7f774ae18b80 1 -- 172.21.15.31:0/445519513 shutdown_connections 2024-04-26T17:56:18.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.454+0000 7f774ae18b80 1 -- 172.21.15.31:0/445519513 wait complete. 2024-04-26T17:56:18.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.454+0000 7f774ae18b80 20 client.24836 trim_cache size 0 max 16384 2024-04-26T17:56:18.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.475+0000 7f774ae18b80 1 Processor -- start 2024-04-26T17:56:18.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.475+0000 7f774ae18b80 1 -- start start 2024-04-26T17:56:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.475+0000 7f774ae18b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28d64cf0 0x558b28e37cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.475+0000 7f774ae18b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e382d0 0x558b28e41a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.475+0000 7f774ae18b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e5c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.475+0000 7f774ae18b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x558b28e514e0 con 0x558b28e382d0 2024-04-26T17:56:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.475+0000 7f774ae18b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x558b28e51bb0 con 0x558b28e3dad0 2024-04-26T17:56:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.475+0000 7f774ae18b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558b28e52280 con 0x558b28d64cf0 2024-04-26T17:56:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.475+0000 7f773db21700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e382d0 0x558b28e41a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.475+0000 7f773db21700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e382d0 0x558b28e41a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53476/0 (socket says 172.21.15.31:53476) 2024-04-26T17:56:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.475+0000 7f773db21700 1 -- 172.21.15.31:0/2337945139 learned_addr learned my addr 172.21.15.31:0/2337945139 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.475+0000 7f773e322700 1 --2- 172.21.15.31:0/2337945139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28d64cf0 0x558b28e37cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.475+0000 7f773eb23700 1 --2- 172.21.15.31:0/2337945139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e5c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.476+0000 7f773db21700 1 -- 172.21.15.31:0/2337945139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28d64cf0 msgr2=0x558b28e37cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.476+0000 7f773db21700 1 --2- 172.21.15.31:0/2337945139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28d64cf0 0x558b28e37cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.476+0000 7f773db21700 1 -- 172.21.15.31:0/2337945139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 msgr2=0x558b28e5c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.476+0000 7f773db21700 1 --2- 172.21.15.31:0/2337945139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e5c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.476+0000 7f773db21700 1 -- 172.21.15.31:0/2337945139 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b28e3d5a0 con 0x558b28e382d0 2024-04-26T17:56:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.476+0000 7f773eb23700 1 --2- 172.21.15.31:0/2337945139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e5c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:56:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.476+0000 7f773db21700 1 --2- 172.21.15.31:0/2337945139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e382d0 0x558b28e41a50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f77280115b0 tx=0x7f7728011fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c43010844e5babe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.476+0000 7f7723fff700 1 -- 172.21.15.31:0/2337945139 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728014850 con 0x558b28e382d0 2024-04-26T17:56:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.476+0000 7f7723fff700 1 -- 172.21.15.31:0/2337945139 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f772801d400 con 0x558b28e382d0 2024-04-26T17:56:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.476+0000 7f7723fff700 1 -- 172.21.15.31:0/2337945139 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280287d0 con 0x558b28e382d0 2024-04-26T17:56:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.477+0000 7f774ae18b80 1 -- 172.21.15.31:0/2337945139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e382d0 msgr2=0x558b28e41a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.477+0000 7f774ae18b80 1 --2- 172.21.15.31:0/2337945139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e382d0 0x558b28e41a50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f77280115b0 tx=0x7f7728011fd0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.477+0000 7f774ae18b80 1 -- 172.21.15.31:0/2337945139 shutdown_connections 2024-04-26T17:56:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.477+0000 7f774ae18b80 1 --2- 172.21.15.31:0/2337945139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e5c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.477+0000 7f774ae18b80 1 --2- 172.21.15.31:0/2337945139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28d64cf0 0x558b28e37cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.477+0000 7f774ae18b80 1 --2- 172.21.15.31:0/2337945139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28e382d0 0x558b28e41a50 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.477+0000 7f774ae18b80 1 -- 172.21.15.31:0/2337945139 >> 172.21.15.31:0/2337945139 conn(0x558b28e2eee0 msgr2=0x558b28e3b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.477+0000 7f774ae18b80 1 -- 172.21.15.31:0/2337945139 shutdown_connections 2024-04-26T17:56:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.477+0000 7f774ae18b80 1 -- 172.21.15.31:0/2337945139 wait complete. 2024-04-26T17:56:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.478+0000 7f774ae18b80 1 -- start start 2024-04-26T17:56:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.478+0000 7f774ae18b80 1 Processor -- start 2024-04-26T17:56:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.478+0000 7f774ae18b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28d64cf0 0x558b28e435a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.478+0000 7f774ae18b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e382d0 0x558b28e43ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.478+0000 7f774ae18b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e5aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.478+0000 7f774ae18b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x558b28e53020 con 0x558b28d64cf0 2024-04-26T17:56:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.478+0000 7f774ae18b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x558b28e52950 con 0x558b28e3dad0 2024-04-26T17:56:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.478+0000 7f774ae18b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558b28e536f0 con 0x558b28e382d0 2024-04-26T17:56:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.478+0000 7f773e322700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28d64cf0 0x558b28e435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.478+0000 7f773e322700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28d64cf0 0x558b28e435a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53490/0 (socket says 172.21.15.31:53490) 2024-04-26T17:56:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.478+0000 7f773e322700 1 -- 172.21.15.31:0/191360505 learned_addr learned my addr 172.21.15.31:0/191360505 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.478+0000 7f774ae18b80 10 client.0 _ll_register_callbacks cb 0x558b28d1e530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:56:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.478+0000 7f773db21700 1 --2- 172.21.15.31:0/191360505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e382d0 0x558b28e43ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.478+0000 7f773eb23700 1 --2- 172.21.15.31:0/191360505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e5aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.479+0000 7f773e322700 1 -- 172.21.15.31:0/191360505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e382d0 msgr2=0x558b28e43ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.479+0000 7f773e322700 1 --2- 172.21.15.31:0/191360505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e382d0 0x558b28e43ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.479+0000 7f773e322700 1 -- 172.21.15.31:0/191360505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 msgr2=0x558b28e5aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.479+0000 7f773e322700 1 --2- 172.21.15.31:0/191360505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e5aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.479+0000 7f773e322700 1 -- 172.21.15.31:0/191360505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b28e443c0 con 0x558b28d64cf0 2024-04-26T17:56:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.479+0000 7f773e322700 1 --2- 172.21.15.31:0/191360505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28d64cf0 0x558b28e435a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7734005d90 tx=0x7f77340047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee15e7a807ec2b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.479+0000 7f7721ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:56:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.479+0000 7f7721ffb700 1 -- 172.21.15.31:0/191360505 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773402a610 con 0x558b28d64cf0 2024-04-26T17:56:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.479+0000 7f7721ffb700 1 -- 172.21.15.31:0/191360505 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f773402a7b0 con 0x558b28d64cf0 2024-04-26T17:56:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.479+0000 7f774ae18b80 1 -- 172.21.15.31:0/191360505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f772801d400 con 0x558b28d64cf0 2024-04-26T17:56:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.479+0000 7f7721ffb700 1 -- 172.21.15.31:0/191360505 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7734004830 con 0x558b28d64cf0 2024-04-26T17:56:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.479+0000 7f774ae18b80 10 client.15093 fetch_fsmap learned FSMap version 25 2024-04-26T17:56:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.479+0000 7f774ae18b80 1 -- 172.21.15.31:0/191360505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558b28eb94e0 con 0x558b28d64cf0 2024-04-26T17:56:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.479+0000 7f7721ffb700 1 -- 172.21.15.31:0/191360505 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f773402e030 con 0x558b28d64cf0 2024-04-26T17:56:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.480+0000 7f7721ffb700 1 -- 172.21.15.31:0/191360505 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f7734033460 con 0x558b28d64cf0 2024-04-26T17:56:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.480+0000 7f774ae18b80 10 client.15093 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T17:56:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.480+0000 7f774ae18b80 10 client.15093 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T17:56:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.480+0000 7f774ae18b80 20 client.15093 populate_metadata read hostname 'smithi031' 2024-04-26T17:56:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.480+0000 7f774ae18b80 1 --2- 172.21.15.31:0/191360505 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x558b28e3e3f0 0x558b28ebce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.480+0000 7f774ae18b80 4 client.15093 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T17:56:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.480+0000 7f774ae18b80 1 -- 172.21.15.31:0/191360505 --> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] -- command(tid 0: {"prefix": "status"}) -- 0x7f772801d400 con 0x558b28e3e3f0 2024-04-26T17:56:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.480+0000 7f773db21700 1 --2- 172.21.15.31:0/191360505 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x558b28e3e3f0 0x558b28ebce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f773db21700 1 --2- 172.21.15.31:0/191360505 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x558b28e3e3f0 0x558b28ebce90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fdcc9a3390110581 server_cookie=4b35b4a2b8f6b237 in_seq=0 out_seq=0 2024-04-26T17:56:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f7721ffb700 10 client.15093 ms_handle_connect on v2:172.21.15.186:6838/1587952500 2024-04-26T17:56:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f7721ffb700 1 -- 172.21.15.31:0/191360505 <== mds.1 v2:172.21.15.186:6838/1587952500 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x7f772801d400 con 0x558b28e3e3f0 2024-04-26T17:56:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f7721ffb700 10 client.15093 handle_command_reply: tid=0 2024-04-26T17:56:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f774ae18b80 1 client.15093 shutdown 2024-04-26T17:56:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f774ae18b80 1 -- 172.21.15.31:0/191360505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28d64cf0 msgr2=0x558b28e435a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f774ae18b80 1 --2- 172.21.15.31:0/191360505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28d64cf0 0x558b28e435a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7734005d90 tx=0x7f77340047c0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f774ae18b80 1 -- 172.21.15.31:0/191360505 shutdown_connections 2024-04-26T17:56:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f774ae18b80 1 --2- 172.21.15.31:0/191360505 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x558b28e3e3f0 0x558b28ebce90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f774ae18b80 1 --2- 172.21.15.31:0/191360505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558b28e3dad0 0x558b28e5aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f774ae18b80 1 --2- 172.21.15.31:0/191360505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b28d64cf0 0x558b28e435a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f7721ffb700 0 client.15093 ms_handle_reset on v2:172.21.15.186:6838/1587952500 2024-04-26T17:56:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f774ae18b80 1 --2- 172.21.15.31:0/191360505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558b28e382d0 0x558b28e43ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f774ae18b80 1 -- 172.21.15.31:0/191360505 >> 172.21.15.31:0/191360505 conn(0x558b28e2eee0 msgr2=0x558b28e3e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f774ae18b80 1 -- 172.21.15.31:0/191360505 shutdown_connections 2024-04-26T17:56:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f774ae18b80 1 -- 172.21.15.31:0/191360505 wait complete. 2024-04-26T17:56:18.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.481+0000 7f774ae18b80 20 client.15093 trim_cache size 0 max 16384 2024-04-26T17:56:18.489 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T17:56:18.489 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "7e53e050-03f4-11ef-bc93-c7b262605968", 2024-04-26T17:56:18.489 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 1, 2024-04-26T17:56:18.489 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24457, 2024-04-26T17:56:18.489 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-26T17:56:18.489 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-26T17:56:18.489 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-26T17:56:18.489 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 179.81227206899999, 2024-04-26T17:56:18.489 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 25, 2024-04-26T17:56:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 90, 2024-04-26T17:56:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T17:56:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 218.57589263099999 2024-04-26T17:56:18.490 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T17:56:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.482+0000 7f7740586700 1 -- 172.21.15.31:0/3871718459 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f77240a3cc0 msgr2=0x7f77240a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.482+0000 7f7740586700 1 --2- 172.21.15.31:0/3871718459 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f77240a3cc0 0x7f77240a6180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7734005950 tx=0x7f773400b410 comp rx=0 tx=0).stop 2024-04-26T17:56:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.482+0000 7f7740586700 1 -- 172.21.15.31:0/3871718459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77381094a0 msgr2=0x7f7738102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.482+0000 7f7740586700 1 --2- 172.21.15.31:0/3871718459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77381094a0 0x7f7738102e70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f773000b830 tx=0x7f7730004ce0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.482+0000 7f7740586700 1 -- 172.21.15.31:0/3871718459 shutdown_connections 2024-04-26T17:56:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.482+0000 7f7740586700 1 --2- 172.21.15.31:0/3871718459 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f77240a3cc0 0x7f77240a6180 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.482+0000 7f7740586700 1 --2- 172.21.15.31:0/3871718459 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f773806b130 0x7f77381023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.482+0000 7f7740586700 1 --2- 172.21.15.31:0/3871718459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77381094a0 0x7f7738102e70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.482+0000 7f7740586700 1 --2- 172.21.15.31:0/3871718459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f773806bad0 0x7f7738102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.482+0000 7f7740586700 1 -- 172.21.15.31:0/3871718459 >> 172.21.15.31:0/3871718459 conn(0x7f77380792d0 msgr2=0x7f7738100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.482+0000 7f7740586700 1 -- 172.21.15.31:0/3871718459 shutdown_connections 2024-04-26T17:56:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:18.482+0000 7f7740586700 1 -- 172.21.15.31:0/3871718459 wait complete. 2024-04-26T17:56:18.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:18 smithi186 ceph-mon[32797]: pgmap v394: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T17:56:18.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:18 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:18.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:18 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:19.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T17:56:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:19 smithi186 ceph-mon[32797]: pgmap v395: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:56:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:56:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:19 smithi186 ceph-mon[32797]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:19.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:19.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:19.541 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:19 smithi031 ceph-mon[29164]: pgmap v395: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:19.541 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:56:19.541 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:56:19.541 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:19 smithi031 ceph-mon[29164]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:19 smithi097 ceph-mon[32419]: pgmap v395: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T17:56:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:56:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:56:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:19 smithi097 ceph-mon[32419]: from='mgr.14150 172.21.15.31:0/2173297372' entity='mgr.x' 2024-04-26T17:56:21.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.083+0000 7fda59106700 1 -- 172.21.15.31:0/3899149187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda54101de0 msgr2=0x7fda54102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:21.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.083+0000 7fda59106700 1 --2- 172.21.15.31:0/3899149187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda54101de0 0x7fda54102260 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fda48009990 tx=0x7fda48009ca0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.083+0000 7fda59106700 1 -- 172.21.15.31:0/3899149187 shutdown_connections 2024-04-26T17:56:21.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.084+0000 7fda59106700 1 --2- 172.21.15.31:0/3899149187 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda541027a0 0x7fda54110210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.084+0000 7fda59106700 1 --2- 172.21.15.31:0/3899149187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda54108500 0x7fda541088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.084+0000 7fda59106700 1 --2- 172.21.15.31:0/3899149187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda54101de0 0x7fda54102260 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.084+0000 7fda59106700 1 -- 172.21.15.31:0/3899149187 >> 172.21.15.31:0/3899149187 conn(0x7fda54077640 msgr2=0x7fda54077a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:21.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.084+0000 7fda59106700 1 -- 172.21.15.31:0/3899149187 shutdown_connections 2024-04-26T17:56:21.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.084+0000 7fda59106700 1 -- 172.21.15.31:0/3899149187 wait complete. 2024-04-26T17:56:21.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.084+0000 7fda59106700 1 Processor -- start 2024-04-26T17:56:21.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda59106700 1 -- start start 2024-04-26T17:56:21.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda59106700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda54101de0 0x7fda5419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda59106700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda541027a0 0x7fda5419fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda59106700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda54108500 0x7fda54199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda59106700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fda54113610 con 0x7fda54108500 2024-04-26T17:56:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda59106700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fda54113490 con 0x7fda54101de0 2024-04-26T17:56:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda59106700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda54113790 con 0x7fda541027a0 2024-04-26T17:56:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda5359e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda54108500 0x7fda54199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda5359e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda54108500 0x7fda54199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53516/0 (socket says 172.21.15.31:53516) 2024-04-26T17:56:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda5359e700 1 -- 172.21.15.31:0/3521769486 learned_addr learned my addr 172.21.15.31:0/3521769486 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda52d9d700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda54101de0 0x7fda5419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda5259c700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda541027a0 0x7fda5419fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda5359e700 1 -- 172.21.15.31:0/3521769486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda541027a0 msgr2=0x7fda5419fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda5359e700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda541027a0 0x7fda5419fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda5359e700 1 -- 172.21.15.31:0/3521769486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda54101de0 msgr2=0x7fda5419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda5359e700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda54101de0 0x7fda5419f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda5359e700 1 -- 172.21.15.31:0/3521769486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5419a0d0 con 0x7fda54108500 2024-04-26T17:56:21.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda5259c700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda541027a0 0x7fda5419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:56:21.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.085+0000 7fda5359e700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda54108500 0x7fda54199970 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fda4c007ee0 tx=0x7fda4c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:21.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.086+0000 7fda3bfff700 1 -- 172.21.15.31:0/3521769486 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c00ddd0 con 0x7fda54108500 2024-04-26T17:56:21.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.086+0000 7fda3bfff700 1 -- 172.21.15.31:0/3521769486 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda4c004020 con 0x7fda54108500 2024-04-26T17:56:21.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.086+0000 7fda59106700 1 -- 172.21.15.31:0/3521769486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda5419a3c0 con 0x7fda54108500 2024-04-26T17:56:21.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.086+0000 7fda3bfff700 1 -- 172.21.15.31:0/3521769486 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c015660 con 0x7fda54108500 2024-04-26T17:56:21.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.086+0000 7fda59106700 1 -- 172.21.15.31:0/3521769486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda541a6680 con 0x7fda54108500 2024-04-26T17:56:21.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.089+0000 7fda3bfff700 1 -- 172.21.15.31:0/3521769486 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fda4c0044f0 con 0x7fda54108500 2024-04-26T17:56:21.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.089+0000 7fda3bfff700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fda3c0a3c40 0x7fda3c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.089+0000 7fda52d9d700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fda3c0a3c40 0x7fda3c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.089+0000 7fda3bfff700 1 -- 172.21.15.31:0/3521769486 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fda4c0cfe90 con 0x7fda54108500 2024-04-26T17:56:21.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.090+0000 7fda52d9d700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fda3c0a3c40 0x7fda3c0a6100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fda44005fd0 tx=0x7fda44005db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:21.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.090+0000 7fda63998b80 1 Processor -- start 2024-04-26T17:56:21.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.090+0000 7fda63998b80 1 -- start start 2024-04-26T17:56:21.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.090+0000 7fda63998b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c198ad0 0x55ef4c198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.090+0000 7fda63998b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1993f0 0x55ef4c192cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.090+0000 7fda63998b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1932d0 0x55ef4c193710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.090+0000 7fda63998b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c0e56f0 con 0x55ef4c1993f0 2024-04-26T17:56:21.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.090+0000 7fda63998b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c084590 con 0x55ef4c1932d0 2024-04-26T17:56:21.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.090+0000 7fda63998b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c0e16b0 con 0x55ef4c198ad0 2024-04-26T17:56:21.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.090+0000 7fda5259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1993f0 0x55ef4c192cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.090+0000 7fda5259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1993f0 0x55ef4c192cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53518/0 (socket says 172.21.15.31:53518) 2024-04-26T17:56:21.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.090+0000 7fda5259c700 1 -- 172.21.15.31:0/613310988 learned_addr learned my addr 172.21.15.31:0/613310988 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:21.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.091+0000 7fda5259c700 1 -- 172.21.15.31:0/613310988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c198ad0 msgr2=0x55ef4c198eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:21.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.091+0000 7fda5259c700 1 --2- 172.21.15.31:0/613310988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c198ad0 0x55ef4c198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.091+0000 7fda5259c700 1 -- 172.21.15.31:0/613310988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1932d0 msgr2=0x55ef4c193710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:21.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.091+0000 7fda5259c700 1 --2- 172.21.15.31:0/613310988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1932d0 0x55ef4c193710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.091+0000 7fda5259c700 1 -- 172.21.15.31:0/613310988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef4c032880 con 0x55ef4c1993f0 2024-04-26T17:56:21.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.091+0000 7fda5259c700 1 --2- 172.21.15.31:0/613310988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1993f0 0x55ef4c192cb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fda48005f30 tx=0x7fda48005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1880e8c9100104 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:21.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.091+0000 7fda397fa700 1 -- 172.21.15.31:0/613310988 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48026070 con 0x55ef4c1993f0 2024-04-26T17:56:21.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.091+0000 7fda397fa700 1 -- 172.21.15.31:0/613310988 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda480050d0 con 0x55ef4c1993f0 2024-04-26T17:56:21.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.091+0000 7fda397fa700 1 -- 172.21.15.31:0/613310988 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4800bb70 con 0x55ef4c1993f0 2024-04-26T17:56:21.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 -- 172.21.15.31:0/613310988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1993f0 msgr2=0x55ef4c192cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:21.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 --2- 172.21.15.31:0/613310988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1993f0 0x55ef4c192cb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fda48005f30 tx=0x7fda48005c00 comp rx=0 tx=0).stop 2024-04-26T17:56:21.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 -- 172.21.15.31:0/613310988 shutdown_connections 2024-04-26T17:56:21.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 --2- 172.21.15.31:0/613310988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1932d0 0x55ef4c193710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 --2- 172.21.15.31:0/613310988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c198ad0 0x55ef4c198eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 --2- 172.21.15.31:0/613310988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1993f0 0x55ef4c192cb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 -- 172.21.15.31:0/613310988 >> 172.21.15.31:0/613310988 conn(0x55ef4c0bfcf0 msgr2=0x55ef4c18e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:21.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 -- 172.21.15.31:0/613310988 shutdown_connections 2024-04-26T17:56:21.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 -- 172.21.15.31:0/613310988 wait complete. 2024-04-26T17:56:21.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 -- start start 2024-04-26T17:56:21.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 Processor -- start 2024-04-26T17:56:21.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1932d0 0x55ef4c19f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c198ad0 0x55ef4c19c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c1993f0 0x55ef4c19ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c19feb0 con 0x55ef4c198ad0 2024-04-26T17:56:21.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c04b900 con 0x55ef4c1932d0 2024-04-26T17:56:21.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.092+0000 7fda63998b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c1a0030 con 0x55ef4c1993f0 2024-04-26T17:56:21.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda63998b80 10 client.0 _ll_register_callbacks cb 0x55ef4c079530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:56:21.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda5259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c198ad0 0x55ef4c19c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda5259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c198ad0 0x55ef4c19c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53530/0 (socket says 172.21.15.31:53530) 2024-04-26T17:56:21.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda5259c700 1 -- 172.21.15.31:0/1917802248 learned_addr learned my addr 172.21.15.31:0/1917802248 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:21.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda5359e700 1 --2- 172.21.15.31:0/1917802248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c1993f0 0x55ef4c19ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda52d9d700 1 --2- 172.21.15.31:0/1917802248 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1932d0 0x55ef4c19f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda5259c700 1 -- 172.21.15.31:0/1917802248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c1993f0 msgr2=0x55ef4c19ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:21.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda5259c700 1 --2- 172.21.15.31:0/1917802248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c1993f0 0x55ef4c19ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda5259c700 1 -- 172.21.15.31:0/1917802248 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1932d0 msgr2=0x55ef4c19f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:21.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda5259c700 1 --2- 172.21.15.31:0/1917802248 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1932d0 0x55ef4c19f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda5259c700 1 -- 172.21.15.31:0/1917802248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef4c19d110 con 0x55ef4c198ad0 2024-04-26T17:56:21.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda5359e700 1 --2- 172.21.15.31:0/1917802248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c1993f0 0x55ef4c19ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:56:21.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda5259c700 1 --2- 172.21.15.31:0/1917802248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c198ad0 0x55ef4c19c4d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fda48024040 tx=0x7fda48005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=123566d4970ecb41 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:21.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda36ff5700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:56:21.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda36ff5700 1 -- 172.21.15.31:0/1917802248 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48031060 con 0x55ef4c198ad0 2024-04-26T17:56:21.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda36ff5700 1 -- 172.21.15.31:0/1917802248 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fda4800b7d0 con 0x55ef4c198ad0 2024-04-26T17:56:21.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda63998b80 1 -- 172.21.15.31:0/1917802248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda480050d0 con 0x55ef4c198ad0 2024-04-26T17:56:21.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda36ff5700 1 -- 172.21.15.31:0/1917802248 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fda48026040 con 0x55ef4c198ad0 2024-04-26T17:56:21.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.093+0000 7fda63998b80 10 client.15105 fetch_fsmap learned FSMap version 25 2024-04-26T17:56:21.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.094+0000 7fda63998b80 1 -- 172.21.15.31:0/1917802248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ef4c1aa990 con 0x55ef4c198ad0 2024-04-26T17:56:21.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.094+0000 7fda36ff5700 1 -- 172.21.15.31:0/1917802248 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda480050d0 con 0x55ef4c198ad0 2024-04-26T17:56:21.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.094+0000 7fda36ff5700 1 -- 172.21.15.31:0/1917802248 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fda4800fc00 con 0x55ef4c198ad0 2024-04-26T17:56:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.094+0000 7fda63998b80 10 client.15105 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T17:56:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.094+0000 7fda63998b80 10 client.15105 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T17:56:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.094+0000 7fda63998b80 20 client.15105 populate_metadata read hostname 'smithi031' 2024-04-26T17:56:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.094+0000 7fda63998b80 1 --2- 172.21.15.31:0/1917802248 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ef4c1acac0 0x55ef4c1adf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.094+0000 7fda63998b80 4 client.15105 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:56:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.094+0000 7fda63998b80 1 -- 172.21.15.31:0/1917802248 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef4c0f30d0 con 0x55ef4c1acac0 2024-04-26T17:56:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.094+0000 7fda52d9d700 1 --2- 172.21.15.31:0/1917802248 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ef4c1acac0 0x55ef4c1adf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.095+0000 7fda52d9d700 1 --2- 172.21.15.31:0/1917802248 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ef4c1acac0 0x55ef4c1adf30 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6f90a98546be275 server_cookie=ba423e783d208494 in_seq=0 out_seq=0 2024-04-26T17:56:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.095+0000 7fda36ff5700 10 client.15105 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:56:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.095+0000 7fda36ff5700 1 -- 172.21.15.31:0/1917802248 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef4c0f30d0 con 0x55ef4c1acac0 2024-04-26T17:56:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.095+0000 7fda36ff5700 10 client.15105 handle_command_reply: tid=0 2024-04-26T17:56:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.096+0000 7fda63998b80 1 client.15105 shutdown 2024-04-26T17:56:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.096+0000 7fda63998b80 1 -- 172.21.15.31:0/1917802248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c198ad0 msgr2=0x55ef4c19c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:21.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.096+0000 7fda63998b80 1 --2- 172.21.15.31:0/1917802248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c198ad0 0x55ef4c19c4d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fda48024040 tx=0x7fda48005900 comp rx=0 tx=0).stop 2024-04-26T17:56:21.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.096+0000 7fda63998b80 1 -- 172.21.15.31:0/1917802248 shutdown_connections 2024-04-26T17:56:21.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.096+0000 7fda63998b80 1 --2- 172.21.15.31:0/1917802248 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ef4c1acac0 0x55ef4c1adf30 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.096+0000 7fda63998b80 1 --2- 172.21.15.31:0/1917802248 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1932d0 0x55ef4c19f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.096+0000 7fda63998b80 1 --2- 172.21.15.31:0/1917802248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c1993f0 0x55ef4c19ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.096+0000 7fda63998b80 1 --2- 172.21.15.31:0/1917802248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c198ad0 0x55ef4c19c4d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.096+0000 7fda63998b80 1 -- 172.21.15.31:0/1917802248 >> 172.21.15.31:0/1917802248 conn(0x55ef4c0bfcf0 msgr2=0x55ef4c18dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:21.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.096+0000 7fda36ff5700 0 client.15105 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:56:21.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.096+0000 7fda63998b80 1 -- 172.21.15.31:0/1917802248 shutdown_connections 2024-04-26T17:56:21.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.096+0000 7fda63998b80 1 -- 172.21.15.31:0/1917802248 wait complete. 2024-04-26T17:56:21.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.096+0000 7fda63998b80 20 client.15105 trim_cache size 0 max 16384 2024-04-26T17:56:21.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.116+0000 7fda63998b80 1 Processor -- start 2024-04-26T17:56:21.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.116+0000 7fda63998b80 1 -- start start 2024-04-26T17:56:21.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda63998b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1932d0 0x55ef4c1a9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda63998b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c198ad0 0x55ef4c192cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda63998b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1993f0 0x55ef4c1b7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda63998b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c1ac270 con 0x55ef4c1932d0 2024-04-26T17:56:21.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda63998b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c1ac940 con 0x55ef4c1993f0 2024-04-26T17:56:21.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda63998b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c1ad010 con 0x55ef4c198ad0 2024-04-26T17:56:21.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda52d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1932d0 0x55ef4c1a9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda52d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1932d0 0x55ef4c1a9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53532/0 (socket says 172.21.15.31:53532) 2024-04-26T17:56:21.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda52d9d700 1 -- 172.21.15.31:0/4140186366 learned_addr learned my addr 172.21.15.31:0/4140186366 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:21.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda5359e700 1 --2- 172.21.15.31:0/4140186366 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1993f0 0x55ef4c1b7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda5259c700 1 --2- 172.21.15.31:0/4140186366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c198ad0 0x55ef4c192cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda52d9d700 1 -- 172.21.15.31:0/4140186366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c198ad0 msgr2=0x55ef4c192cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:21.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda52d9d700 1 --2- 172.21.15.31:0/4140186366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c198ad0 0x55ef4c192cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda52d9d700 1 -- 172.21.15.31:0/4140186366 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1993f0 msgr2=0x55ef4c1b7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:21.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda52d9d700 1 --2- 172.21.15.31:0/4140186366 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1993f0 0x55ef4c1b7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda52d9d700 1 -- 172.21.15.31:0/4140186366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef4c18e6f0 con 0x55ef4c1932d0 2024-04-26T17:56:21.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda5359e700 1 --2- 172.21.15.31:0/4140186366 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1993f0 0x55ef4c1b7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:56:21.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.117+0000 7fda52d9d700 1 --2- 172.21.15.31:0/4140186366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1932d0 0x55ef4c1a9300 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fda54078bc0 tx=0x7fda4400af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb54d5fab0086af7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:21.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.118+0000 7fda38ff9700 1 -- 172.21.15.31:0/4140186366 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44025250 con 0x55ef4c1932d0 2024-04-26T17:56:21.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.118+0000 7fda38ff9700 1 -- 172.21.15.31:0/4140186366 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda4402b070 con 0x55ef4c1932d0 2024-04-26T17:56:21.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.118+0000 7fda38ff9700 1 -- 172.21.15.31:0/4140186366 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44010430 con 0x55ef4c1932d0 2024-04-26T17:56:21.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.118+0000 7fda63998b80 1 -- 172.21.15.31:0/4140186366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1932d0 msgr2=0x55ef4c1a9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:21.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.118+0000 7fda63998b80 1 --2- 172.21.15.31:0/4140186366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1932d0 0x55ef4c1a9300 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fda54078bc0 tx=0x7fda4400af90 comp rx=0 tx=0).stop 2024-04-26T17:56:21.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.118+0000 7fda63998b80 1 -- 172.21.15.31:0/4140186366 shutdown_connections 2024-04-26T17:56:21.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.118+0000 7fda63998b80 1 --2- 172.21.15.31:0/4140186366 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1993f0 0x55ef4c1b7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.118+0000 7fda63998b80 1 --2- 172.21.15.31:0/4140186366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1932d0 0x55ef4c1a9300 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.118+0000 7fda63998b80 1 --2- 172.21.15.31:0/4140186366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c198ad0 0x55ef4c192cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.118+0000 7fda63998b80 1 -- 172.21.15.31:0/4140186366 >> 172.21.15.31:0/4140186366 conn(0x55ef4c0bfcf0 msgr2=0x55ef4c196620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:21.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.118+0000 7fda63998b80 1 -- 172.21.15.31:0/4140186366 shutdown_connections 2024-04-26T17:56:21.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.118+0000 7fda63998b80 1 -- 172.21.15.31:0/4140186366 wait complete. 2024-04-26T17:56:21.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.119+0000 7fda63998b80 1 -- start start 2024-04-26T17:56:21.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.119+0000 7fda63998b80 1 Processor -- start 2024-04-26T17:56:21.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.119+0000 7fda63998b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1932d0 0x55ef4c1a86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.119+0000 7fda63998b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c198ad0 0x55ef4c1a8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.119+0000 7fda63998b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1993f0 0x55ef4c1a9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.119+0000 7fda63998b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c1addb0 con 0x55ef4c1993f0 2024-04-26T17:56:21.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.119+0000 7fda63998b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c1ad6e0 con 0x55ef4c1932d0 2024-04-26T17:56:21.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.119+0000 7fda63998b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c1ae480 con 0x55ef4c198ad0 2024-04-26T17:56:21.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.119+0000 7fda63998b80 10 client.0 _ll_register_callbacks cb 0x55ef4c079530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:56:21.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.119+0000 7fda5359e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1993f0 0x55ef4c1a9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.119+0000 7fda5359e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1993f0 0x55ef4c1a9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53536/0 (socket says 172.21.15.31:53536) 2024-04-26T17:56:21.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.119+0000 7fda5359e700 1 -- 172.21.15.31:0/3639628154 learned_addr learned my addr 172.21.15.31:0/3639628154 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:21.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.119+0000 7fda52d9d700 1 --2- 172.21.15.31:0/3639628154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1932d0 0x55ef4c1a86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.119+0000 7fda5259c700 1 --2- 172.21.15.31:0/3639628154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c198ad0 0x55ef4c1a8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.120+0000 7fda5359e700 1 -- 172.21.15.31:0/3639628154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c198ad0 msgr2=0x55ef4c1a8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.120+0000 7fda5359e700 1 --2- 172.21.15.31:0/3639628154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c198ad0 0x55ef4c1a8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.120+0000 7fda5359e700 1 -- 172.21.15.31:0/3639628154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1932d0 msgr2=0x55ef4c1a86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.120+0000 7fda5359e700 1 --2- 172.21.15.31:0/3639628154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1932d0 0x55ef4c1a86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.120+0000 7fda5359e700 1 -- 172.21.15.31:0/3639628154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef4c216e30 con 0x55ef4c1993f0 2024-04-26T17:56:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.120+0000 7fda5359e700 1 --2- 172.21.15.31:0/3639628154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1993f0 0x55ef4c1a9130 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fda4c009350 tx=0x7fda4c09c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06204abe4bcc3f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.120+0000 7fda36ff5700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:56:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.120+0000 7fda36ff5700 1 -- 172.21.15.31:0/3639628154 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c0a4e10 con 0x55ef4c1993f0 2024-04-26T17:56:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.120+0000 7fda36ff5700 1 -- 172.21.15.31:0/3639628154 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fda4c0a0170 con 0x55ef4c1993f0 2024-04-26T17:56:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.120+0000 7fda63998b80 1 -- 172.21.15.31:0/3639628154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda4402b070 con 0x55ef4c1993f0 2024-04-26T17:56:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.120+0000 7fda36ff5700 1 -- 172.21.15.31:0/3639628154 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fda4c004bd0 con 0x55ef4c1993f0 2024-04-26T17:56:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.120+0000 7fda63998b80 10 client.15111 fetch_fsmap learned FSMap version 25 2024-04-26T17:56:21.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.120+0000 7fda63998b80 1 -- 172.21.15.31:0/3639628154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ef4c218060 con 0x55ef4c1993f0 2024-04-26T17:56:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.120+0000 7fda36ff5700 1 -- 172.21.15.31:0/3639628154 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda4402b070 con 0x55ef4c1993f0 2024-04-26T17:56:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.121+0000 7fda36ff5700 1 -- 172.21.15.31:0/3639628154 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fda4c097ef0 con 0x55ef4c1993f0 2024-04-26T17:56:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.121+0000 7fda63998b80 10 client.15111 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T17:56:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.121+0000 7fda63998b80 10 client.15111 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T17:56:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.121+0000 7fda63998b80 20 client.15111 populate_metadata read hostname 'smithi031' 2024-04-26T17:56:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.121+0000 7fda63998b80 1 --2- 172.21.15.31:0/3639628154 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ef4c21cbe0 0x55ef4c21b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.121+0000 7fda63998b80 4 client.15111 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T17:56:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.121+0000 7fda63998b80 1 -- 172.21.15.31:0/3639628154 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "status"}) -- 0x55ef4c0ed170 con 0x55ef4c21cbe0 2024-04-26T17:56:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.121+0000 7fda52d9d700 1 --2- 172.21.15.31:0/3639628154 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ef4c21cbe0 0x55ef4c21b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.121+0000 7fda52d9d700 1 --2- 172.21.15.31:0/3639628154 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ef4c21cbe0 0x55ef4c21b8f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb0534eab99859f5 server_cookie=9b5ec44ae7d98998 in_seq=0 out_seq=0 2024-04-26T17:56:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.121+0000 7fda36ff5700 10 client.15111 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:56:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.121+0000 7fda36ff5700 1 -- 172.21.15.31:0/3639628154 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ef4c0ed170 con 0x55ef4c21cbe0 2024-04-26T17:56:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.121+0000 7fda36ff5700 10 client.15111 handle_command_reply: tid=0 2024-04-26T17:56:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda63998b80 1 client.15111 shutdown 2024-04-26T17:56:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda63998b80 1 -- 172.21.15.31:0/3639628154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1993f0 msgr2=0x55ef4c1a9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda63998b80 1 --2- 172.21.15.31:0/3639628154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1993f0 0x55ef4c1a9130 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fda4c009350 tx=0x7fda4c09c9e0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda63998b80 1 -- 172.21.15.31:0/3639628154 shutdown_connections 2024-04-26T17:56:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda63998b80 1 --2- 172.21.15.31:0/3639628154 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ef4c21cbe0 0x55ef4c21b8f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda36ff5700 0 client.15111 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:56:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda63998b80 1 --2- 172.21.15.31:0/3639628154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ef4c1932d0 0x55ef4c1a86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda63998b80 1 --2- 172.21.15.31:0/3639628154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef4c1993f0 0x55ef4c1a9130 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda63998b80 1 --2- 172.21.15.31:0/3639628154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef4c198ad0 0x55ef4c1a8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda63998b80 1 -- 172.21.15.31:0/3639628154 >> 172.21.15.31:0/3639628154 conn(0x55ef4c0bfcf0 msgr2=0x55ef4c1997d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda63998b80 1 -- 172.21.15.31:0/3639628154 shutdown_connections 2024-04-26T17:56:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda63998b80 1 -- 172.21.15.31:0/3639628154 wait complete. 2024-04-26T17:56:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda63998b80 20 client.15111 trim_cache size 0 max 16384 2024-04-26T17:56:21.128 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T17:56:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "7e53e050-03f4-11ef-bc93-c7b262605968", 2024-04-26T17:56:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-04-26T17:56:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24473, 2024-04-26T17:56:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-26T17:56:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-26T17:56:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-26T17:56:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 195.50958606699999, 2024-04-26T17:56:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 25, 2024-04-26T17:56:21.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 90, 2024-04-26T17:56:21.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 88, 2024-04-26T17:56:21.129 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 210.37329153100001 2024-04-26T17:56:21.129 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T17:56:21.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda59106700 1 -- 172.21.15.31:0/3521769486 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fda3c0a3c40 msgr2=0x7fda3c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:21.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda59106700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fda3c0a3c40 0x7fda3c0a6100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fda44005fd0 tx=0x7fda44005db0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda59106700 1 -- 172.21.15.31:0/3521769486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda54108500 msgr2=0x7fda54199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:21.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.122+0000 7fda59106700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda54108500 0x7fda54199970 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fda4c007ee0 tx=0x7fda4c00d3b0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.123+0000 7fda59106700 1 -- 172.21.15.31:0/3521769486 shutdown_connections 2024-04-26T17:56:21.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.123+0000 7fda59106700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fda3c0a3c40 0x7fda3c0a6100 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.123+0000 7fda59106700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda54101de0 0x7fda5419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.123+0000 7fda59106700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda54108500 0x7fda54199970 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.123+0000 7fda59106700 1 --2- 172.21.15.31:0/3521769486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda541027a0 0x7fda5419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.123+0000 7fda59106700 1 -- 172.21.15.31:0/3521769486 >> 172.21.15.31:0/3521769486 conn(0x7fda54077640 msgr2=0x7fda54100b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:21.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.123+0000 7fda59106700 1 -- 172.21.15.31:0/3521769486 shutdown_connections 2024-04-26T17:56:21.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.123+0000 7fda59106700 1 -- 172.21.15.31:0/3521769486 wait complete. 2024-04-26T17:56:21.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:21 smithi186 ceph-mon[32797]: pgmap v396: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:21.781 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:21 smithi031 ceph-mon[29164]: pgmap v396: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:21.812 INFO:teuthology.run_tasks:Running task sequential... 2024-04-26T17:56:21.821 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-26T17:56:21.822 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-26T17:56:21.822 INFO:teuthology.task.print:Enabling mgr modules 2024-04-26T17:56:21.822 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-26T17:56:21.823 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T17:56:21.823 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-26T17:56:21.824 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-26T17:56:21.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:21 smithi097 ceph-mon[32419]: pgmap v396: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T17:56:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.930+0000 7fbee5cf8640 1 -- 172.21.15.31:0/993556943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee0102220 msgr2=0x7fbee01026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.930+0000 7fbee5cf8640 1 --2- 172.21.15.31:0/993556943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee0102220 0x7fbee01026a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbed4009900 tx=0x7fbed402a410 comp rx=0 tx=0).stop 2024-04-26T17:56:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.930+0000 7fbee5cf8640 1 -- 172.21.15.31:0/993556943 shutdown_connections 2024-04-26T17:56:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.930+0000 7fbee5cf8640 1 --2- 172.21.15.31:0/993556943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbee0102be0 0x7fbee0109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.930+0000 7fbee5cf8640 1 --2- 172.21.15.31:0/993556943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee0102220 0x7fbee01026a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.930+0000 7fbee5cf8640 1 --2- 172.21.15.31:0/993556943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbee0101030 0x7fbee0101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.930+0000 7fbee5cf8640 1 -- 172.21.15.31:0/993556943 >> 172.21.15.31:0/993556943 conn(0x7fbee00fc5a0 msgr2=0x7fbee00fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.930+0000 7fbee5cf8640 1 -- 172.21.15.31:0/993556943 shutdown_connections 2024-04-26T17:56:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.930+0000 7fbee5cf8640 1 -- 172.21.15.31:0/993556943 wait complete. 2024-04-26T17:56:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.931+0000 7fbee5cf8640 1 Processor -- start 2024-04-26T17:56:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.931+0000 7fbee5cf8640 1 -- start start 2024-04-26T17:56:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.931+0000 7fbee5cf8640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbee0101030 0x7fbee0198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.931+0000 7fbee5cf8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbee0102220 0x7fbee0198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.931+0000 7fbee5cf8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee0102be0 0x7fbee01a0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.931+0000 7fbee5cf8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbee010d3c0 con 0x7fbee0102be0 2024-04-26T17:56:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.931+0000 7fbee5cf8640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbee010d240 con 0x7fbee0101030 2024-04-26T17:56:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.931+0000 7fbee5cf8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbee010d540 con 0x7fbee0102220 2024-04-26T17:56:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.931+0000 7fbedffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee0102be0 0x7fbee01a0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.931+0000 7fbedffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee0102be0 0x7fbee01a0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53560/0 (socket says 172.21.15.31:53560) 2024-04-26T17:56:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.931+0000 7fbedffff640 1 -- 172.21.15.31:0/1600712858 learned_addr learned my addr 172.21.15.31:0/1600712858 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.932+0000 7fbedf7fe640 1 --2- 172.21.15.31:0/1600712858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbee0101030 0x7fbee0198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.932+0000 7fbedeffd640 1 --2- 172.21.15.31:0/1600712858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbee0102220 0x7fbee0198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.932+0000 7fbedffff640 1 -- 172.21.15.31:0/1600712858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbee0102220 msgr2=0x7fbee0198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.932+0000 7fbedffff640 1 --2- 172.21.15.31:0/1600712858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbee0102220 0x7fbee0198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.932+0000 7fbedffff640 1 -- 172.21.15.31:0/1600712858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbee0101030 msgr2=0x7fbee0198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.932+0000 7fbedffff640 1 --2- 172.21.15.31:0/1600712858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbee0101030 0x7fbee0198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.932+0000 7fbedffff640 1 -- 172.21.15.31:0/1600712858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee01a06c0 con 0x7fbee0102be0 2024-04-26T17:56:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.932+0000 7fbedffff640 1 --2- 172.21.15.31:0/1600712858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee0102be0 0x7fbee01a0050 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbed800eaa0 tx=0x7fbed800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.932+0000 7fbedcff9640 1 -- 172.21.15.31:0/1600712858 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed800cdb0 con 0x7fbee0102be0 2024-04-26T17:56:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.932+0000 7fbedcff9640 1 -- 172.21.15.31:0/1600712858 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbed8005020 con 0x7fbee0102be0 2024-04-26T17:56:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.932+0000 7fbee5cf8640 1 -- 172.21.15.31:0/1600712858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee01a09b0 con 0x7fbee0102be0 2024-04-26T17:56:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.932+0000 7fbedcff9640 1 -- 172.21.15.31:0/1600712858 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed80116d0 con 0x7fbee0102be0 2024-04-26T17:56:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.932+0000 7fbee5cf8640 1 -- 172.21.15.31:0/1600712858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee01a0eb0 con 0x7fbee0102be0 2024-04-26T17:56:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.933+0000 7fbee5cf8640 1 -- 172.21.15.31:0/1600712858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbee01a10e0 con 0x7fbee0102be0 2024-04-26T17:56:21.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.935+0000 7fbedcff9640 1 -- 172.21.15.31:0/1600712858 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbed80051d0 con 0x7fbee0102be0 2024-04-26T17:56:21.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.935+0000 7fbedcff9640 1 --2- 172.21.15.31:0/1600712858 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fbec80a3c50 0x7fbec80a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:21.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.935+0000 7fbedcff9640 1 -- 172.21.15.31:0/1600712858 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fbed80d09c0 con 0x7fbee0102be0 2024-04-26T17:56:21.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.935+0000 7fbedf7fe640 1 --2- 172.21.15.31:0/1600712858 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fbec80a3c50 0x7fbec80a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:21.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.936+0000 7fbedf7fe640 1 --2- 172.21.15.31:0/1600712858 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fbec80a3c50 0x7fbec80a6110 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbee0102080 tx=0x7fbed0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:21.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:21.939+0000 7fbedcff9640 1 -- 172.21.15.31:0/1600712858 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbed8016030 con 0x7fbee0102be0 2024-04-26T17:56:22.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.177+0000 7fbee5cf8640 1 -- 172.21.15.31:0/1600712858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fbee01a12b0 con 0x7fbee0102be0 2024-04-26T17:56:22.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.418+0000 7fbedcff9640 1 -- 172.21.15.31:0/1600712858 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fbed809d3d0 con 0x7fbee0102be0 2024-04-26T17:56:22.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.420+0000 7fbee5cf8640 1 -- 172.21.15.31:0/1600712858 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fbec80a3c50 msgr2=0x7fbec80a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:22.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.420+0000 7fbee5cf8640 1 --2- 172.21.15.31:0/1600712858 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fbec80a3c50 0x7fbec80a6110 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbee0102080 tx=0x7fbed0005f90 comp rx=0 tx=0).stop 2024-04-26T17:56:22.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.420+0000 7fbee5cf8640 1 -- 172.21.15.31:0/1600712858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee0102be0 msgr2=0x7fbee01a0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:22.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.420+0000 7fbee5cf8640 1 --2- 172.21.15.31:0/1600712858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee0102be0 0x7fbee01a0050 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbed800eaa0 tx=0x7fbed800ef60 comp rx=0 tx=0).stop 2024-04-26T17:56:22.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.421+0000 7fbee5cf8640 1 -- 172.21.15.31:0/1600712858 shutdown_connections 2024-04-26T17:56:22.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.421+0000 7fbee5cf8640 1 --2- 172.21.15.31:0/1600712858 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fbec80a3c50 0x7fbec80a6110 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.421+0000 7fbee5cf8640 1 --2- 172.21.15.31:0/1600712858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbee0102be0 0x7fbee01a0050 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.421+0000 7fbee5cf8640 1 --2- 172.21.15.31:0/1600712858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbee0102220 0x7fbee0198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.421+0000 7fbee5cf8640 1 --2- 172.21.15.31:0/1600712858 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbee0101030 0x7fbee0198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.421+0000 7fbee5cf8640 1 -- 172.21.15.31:0/1600712858 >> 172.21.15.31:0/1600712858 conn(0x7fbee00fc5a0 msgr2=0x7fbee00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:22.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.422+0000 7fbee5cf8640 1 -- 172.21.15.31:0/1600712858 shutdown_connections 2024-04-26T17:56:22.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.422+0000 7fbee5cf8640 1 -- 172.21.15.31:0/1600712858 wait complete. 2024-04-26T17:56:22.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-26T17:56:22.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.547+0000 7f881e0bd640 1 -- 172.21.15.31:0/2209789088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8818101890 msgr2=0x7f8818101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:22.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.547+0000 7f881e0bd640 1 --2- 172.21.15.31:0/2209789088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8818101890 0x7f8818101cb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8808009900 tx=0x7f880802a410 comp rx=0 tx=0).stop 2024-04-26T17:56:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.547+0000 7f881e0bd640 1 -- 172.21.15.31:0/2209789088 shutdown_connections 2024-04-26T17:56:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.547+0000 7f881e0bd640 1 --2- 172.21.15.31:0/2209789088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8818103440 0x7f8818109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.547+0000 7f881e0bd640 1 --2- 172.21.15.31:0/2209789088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8818102a80 0x7f8818102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.547+0000 7f881e0bd640 1 --2- 172.21.15.31:0/2209789088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8818101890 0x7f8818101cb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.547+0000 7f881e0bd640 1 -- 172.21.15.31:0/2209789088 >> 172.21.15.31:0/2209789088 conn(0x7f88180fce00 msgr2=0x7f88180ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.547+0000 7f881e0bd640 1 -- 172.21.15.31:0/2209789088 shutdown_connections 2024-04-26T17:56:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.547+0000 7f881e0bd640 1 -- 172.21.15.31:0/2209789088 wait complete. 2024-04-26T17:56:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.548+0000 7f881e0bd640 1 Processor -- start 2024-04-26T17:56:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.548+0000 7f881e0bd640 1 -- start start 2024-04-26T17:56:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.548+0000 7f881e0bd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8818101890 0x7f881806ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.548+0000 7f881e0bd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8818102a80 0x7f881819d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.548+0000 7f881e0bd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8818103440 0x7f881819fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.548+0000 7f881e0bd640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f881810d3c0 con 0x7f8818102a80 2024-04-26T17:56:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.548+0000 7f881e0bd640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f881810d240 con 0x7f8818101890 2024-04-26T17:56:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.548+0000 7f881e0bd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f881810d540 con 0x7f8818103440 2024-04-26T17:56:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.548+0000 7f8816ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8818102a80 0x7f881819d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.548+0000 7f8816ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8818102a80 0x7f881819d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53600/0 (socket says 172.21.15.31:53600) 2024-04-26T17:56:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.548+0000 7f8816ffd640 1 -- 172.21.15.31:0/2813414018 learned_addr learned my addr 172.21.15.31:0/2813414018 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.548+0000 7f88177fe640 1 --2- 172.21.15.31:0/2813414018 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8818101890 0x7f881806ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.548+0000 7f8817fff640 1 --2- 172.21.15.31:0/2813414018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8818103440 0x7f881819fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.549+0000 7f8816ffd640 1 -- 172.21.15.31:0/2813414018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8818103440 msgr2=0x7f881819fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.549+0000 7f8816ffd640 1 --2- 172.21.15.31:0/2813414018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8818103440 0x7f881819fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.549+0000 7f8816ffd640 1 -- 172.21.15.31:0/2813414018 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8818101890 msgr2=0x7f881806ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:22.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.549+0000 7f8816ffd640 1 --2- 172.21.15.31:0/2813414018 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8818101890 0x7f881806ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.549+0000 7f8816ffd640 1 -- 172.21.15.31:0/2813414018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88181a0780 con 0x7f8818102a80 2024-04-26T17:56:22.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.549+0000 7f8816ffd640 1 --2- 172.21.15.31:0/2813414018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8818102a80 0x7f881819d880 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f880c00e9f0 tx=0x7f880c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:22.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.549+0000 7f8814ff9640 1 -- 172.21.15.31:0/2813414018 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c00cd80 con 0x7f8818102a80 2024-04-26T17:56:22.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.549+0000 7f881e0bd640 1 -- 172.21.15.31:0/2813414018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88181a0a70 con 0x7f8818102a80 2024-04-26T17:56:22.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.549+0000 7f8814ff9640 1 -- 172.21.15.31:0/2813414018 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f880c0108e0 con 0x7f8818102a80 2024-04-26T17:56:22.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.549+0000 7f8814ff9640 1 -- 172.21.15.31:0/2813414018 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c018980 con 0x7f8818102a80 2024-04-26T17:56:22.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.549+0000 7f881e0bd640 1 -- 172.21.15.31:0/2813414018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88181a1010 con 0x7f8818102a80 2024-04-26T17:56:22.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.551+0000 7f881e0bd640 1 -- 172.21.15.31:0/2813414018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8818108cd0 con 0x7f8818102a80 2024-04-26T17:56:22.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.551+0000 7f8814ff9640 1 -- 172.21.15.31:0/2813414018 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f880c010430 con 0x7f8818102a80 2024-04-26T17:56:22.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.552+0000 7f8814ff9640 1 --2- 172.21.15.31:0/2813414018 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f88000a3ca0 0x7f88000a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:22.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.552+0000 7f88177fe640 1 -- 172.21.15.31:0/2813414018 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f88000a3ca0 msgr2=0x7f88000a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2410008001 2024-04-26T17:56:22.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.552+0000 7f8814ff9640 1 -- 172.21.15.31:0/2813414018 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f880c0d06e0 con 0x7f8818102a80 2024-04-26T17:56:22.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.552+0000 7f88177fe640 1 --2- 172.21.15.31:0/2813414018 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f88000a3ca0 0x7f88000a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T17:56:22.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.557+0000 7f8814ff9640 1 -- 172.21.15.31:0/2813414018 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f880c09caa0 con 0x7f8818102a80 2024-04-26T17:56:22.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.753+0000 7f88177fe640 1 -- 172.21.15.31:0/2813414018 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f88000a3ca0 msgr2=0x7f88000a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2410008001 2024-04-26T17:56:22.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.753+0000 7f88177fe640 1 --2- 172.21.15.31:0/2813414018 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f88000a3ca0 0x7f88000a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T17:56:22.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.763+0000 7f881e0bd640 1 -- 172.21.15.31:0/2813414018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f8818063530 con 0x7f8818102a80 2024-04-26T17:56:22.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:22 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1600712858' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T17:56:22.767 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:22 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: ignoring --setuser ceph since I am not root 2024-04-26T17:56:22.767 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:22 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: ignoring --setgroup ceph since I am not root 2024-04-26T17:56:22.767 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:22 smithi186 ceph-mgr[34656]: -- 172.21.15.186:0/3228681593 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e07186a5a0 con 0x55e07183e800 2024-04-26T17:56:22.767 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:22 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:22.597+0000 7f435bc90200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T17:56:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.769+0000 7f8814ff9640 1 -- 172.21.15.31:0/2813414018 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7f880c0a0da0 con 0x7f8818102a80 2024-04-26T17:56:22.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.771+0000 7f881e0bd640 1 -- 172.21.15.31:0/2813414018 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f88000a3ca0 msgr2=0x7f88000a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T17:56:22.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.771+0000 7f881e0bd640 1 --2- 172.21.15.31:0/2813414018 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f88000a3ca0 0x7f88000a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.771+0000 7f881e0bd640 1 -- 172.21.15.31:0/2813414018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8818102a80 msgr2=0x7f881819d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:22.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.771+0000 7f881e0bd640 1 --2- 172.21.15.31:0/2813414018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8818102a80 0x7f881819d880 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f880c00e9f0 tx=0x7f880c00eeb0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.771+0000 7f881e0bd640 1 -- 172.21.15.31:0/2813414018 shutdown_connections 2024-04-26T17:56:22.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.771+0000 7f881e0bd640 1 --2- 172.21.15.31:0/2813414018 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f88000a3ca0 0x7f88000a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.771+0000 7f881e0bd640 1 --2- 172.21.15.31:0/2813414018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8818103440 0x7f881819fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.771+0000 7f881e0bd640 1 --2- 172.21.15.31:0/2813414018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8818102a80 0x7f881819d880 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.771+0000 7f881e0bd640 1 --2- 172.21.15.31:0/2813414018 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8818101890 0x7f881806ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.771+0000 7f881e0bd640 1 -- 172.21.15.31:0/2813414018 >> 172.21.15.31:0/2813414018 conn(0x7f88180fce00 msgr2=0x7f8818105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:22.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.771+0000 7f881e0bd640 1 -- 172.21.15.31:0/2813414018 shutdown_connections 2024-04-26T17:56:22.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.771+0000 7f881e0bd640 1 -- 172.21.15.31:0/2813414018 wait complete. 2024-04-26T17:56:22.781 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:22 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1600712858' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T17:56:22.782 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:22 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: ignoring --setuser ceph since I am not root 2024-04-26T17:56:22.782 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:22 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: ignoring --setgroup ceph since I am not root 2024-04-26T17:56:22.782 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:22 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:22.621+0000 7f519ca9f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T17:56:22.783 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-26T17:56:22.784 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:22 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1600712858' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T17:56:22.785 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:22 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: ignoring --setuser ceph since I am not root 2024-04-26T17:56:22.785 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:22 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: ignoring --setgroup ceph since I am not root 2024-04-26T17:56:22.785 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:22 smithi097 ceph-mgr[33811]: -- 172.21.15.97:0/2016158712 <== mon.2 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5625a53825a0 con 0x5625a5357000 2024-04-26T17:56:22.785 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:22 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:22.601+0000 7fcb95bc2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T17:56:22.785 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:22 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:22.785+0000 7fcb95bc2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:56:22.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.887+0000 7f25ab946640 1 -- 172.21.15.31:0/113437953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25a4101890 msgr2=0x7f25a4101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:22.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.887+0000 7f25ab946640 1 --2- 172.21.15.31:0/113437953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25a4101890 0x7f25a4101cb0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2598009900 tx=0x7f259802a410 comp rx=0 tx=0).stop 2024-04-26T17:56:22.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.887+0000 7f25ab946640 1 -- 172.21.15.31:0/113437953 shutdown_connections 2024-04-26T17:56:22.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.887+0000 7f25ab946640 1 --2- 172.21.15.31:0/113437953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25a4103440 0x7f25a4109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.887+0000 7f25ab946640 1 --2- 172.21.15.31:0/113437953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25a4102a80 0x7f25a4102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.887+0000 7f25ab946640 1 --2- 172.21.15.31:0/113437953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25a4101890 0x7f25a4101cb0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.887+0000 7f25ab946640 1 -- 172.21.15.31:0/113437953 >> 172.21.15.31:0/113437953 conn(0x7f25a40fce00 msgr2=0x7f25a40ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:22.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.887+0000 7f25ab946640 1 -- 172.21.15.31:0/113437953 shutdown_connections 2024-04-26T17:56:22.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.887+0000 7f25ab946640 1 -- 172.21.15.31:0/113437953 wait complete. 2024-04-26T17:56:22.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25ab946640 1 Processor -- start 2024-04-26T17:56:22.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25ab946640 1 -- start start 2024-04-26T17:56:22.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25ab946640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25a4101890 0x7f25a4198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:22.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25ab946640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25a4102a80 0x7f25a4199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:22.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25ab946640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25a4103440 0x7f25a41a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:22.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25ab946640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f25a410d3c0 con 0x7f25a4101890 2024-04-26T17:56:22.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25ab946640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f25a410d240 con 0x7f25a4102a80 2024-04-26T17:56:22.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25ab946640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f25a410d540 con 0x7f25a4103440 2024-04-26T17:56:22.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25a96e2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25a4101890 0x7f25a4198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:22.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25a96e2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25a4101890 0x7f25a4198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53618/0 (socket says 172.21.15.31:53618) 2024-04-26T17:56:22.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25a96e2640 1 -- 172.21.15.31:0/580045466 learned_addr learned my addr 172.21.15.31:0/580045466 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:22.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25a9ee3640 1 --2- 172.21.15.31:0/580045466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25a4103440 0x7f25a41a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:22.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25a8ee1640 1 --2- 172.21.15.31:0/580045466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25a4102a80 0x7f25a4199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:22.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25a96e2640 1 -- 172.21.15.31:0/580045466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25a4103440 msgr2=0x7f25a41a00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:22.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25a96e2640 1 --2- 172.21.15.31:0/580045466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25a4103440 0x7f25a41a00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25a96e2640 1 -- 172.21.15.31:0/580045466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25a4102a80 msgr2=0x7f25a4199070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:22.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25a96e2640 1 --2- 172.21.15.31:0/580045466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25a4102a80 0x7f25a4199070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:22.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25a96e2640 1 -- 172.21.15.31:0/580045466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a41a0750 con 0x7f25a4101890 2024-04-26T17:56:22.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.888+0000 7f25a8ee1640 1 --2- 172.21.15.31:0/580045466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25a4102a80 0x7f25a4199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:56:22.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.889+0000 7f25a96e2640 1 --2- 172.21.15.31:0/580045466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25a4101890 0x7f25a4198b30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2598006010 tx=0x7f25980042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:22.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.889+0000 7f25967fc640 1 -- 172.21.15.31:0/580045466 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259803f030 con 0x7f25a4101890 2024-04-26T17:56:22.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.889+0000 7f25967fc640 1 -- 172.21.15.31:0/580045466 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25980318f0 con 0x7f25a4101890 2024-04-26T17:56:22.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.889+0000 7f25ab946640 1 -- 172.21.15.31:0/580045466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a41a09e0 con 0x7f25a4101890 2024-04-26T17:56:22.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.889+0000 7f25967fc640 1 -- 172.21.15.31:0/580045466 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2598035070 con 0x7f25a4101890 2024-04-26T17:56:22.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.889+0000 7f25ab946640 1 -- 172.21.15.31:0/580045466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a41a0ed0 con 0x7f25a4101890 2024-04-26T17:56:22.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.890+0000 7f25ab946640 1 -- 172.21.15.31:0/580045466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a41a10a0 con 0x7f25a4101890 2024-04-26T17:56:22.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.891+0000 7f25967fc640 1 -- 172.21.15.31:0/580045466 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f2598042030 con 0x7f25a4101890 2024-04-26T17:56:22.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.892+0000 7f25967fc640 1 --2- 172.21.15.31:0/580045466 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f25900a3ca0 0x7f25900a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:22.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.892+0000 7f25a8ee1640 1 -- 172.21.15.31:0/580045466 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f25900a3ca0 msgr2=0x7f25900a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2410008001 2024-04-26T17:56:22.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.892+0000 7f25967fc640 1 -- 172.21.15.31:0/580045466 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f25980f0990 con 0x7f25a4101890 2024-04-26T17:56:22.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.892+0000 7f25a8ee1640 1 --2- 172.21.15.31:0/580045466 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f25900a3ca0 0x7f25900a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T17:56:22.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:22.896+0000 7f25967fc640 1 -- 172.21.15.31:0/580045466 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f25980bd3a0 con 0x7f25a4101890 2024-04-26T17:56:23.055 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:22 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:22.817+0000 7f519ca9f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:56:23.399 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:22 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:22.929+0000 7f519ca9f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T17:56:23.401 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:22 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:22.785+0000 7f435bc90200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:56:23.401 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:22 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:22.888+0000 7f435bc90200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T17:56:23.401 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:23 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:23.198+0000 7f435bc90200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T17:56:23.401 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:22 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:22.887+0000 7fcb95bc2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T17:56:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:23.085+0000 7f25ab946640 1 -- 172.21.15.31:0/580045466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f25a41a1270 con 0x7f25a4101890 2024-04-26T17:56:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:23.088+0000 7f25967fc640 1 -- 172.21.15.31:0/580045466 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f25980bcca0 con 0x7f25a4101890 2024-04-26T17:56:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:23.091+0000 7f25ab946640 1 -- 172.21.15.31:0/580045466 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f25900a3ca0 msgr2=0x7f25900a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T17:56:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:23.091+0000 7f25ab946640 1 --2- 172.21.15.31:0/580045466 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f25900a3ca0 0x7f25900a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:23.091+0000 7f25ab946640 1 -- 172.21.15.31:0/580045466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25a4101890 msgr2=0x7f25a4198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:23.091+0000 7f25ab946640 1 --2- 172.21.15.31:0/580045466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25a4101890 0x7f25a4198b30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2598006010 tx=0x7f25980042f0 comp rx=0 tx=0).stop 2024-04-26T17:56:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:23.091+0000 7f25ab946640 1 -- 172.21.15.31:0/580045466 shutdown_connections 2024-04-26T17:56:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:23.091+0000 7f25ab946640 1 --2- 172.21.15.31:0/580045466 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f25900a3ca0 0x7f25900a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:23.091+0000 7f25ab946640 1 --2- 172.21.15.31:0/580045466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25a4103440 0x7f25a41a00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:23.091+0000 7f25ab946640 1 --2- 172.21.15.31:0/580045466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25a4102a80 0x7f25a4199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:23.091+0000 7f25ab946640 1 --2- 172.21.15.31:0/580045466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25a4101890 0x7f25a4198b30 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:23.091+0000 7f25ab946640 1 -- 172.21.15.31:0/580045466 >> 172.21.15.31:0/580045466 conn(0x7f25a40fce00 msgr2=0x7f25a4069fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:23.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:23.091+0000 7f25ab946640 1 -- 172.21.15.31:0/580045466 shutdown_connections 2024-04-26T17:56:23.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:23.091+0000 7f25ab946640 1 -- 172.21.15.31:0/580045466 wait complete. 2024-04-26T17:56:23.404 INFO:teuthology.run_tasks:Running task kclient... 2024-04-26T17:56:23.473 INFO:tasks.kclient:Mounting kernel clients... 2024-04-26T17:56:23.474 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-26T17:56:23.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T17:56:23.474 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-04-26T17:56:23.475 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:23 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:23.262+0000 7f519ca9f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T17:56:23.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T17:56:23.491 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-04-26T17:56:23.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:23 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1600712858' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T17:56:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:23 smithi186 ceph-mon[32797]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-26T17:56:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:23 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2813414018' entity='client.admin' 2024-04-26T17:56:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:23 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/580045466' entity='client.admin' 2024-04-26T17:56:23.517 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:23 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:23.283+0000 7f435bc90200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T17:56:23.558 INFO:teuthology.orchestra.run.smithi031.stderr:Cannot find device "ceph-brx" 2024-04-26T17:56:23.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T17:56:23.560 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync', 'context=system_u:object_r:postgresql_db_t:s0'], 'syntax': 'v2', 'mount_subvol_num': 1, 'mountpoint': '/tmp/cephfs'} 2024-04-26T17:56:23.560 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T17:56:23.560 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2024-04-26T17:56:23.560 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2024-04-26T17:56:23.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T17:56:23.623 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:23 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:23.350+0000 7f519ca9f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T17:56:23.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:23 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1600712858' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T17:56:23.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:23 smithi031 ceph-mon[29164]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-26T17:56:23.624 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:23 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2813414018' entity='client.admin' 2024-04-26T17:56:23.624 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:23 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/580045466' entity='client.admin' 2024-04-26T17:56:23.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:23 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1600712858' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T17:56:23.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:23 smithi097 ceph-mon[32419]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-26T17:56:23.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:23 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2813414018' entity='client.admin' 2024-04-26T17:56:23.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:23 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/580045466' entity='client.admin' 2024-04-26T17:56:23.634 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:23 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:23.194+0000 7fcb95bc2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T17:56:23.634 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:23 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:23.278+0000 7fcb95bc2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T17:56:23.897 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:24.766 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:24.466+0000 7f435bc90200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T17:56:24.767 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:24.636+0000 7f435bc90200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T17:56:24.786 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:24 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:24.450+0000 7fcb95bc2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T17:56:24.786 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:24 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:24.617+0000 7fcb95bc2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T17:56:24.989 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:24 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:24.629+0000 7f519ca9f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T17:56:24.989 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:24 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:24.809+0000 7f519ca9f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T17:56:24.989 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:24 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:24.988+0000 7f519ca9f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T17:56:25.072 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:24 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:24.786+0000 7fcb95bc2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T17:56:25.094 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:24.807+0000 7f435bc90200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T17:56:25.366 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:25 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:25.072+0000 7fcb95bc2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T17:56:25.366 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:25 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:25.165+0000 7fcb95bc2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T17:56:25.366 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:25 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:25.367+0000 7fcb95bc2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T17:56:25.390 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:25.094+0000 7f435bc90200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T17:56:25.391 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:25.186+0000 7f435bc90200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T17:56:25.560 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:25 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:25.300+0000 7f519ca9f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T17:56:25.560 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:25 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:25.402+0000 7f519ca9f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T17:56:25.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.569+0000 7f846a8fe700 1 -- 172.21.15.31:0/2598354545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846406b9c0 msgr2=0x7f846406be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:25.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.569+0000 7f846a8fe700 1 --2- 172.21.15.31:0/2598354545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846406b9c0 0x7f846406be20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8454009990 tx=0x7f8454009ca0 comp rx=0 tx=0).stop 2024-04-26T17:56:25.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.569+0000 7f846a8fe700 1 -- 172.21.15.31:0/2598354545 shutdown_connections 2024-04-26T17:56:25.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.569+0000 7f846a8fe700 1 --2- 172.21.15.31:0/2598354545 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8464109490 0x7f8464109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:25.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.569+0000 7f846a8fe700 1 --2- 172.21.15.31:0/2598354545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f846406c360 0x7f84641102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:25.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.569+0000 7f846a8fe700 1 --2- 172.21.15.31:0/2598354545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846406b9c0 0x7f846406be20 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:25.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.569+0000 7f846a8fe700 1 -- 172.21.15.31:0/2598354545 >> 172.21.15.31:0/2598354545 conn(0x7f8464079290 msgr2=0x7f84640796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:25.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.569+0000 7f846a8fe700 1 -- 172.21.15.31:0/2598354545 shutdown_connections 2024-04-26T17:56:25.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.569+0000 7f846a8fe700 1 -- 172.21.15.31:0/2598354545 wait complete. 2024-04-26T17:56:25.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.570+0000 7f846a8fe700 1 Processor -- start 2024-04-26T17:56:25.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.570+0000 7f846a8fe700 1 -- start start 2024-04-26T17:56:25.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.570+0000 7f846a8fe700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f846406b9c0 0x7f84641023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:25.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.570+0000 7f846a8fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846406c360 0x7f8464102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:25.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.570+0000 7f846a8fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8464109490 0x7f8464102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:25.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.570+0000 7f846a8fe700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f84641135a0 con 0x7f846406c360 2024-04-26T17:56:25.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.570+0000 7f846a8fe700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f8464113420 con 0x7f846406b9c0 2024-04-26T17:56:25.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.570+0000 7f846a8fe700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8464113720 con 0x7f8464109490 2024-04-26T17:56:25.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.570+0000 7f84637fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846406c360 0x7f8464102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:25.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.570+0000 7f84637fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846406c360 0x7f8464102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50224/0 (socket says 172.21.15.31:50224) 2024-04-26T17:56:25.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.570+0000 7f8463fff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f846406b9c0 0x7f84641023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:25.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.570+0000 7f84637fe700 1 -- 172.21.15.31:0/2280560661 learned_addr learned my addr 172.21.15.31:0/2280560661 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:25.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.570+0000 7f8468e9b700 1 --2- 172.21.15.31:0/2280560661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8464109490 0x7f8464102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:25.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.571+0000 7f84637fe700 1 -- 172.21.15.31:0/2280560661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8464109490 msgr2=0x7f8464102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:25.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.571+0000 7f84637fe700 1 --2- 172.21.15.31:0/2280560661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8464109490 0x7f8464102e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:25.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.571+0000 7f84637fe700 1 -- 172.21.15.31:0/2280560661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f846406b9c0 msgr2=0x7f84641023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:25.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.571+0000 7f84637fe700 1 --2- 172.21.15.31:0/2280560661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f846406b9c0 0x7f84641023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:25.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.571+0000 7f84637fe700 1 -- 172.21.15.31:0/2280560661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84641a7fd0 con 0x7f846406c360 2024-04-26T17:56:25.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.571+0000 7f84637fe700 1 --2- 172.21.15.31:0/2280560661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846406c360 0x7f8464102920 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8454000c00 tx=0x7f845400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:25.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.571+0000 7f84617fa700 1 -- 172.21.15.31:0/2280560661 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8454027070 con 0x7f846406c360 2024-04-26T17:56:25.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.571+0000 7f846a8fe700 1 -- 172.21.15.31:0/2280560661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84641a8260 con 0x7f846406c360 2024-04-26T17:56:25.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.571+0000 7f846a8fe700 1 -- 172.21.15.31:0/2280560661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84641a8710 con 0x7f846406c360 2024-04-26T17:56:25.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.571+0000 7f84617fa700 1 -- 172.21.15.31:0/2280560661 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8454022470 con 0x7f846406c360 2024-04-26T17:56:25.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.571+0000 7f84617fa700 1 -- 172.21.15.31:0/2280560661 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8454017650 con 0x7f846406c360 2024-04-26T17:56:25.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.572+0000 7f846a8fe700 1 -- 172.21.15.31:0/2280560661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8450005330 con 0x7f846406c360 2024-04-26T17:56:25.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.573+0000 7f84617fa700 1 -- 172.21.15.31:0/2280560661 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f845402a070 con 0x7f846406c360 2024-04-26T17:56:25.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.574+0000 7f84617fa700 1 --2- 172.21.15.31:0/2280560661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f844c0a3d10 0x7f844c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:25.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.574+0000 7f8463fff700 1 -- 172.21.15.31:0/2280560661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f844c0a3d10 msgr2=0x7f844c0a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2410008001 2024-04-26T17:56:25.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.574+0000 7f84617fa700 1 -- 172.21.15.31:0/2280560661 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f84540d11d0 con 0x7f846406c360 2024-04-26T17:56:25.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.574+0000 7f8463fff700 1 --2- 172.21.15.31:0/2280560661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f844c0a3d10 0x7f844c0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T17:56:25.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.578+0000 7f84617fa700 1 -- 172.21.15.31:0/2280560661 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8454013070 con 0x7f846406c360 2024-04-26T17:56:25.634 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:25 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:25.559+0000 7fcb95bc2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:56:25.766 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:25.390+0000 7f435bc90200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T17:56:25.767 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:25.583+0000 7f435bc90200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:56:25.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.775+0000 7f8463fff700 1 -- 172.21.15.31:0/2280560661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f844c0a3d10 msgr2=0x7f844c0a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2410008001 2024-04-26T17:56:25.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.775+0000 7f8463fff700 1 --2- 172.21.15.31:0/2280560661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f844c0a3d10 0x7f844c0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T17:56:25.825 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:25 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:25.620+0000 7f519ca9f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T17:56:25.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.833+0000 7f846a8fe700 1 -- 172.21.15.31:0/2280560661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f84500058f0 con 0x7f846406c360 2024-04-26T17:56:25.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.833+0000 7f84617fa700 1 -- 172.21.15.31:0/2280560661 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f845409dc10 con 0x7f846406c360 2024-04-26T17:56:25.833 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-26T17:56:25.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.835+0000 7f846a8fe700 1 -- 172.21.15.31:0/2280560661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f844c0a3d10 msgr2=0x7f844c0a61d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T17:56:25.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.835+0000 7f846a8fe700 1 --2- 172.21.15.31:0/2280560661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f844c0a3d10 0x7f844c0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:25.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.835+0000 7f846a8fe700 1 -- 172.21.15.31:0/2280560661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846406c360 msgr2=0x7f8464102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:25.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.835+0000 7f846a8fe700 1 --2- 172.21.15.31:0/2280560661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846406c360 0x7f8464102920 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8454000c00 tx=0x7f845400bfd0 comp rx=0 tx=0).stop 2024-04-26T17:56:25.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.835+0000 7f846a8fe700 1 -- 172.21.15.31:0/2280560661 shutdown_connections 2024-04-26T17:56:25.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.835+0000 7f846a8fe700 1 --2- 172.21.15.31:0/2280560661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f844c0a3d10 0x7f844c0a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:25.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.835+0000 7f846a8fe700 1 --2- 172.21.15.31:0/2280560661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f846406b9c0 0x7f84641023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:25.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.835+0000 7f846a8fe700 1 --2- 172.21.15.31:0/2280560661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8464109490 0x7f8464102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:25.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.835+0000 7f846a8fe700 1 --2- 172.21.15.31:0/2280560661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846406c360 0x7f8464102920 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:25.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.835+0000 7f846a8fe700 1 -- 172.21.15.31:0/2280560661 >> 172.21.15.31:0/2280560661 conn(0x7f8464079290 msgr2=0x7f84641003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:25.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.835+0000 7f846a8fe700 1 -- 172.21.15.31:0/2280560661 shutdown_connections 2024-04-26T17:56:25.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:25.835+0000 7f846a8fe700 1 -- 172.21.15.31:0/2280560661 wait complete. 2024-04-26T17:56:26.154 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:25 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2280560661' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T17:56:26.186 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:25 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2280560661' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T17:56:26.214 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:25 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:25.825+0000 7f519ca9f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T17:56:26.215 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:25 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2280560661' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T17:56:26.516 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:26 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:26.186+0000 7f435bc90200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T17:56:26.534 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T17:56:26.534 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T17:56:26.534 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-04-26T17:56:26.534 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T17:56:26.534 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2024-04-26T17:56:26.534 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T17:56:26.534 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T17:56:26.535 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T17:56:26.535 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2024-04-26T17:56:26.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T17:56:26.535 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-04-26T17:56:26.570 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:26 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:26.154+0000 7fcb95bc2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T17:56:26.592 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T17:56:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T17:56:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T17:56:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-04-26T17:56:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-04-26T17:56:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-04-26T17:56:26.592 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T17:56:26.592 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-04-26T17:56:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-04-26T17:56:26.593 INFO:teuthology.orchestra.run.smithi031.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T17:56:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-04-26T17:56:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-04-26T17:56:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-26T17:56:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 41986sec preferred_lft 41986sec 2024-04-26T17:56:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link noprefixroute 2024-04-26T17:56:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-04-26T17:56:26.593 INFO:teuthology.orchestra.run.smithi031.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T17:56:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-04-26T17:56:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-04-26T17:56:26.593 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T17:56:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-04-26T17:56:26.593 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-04-26T17:56:26.594 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T17:56:26.594 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T17:56:26.594 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-26T17:56:26.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add name ceph-brx type bridge 2024-04-26T17:56:26.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr flush dev ceph-brx 2024-04-26T17:56:26.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set ceph-brx up 2024-04-26T17:56:26.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T17:56:26.595 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-26T17:56:26.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T17:56:26.745 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:26.467+0000 7f519ca9f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T17:56:26.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T17:56:26.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T17:56:26.788 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T17:56:26.814 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-04-26T17:56:26.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T17:56:26.815 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-04-26T17:56:26.866 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:26 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:26.570+0000 7fcb95bc2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T17:56:26.866 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:26 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:26.666+0000 7fcb95bc2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T17:56:26.891 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-04-26T17:56:26.891 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T17:56:26.891 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T17:56:26.891 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T17:56:26.891 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T17:56:26.892 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T17:56:26.892 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-26T17:56:26.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T17:56:26.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T17:56:26.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T17:56:26.893 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-26T17:56:26.904 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:26 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:26.606+0000 7f435bc90200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T17:56:26.904 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:26 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:26.703+0000 7f435bc90200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T17:56:26.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T17:56:27.037 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:26.931+0000 7f519ca9f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T17:56:27.133 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:26 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:26.867+0000 7fcb95bc2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T17:56:27.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T17:56:27.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T17:56:27.173 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-04-26T17:56:27.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T17:56:27.189 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list-id 2024-04-26T17:56:27.244 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T17:56:27.244 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-26T17:56:27.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns add ceph-ns--tmp-cephfs 2024-04-26T17:56:27.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns set ceph-ns--tmp-cephfs 0 2024-04-26T17:56:27.244 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-26T17:56:27.266 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:26 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:26.904+0000 7f435bc90200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T17:56:27.309 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:27 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:27.037+0000 7f519ca9f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T17:56:27.309 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:27 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:27.257+0000 7f519ca9f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T17:56:27.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T17:56:27.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T17:56:27.365 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2024-04-26T17:56:27.365 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T17:56:27.365 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-26T17:56:27.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2024-04-26T17:56:27.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T17:56:27.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2024-04-26T17:56:27.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2024-04-26T17:56:27.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2024-04-26T17:56:27.365 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-26T17:56:27.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T17:56:27.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T17:56:27.539 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T17:56:27.539 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-26T17:56:27.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-04-26T17:56:27.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T17:56:27.539 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-26T17:56:27.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T17:56:27.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T17:56:27.620 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T17:56:27.620 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /tmp/cephfs 2024-04-26T17:56:27.635 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/tmp/cephfs' 2024-04-26T17:56:27.636 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T17:56:27.636 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /tmp/cephfs 2024-04-26T17:56:27.691 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-26T17:56:27.691 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-26T17:56:27.691 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/26ef2745-8372-4260-b242-0000b032cf66 2024-04-26T17:56:27.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T17:56:27.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/26ef2745-8372-4260-b242-0000b032cf66 /tmp/cephfs -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context=system_u:object_r:postgresql_db_t:s0,nofallback 2024-04-26T17:56:27.716 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:27 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:27.386+0000 7fcb95bc2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T17:56:27.716 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:27 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:27.477+0000 7fcb95bc2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T17:56:27.760 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:27 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:27.428+0000 7f435bc90200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T17:56:27.760 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:27 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:27.520+0000 7f435bc90200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T17:56:27.786 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",nofallback 2024-04-26T17:56:27.786 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-26T17:56:27.851 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:27 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:27.826+0000 7f519ca9f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T17:56:27.898 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",nofallback 2024-04-26T17:56:27.898 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-26T17:56:27.898 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.97:3300,172.21.15.186:3300" 2024-04-26T17:56:27.898 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 0@7e53e050-03f4-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/26ef2745-8372-4260-b242-0000b032cf66 2024-04-26T17:56:27.899 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",name=0,key=0,mon_addr=172.21.15.31:3300/172.21.15.97:3300/172.21.15.186:3300" will pass to kernel 2024-04-26T17:56:27.902 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T17:56:27.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T17:56:27.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /tmp/cephfs 2024-04-26T17:56:27.930 DEBUG:teuthology.orchestra.run.smithi031:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T17:56:28.017 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:27 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:27.760+0000 7f435bc90200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T17:56:28.017 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:27 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:27.893+0000 7f435bc90200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T17:56:28.017 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:27 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:27.989+0000 7f435bc90200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T17:56:28.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T17:56:28.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T17:56:28.029 INFO:teuthology.orchestra.run.smithi031.stdout:client15156 2024-04-26T17:56:28.030 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T17:56:28.030 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T17:56:28.050 DEBUG:teuthology.orchestra.run.smithi031:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T17:56:28.058 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:27 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:27.717+0000 7fcb95bc2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T17:56:28.058 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:27 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:27.848+0000 7fcb95bc2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T17:56:28.058 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:27 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:27.944+0000 7fcb95bc2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T17:56:28.059 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:28 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:28.059+0000 7fcb95bc2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T17:56:28.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T17:56:28.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T17:56:28.149 INFO:teuthology.orchestra.run.smithi031.stdout:client15156 2024-04-26T17:56:28.149 DEBUG:teuthology.orchestra.run.smithi031:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T17:56:28.196 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:27 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:27.930+0000 7f519ca9f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T17:56:28.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T17:56:28.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T17:56:28.208 INFO:teuthology.orchestra.run.smithi031.stdout:client15156 2024-04-26T17:56:28.208 DEBUG:teuthology.orchestra.run.smithi031:> (cd /tmp/cephfs && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/7e53e050-03f4-11ef-bc93-c7b262605968.client15156/status') 2024-04-26T17:56:28.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T17:56:28.267 INFO:teuthology.orchestra.run.smithi031.stdout:instance: client.15156 (3)192.168.0.1:0/2547335269 2024-04-26T17:56:28.268 INFO:teuthology.orchestra.run.smithi031.stdout:blocklisted: false 2024-04-26T17:56:28.268 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records in 2024-04-26T17:56:28.268 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records out 2024-04-26T17:56:28.268 INFO:teuthology.orchestra.run.smithi031.stderr:70 bytes copied, 3.5202e-05 s, 2.0 MB/s 2024-04-26T17:56:28.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T17:56:28.276 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T17:56:28.286 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T17:56:28.286 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-26T17:56:28.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-26T17:56:28.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.390+0000 7fca43324640 1 -- 172.21.15.31:0/2125377056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca3c102a80 msgr2=0x7fca3c102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:28.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.390+0000 7fca43324640 1 --2- 172.21.15.31:0/2125377056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca3c102a80 0x7fca3c102f00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fca30009900 tx=0x7fca3002a410 comp rx=0 tx=0).stop 2024-04-26T17:56:28.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.390+0000 7fca43324640 1 -- 172.21.15.31:0/2125377056 shutdown_connections 2024-04-26T17:56:28.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.390+0000 7fca43324640 1 --2- 172.21.15.31:0/2125377056 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca3c103440 0x7fca3c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.390+0000 7fca43324640 1 --2- 172.21.15.31:0/2125377056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca3c102a80 0x7fca3c102f00 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.390+0000 7fca43324640 1 --2- 172.21.15.31:0/2125377056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca3c101890 0x7fca3c101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.390+0000 7fca43324640 1 -- 172.21.15.31:0/2125377056 >> 172.21.15.31:0/2125377056 conn(0x7fca3c0fce00 msgr2=0x7fca3c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.390+0000 7fca43324640 1 -- 172.21.15.31:0/2125377056 shutdown_connections 2024-04-26T17:56:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.390+0000 7fca43324640 1 -- 172.21.15.31:0/2125377056 wait complete. 2024-04-26T17:56:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.390+0000 7fca43324640 1 Processor -- start 2024-04-26T17:56:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca43324640 1 -- start start 2024-04-26T17:56:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca43324640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca3c101890 0x7fca3c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca43324640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca3c102a80 0x7fca3c199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca43324640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca3c103440 0x7fca3c1a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca43324640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fca3c10d3c0 con 0x7fca3c101890 2024-04-26T17:56:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca43324640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fca3c10d240 con 0x7fca3c103440 2024-04-26T17:56:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca43324640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fca3c10d540 con 0x7fca3c102a80 2024-04-26T17:56:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca410c0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca3c101890 0x7fca3c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca410c0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca3c101890 0x7fca3c198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50238/0 (socket says 172.21.15.31:50238) 2024-04-26T17:56:28.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca410c0640 1 -- 172.21.15.31:0/1826931661 learned_addr learned my addr 172.21.15.31:0/1826931661 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:28.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca418c1640 1 --2- 172.21.15.31:0/1826931661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca3c103440 0x7fca3c1a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:28.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca408bf640 1 --2- 172.21.15.31:0/1826931661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca3c102a80 0x7fca3c199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:28.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca410c0640 1 -- 172.21.15.31:0/1826931661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca3c102a80 msgr2=0x7fca3c199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:28.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca410c0640 1 --2- 172.21.15.31:0/1826931661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca3c102a80 0x7fca3c199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca410c0640 1 -- 172.21.15.31:0/1826931661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca3c103440 msgr2=0x7fca3c1a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:28.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca410c0640 1 --2- 172.21.15.31:0/1826931661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca3c103440 0x7fca3c1a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca410c0640 1 -- 172.21.15.31:0/1826931661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3c1a0790 con 0x7fca3c101890 2024-04-26T17:56:28.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca410c0640 1 --2- 172.21.15.31:0/1826931661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca3c101890 0x7fca3c198bc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fca3800e9c0 tx=0x7fca3800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:28.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.391+0000 7fca2e7fc640 1 -- 172.21.15.31:0/1826931661 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3800cd80 con 0x7fca3c101890 2024-04-26T17:56:28.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.392+0000 7fca2e7fc640 1 -- 172.21.15.31:0/1826931661 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca380035f0 con 0x7fca3c101890 2024-04-26T17:56:28.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.392+0000 7fca43324640 1 -- 172.21.15.31:0/1826931661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca3c1a0a80 con 0x7fca3c101890 2024-04-26T17:56:28.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.392+0000 7fca2e7fc640 1 -- 172.21.15.31:0/1826931661 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38010690 con 0x7fca3c101890 2024-04-26T17:56:28.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.392+0000 7fca43324640 1 -- 172.21.15.31:0/1826931661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca3c1a0f50 con 0x7fca3c101890 2024-04-26T17:56:28.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.393+0000 7fca43324640 1 -- 172.21.15.31:0/1826931661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca3c1a1180 con 0x7fca3c101890 2024-04-26T17:56:28.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.394+0000 7fca2e7fc640 1 -- 172.21.15.31:0/1826931661 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fca38004000 con 0x7fca3c101890 2024-04-26T17:56:28.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.394+0000 7fca2e7fc640 1 --2- 172.21.15.31:0/1826931661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fca280a3ca0 0x7fca280a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:28.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.394+0000 7fca408bf640 1 -- 172.21.15.31:0/1826931661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fca280a3ca0 msgr2=0x7fca280a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2410008001 2024-04-26T17:56:28.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.394+0000 7fca2e7fc640 1 -- 172.21.15.31:0/1826931661 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7fca380d0910 con 0x7fca3c101890 2024-04-26T17:56:28.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.394+0000 7fca408bf640 1 --2- 172.21.15.31:0/1826931661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fca280a3ca0 0x7fca280a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T17:56:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.399+0000 7fca2e7fc640 1 -- 172.21.15.31:0/1826931661 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fca3809d320 con 0x7fca3c101890 2024-04-26T17:56:28.407 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:28 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:28.103+0000 7f435bc90200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T17:56:28.408 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:28 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:28.407+0000 7f435bc90200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T17:56:28.449 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:28 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:28.196+0000 7f519ca9f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T17:56:28.450 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:28 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:28.344+0000 7f519ca9f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T17:56:28.450 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:28 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:28.449+0000 7f519ca9f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T17:56:28.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.595+0000 7fca408bf640 1 -- 172.21.15.31:0/1826931661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fca280a3ca0 msgr2=0x7fca280a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2410008001 2024-04-26T17:56:28.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.595+0000 7fca408bf640 1 --2- 172.21.15.31:0/1826931661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fca280a3ca0 0x7fca280a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T17:56:28.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.627+0000 7fca43324640 1 -- 172.21.15.31:0/1826931661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fca3c1a1570 con 0x7fca3c101890 2024-04-26T17:56:28.633 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:28 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:28.358+0000 7fcb95bc2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T17:56:28.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.678+0000 7fca2e7fc640 1 -- 172.21.15.31:0/1826931661 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v26) ==== 117+0+0 (secure 0 0 0) 0x7fca3809cc20 con 0x7fca3c101890 2024-04-26T17:56:28.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.681+0000 7fca43324640 1 -- 172.21.15.31:0/1826931661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fca280a3ca0 msgr2=0x7fca280a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T17:56:28.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.681+0000 7fca43324640 1 --2- 172.21.15.31:0/1826931661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fca280a3ca0 0x7fca280a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.681+0000 7fca43324640 1 -- 172.21.15.31:0/1826931661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca3c101890 msgr2=0x7fca3c198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:28.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.681+0000 7fca43324640 1 --2- 172.21.15.31:0/1826931661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca3c101890 0x7fca3c198bc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fca3800e9c0 tx=0x7fca3800ee80 comp rx=0 tx=0).stop 2024-04-26T17:56:28.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.682+0000 7fca43324640 1 -- 172.21.15.31:0/1826931661 shutdown_connections 2024-04-26T17:56:28.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.682+0000 7fca43324640 1 --2- 172.21.15.31:0/1826931661 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7fca280a3ca0 0x7fca280a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.682+0000 7fca43324640 1 --2- 172.21.15.31:0/1826931661 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fca3c103440 0x7fca3c1a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.682+0000 7fca43324640 1 --2- 172.21.15.31:0/1826931661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca3c102a80 0x7fca3c199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.682+0000 7fca43324640 1 --2- 172.21.15.31:0/1826931661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca3c101890 0x7fca3c198bc0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.682+0000 7fca43324640 1 -- 172.21.15.31:0/1826931661 >> 172.21.15.31:0/1826931661 conn(0x7fca3c0fce00 msgr2=0x7fca3c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:28.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.682+0000 7fca43324640 1 -- 172.21.15.31:0/1826931661 shutdown_connections 2024-04-26T17:56:28.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.682+0000 7fca43324640 1 -- 172.21.15.31:0/1826931661 wait complete. 2024-04-26T17:56:28.694 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-26T17:56:28.706 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T17:56:28.717 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T17:56:28.717 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-26T17:56:28.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-26T17:56:28.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:28 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1826931661' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T17:56:28.739 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:28 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:28.574+0000 7f519ca9f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T17:56:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.826+0000 7f3561623640 1 -- 172.21.15.31:0/1849439498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f355c101b70 msgr2=0x7f355c101ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.826+0000 7f3561623640 1 --2- 172.21.15.31:0/1849439498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f355c101b70 0x7f355c101ff0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3544009900 tx=0x7f354402a410 comp rx=0 tx=0).stop 2024-04-26T17:56:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 -- 172.21.15.31:0/1849439498 shutdown_connections 2024-04-26T17:56:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 --2- 172.21.15.31:0/1849439498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f355c102530 0x7f355c109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 --2- 172.21.15.31:0/1849439498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f355c101b70 0x7f355c101ff0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 --2- 172.21.15.31:0/1849439498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f355c102e40 0x7f355c103260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 -- 172.21.15.31:0/1849439498 >> 172.21.15.31:0/1849439498 conn(0x7f355c0fc5a0 msgr2=0x7f355c0fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 -- 172.21.15.31:0/1849439498 shutdown_connections 2024-04-26T17:56:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 -- 172.21.15.31:0/1849439498 wait complete. 2024-04-26T17:56:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 Processor -- start 2024-04-26T17:56:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 -- start start 2024-04-26T17:56:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f355c101b70 0x7f355c105920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f355c102530 0x7f355c103f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f355c102e40 0x7f355c1044a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f355c10d3f0 con 0x7f355c102530 2024-04-26T17:56:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f355c10d270 con 0x7f355c102e40 2024-04-26T17:56:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3561623640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f355c10d570 con 0x7f355c101b70 2024-04-26T17:56:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.827+0000 7f3553fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f355c102530 0x7f355c103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f355affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f355c101b70 0x7f355c105920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f3553fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f355c102530 0x7f355c103f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50256/0 (socket says 172.21.15.31:50256) 2024-04-26T17:56:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f3553fff640 1 -- 172.21.15.31:0/403804234 learned_addr learned my addr 172.21.15.31:0/403804234 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f355b7fe640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f355c102e40 0x7f355c1044a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f3553fff640 1 -- 172.21.15.31:0/403804234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f355c101b70 msgr2=0x7f355c105920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:28.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f3553fff640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f355c101b70 0x7f355c105920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f3553fff640 1 -- 172.21.15.31:0/403804234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f355c102e40 msgr2=0x7f355c1044a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:28.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f3553fff640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f355c102e40 0x7f355c1044a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:28.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f3553fff640 1 -- 172.21.15.31:0/403804234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355c104d30 con 0x7f355c102530 2024-04-26T17:56:28.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f355affd640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f355c101b70 0x7f355c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:56:28.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f3553fff640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f355c102530 0x7f355c103f60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3544005940 tx=0x7f35440042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:28.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f3558ff9640 1 -- 172.21.15.31:0/403804234 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354403f030 con 0x7f355c102530 2024-04-26T17:56:28.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f3561623640 1 -- 172.21.15.31:0/403804234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f355c1a2bb0 con 0x7f355c102530 2024-04-26T17:56:28.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f3558ff9640 1 -- 172.21.15.31:0/403804234 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3544003cb0 con 0x7f355c102530 2024-04-26T17:56:28.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f3558ff9640 1 -- 172.21.15.31:0/403804234 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544035070 con 0x7f355c102530 2024-04-26T17:56:28.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.828+0000 7f3561623640 1 -- 172.21.15.31:0/403804234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f355c1a3100 con 0x7f355c102530 2024-04-26T17:56:28.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.829+0000 7f3561623640 1 -- 172.21.15.31:0/403804234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f355c1a32d0 con 0x7f355c102530 2024-04-26T17:56:28.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.830+0000 7f3558ff9640 1 -- 172.21.15.31:0/403804234 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f3544042030 con 0x7f355c102530 2024-04-26T17:56:28.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.831+0000 7f3558ff9640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35480a3ca0 0x7f35480a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:28.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.831+0000 7f355affd640 1 -- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35480a3ca0 msgr2=0x7f35480a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2410008001 2024-04-26T17:56:28.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.832+0000 7f3558ff9640 1 -- 172.21.15.31:0/403804234 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8182+0+0 (secure 0 0 0) 0x7f35440f0e70 con 0x7f355c102530 2024-04-26T17:56:28.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.832+0000 7f355affd640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35480a3ca0 0x7f35480a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T17:56:28.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:28.836+0000 7f3558ff9640 1 -- 172.21.15.31:0/403804234 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3544003e60 con 0x7f355c102530 2024-04-26T17:56:28.961 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:28 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1826931661' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T17:56:29.013 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:28 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1826931661' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T17:56:29.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:29.033+0000 7f355affd640 1 -- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35480a3ca0 msgr2=0x7f35480a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2410008001 2024-04-26T17:56:29.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:29.033+0000 7f355affd640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35480a3ca0 0x7f35480a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T17:56:29.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:29.046+0000 7f3561623640 1 -- 172.21.15.31:0/403804234 --> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f355c0fe6f0 con 0x7f35480a3ca0 2024-04-26T17:56:29.059 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:28 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:28.896+0000 7f519ca9f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T17:56:29.266 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:29.013+0000 7f435bc90200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T17:56:29.267 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z[34652]: 2024-04-26T17:56:29.231+0000 7f435bc90200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T17:56:29.384 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:28 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:28.962+0000 7fcb95bc2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T17:56:29.384 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y[33807]: 2024-04-26T17:56:29.177+0000 7fcb95bc2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T17:56:29.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:29.434+0000 7f355affd640 1 -- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35480a3ca0 msgr2=0x7f35480a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2410008001 2024-04-26T17:56:29.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:29.434+0000 7f355affd640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35480a3ca0 0x7f35480a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T17:56:29.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:29.697+0000 7f3558ff9640 1 -- 172.21.15.31:0/403804234 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 20) ==== 138663+0+0 (secure 0 0 0) 0x7f35440b98a0 con 0x7f355c102530 2024-04-26T17:56:29.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:29.787+0000 7f3558ff9640 1 -- 172.21.15.31:0/403804234 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7f35440c5960 con 0x7f355c102530 2024-04-26T17:56:29.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:29.788+0000 7f3558ff9640 1 -- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35480a3ca0 msgr2=0x7f35480a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T17:56:29.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:29.788+0000 7f3558ff9640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:6800/2410008001,v1:172.21.15.31:6801/2410008001] conn(0x7f35480a3ca0 0x7f35480a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:29.810 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:29.538+0000 7f519ca9f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T17:56:29.810 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:29.773+0000 7f519ca9f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T17:56:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1826931661' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T17:56:29.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:56:29.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-mon[29164]: Standby manager daemon y restarted 2024-04-26T17:56:29.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-mon[29164]: Standby manager daemon y started 2024-04-26T17:56:29.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.97:0/4102267669' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T17:56:29.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.97:0/4102267669' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T17:56:29.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.97:0/4102267669' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T17:56:29.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.97:0/4102267669' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T17:56:29.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-mon[29164]: Standby manager daemon z restarted 2024-04-26T17:56:29.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-mon[29164]: Standby manager daemon z started 2024-04-26T17:56:29.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.186:0/2789786472' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T17:56:29.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.186:0/2789786472' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T17:56:29.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.186:0/2789786472' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T17:56:29.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:29 smithi031 ceph-mon[29164]: from='mgr.? 172.21.15.186:0/2789786472' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T17:56:29.939 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1826931661' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T17:56:29.940 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:56:29.940 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-mon[32419]: Standby manager daemon y restarted 2024-04-26T17:56:29.940 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-mon[32419]: Standby manager daemon y started 2024-04-26T17:56:29.940 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.97:0/4102267669' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T17:56:29.940 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.97:0/4102267669' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T17:56:29.940 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.97:0/4102267669' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T17:56:29.940 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.97:0/4102267669' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T17:56:29.940 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-mon[32419]: Standby manager daemon z restarted 2024-04-26T17:56:29.941 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-mon[32419]: Standby manager daemon z started 2024-04-26T17:56:29.941 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.186:0/2789786472' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T17:56:29.941 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.186:0/2789786472' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T17:56:29.941 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.186:0/2789786472' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T17:56:29.941 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:29 smithi097 ceph-mon[32419]: from='mgr.? 172.21.15.186:0/2789786472' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T17:56:29.941 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1826931661' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T17:56:29.942 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:56:29.942 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-mon[32797]: Standby manager daemon y restarted 2024-04-26T17:56:29.942 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-mon[32797]: Standby manager daemon y started 2024-04-26T17:56:29.942 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.97:0/4102267669' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T17:56:29.942 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.97:0/4102267669' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T17:56:29.942 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.97:0/4102267669' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T17:56:29.943 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.97:0/4102267669' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T17:56:29.943 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-mon[32797]: Standby manager daemon z restarted 2024-04-26T17:56:29.943 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-mon[32797]: Standby manager daemon z started 2024-04-26T17:56:29.943 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.186:0/2789786472' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T17:56:29.943 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.186:0/2789786472' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T17:56:29.943 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.186:0/2789786472' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T17:56:29.944 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:29 smithi186 ceph-mon[32797]: from='mgr.? 172.21.15.186:0/2789786472' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T17:56:30.310 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:30.038+0000 7f50fec0f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:56:30.310 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:30.038+0000 7f50fec0f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:56:30.310 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:30.038+0000 7f50fec0f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:56:30.310 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:30.038+0000 7f50fec0f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:56:30.310 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:30.038+0000 7f50fec0f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:56:30.310 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:30.039+0000 7f50fa406700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:56:30.310 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:30.039+0000 7f50fa406700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:56:30.311 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:30.039+0000 7f50fa406700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:56:30.311 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:30.039+0000 7f50fa406700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:56:30.311 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:56:30.039+0000 7f50fa406700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:56:30.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.792+0000 7f3558ff9640 1 -- 172.21.15.31:0/403804234 <== mon.0 v2:172.21.15.31:3300/0 9 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f35440c2290 con 0x7f355c102530 2024-04-26T17:56:30.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.792+0000 7f3558ff9640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3548035890 0x7f3548035c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:30.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.793+0000 7f3558ff9640 1 -- 172.21.15.31:0/403804234 --> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f35440a6540 con 0x7f3548035890 2024-04-26T17:56:30.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.802+0000 7f355affd640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3548035890 0x7f3548035c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:30.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.803+0000 7f355affd640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3548035890 0x7f3548035c70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f354c00a010 tx=0x7f354c005f90 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:30.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.875+0000 7f3558ff9640 1 -- 172.21.15.31:0/403804234 <== mgr.24826 v2:172.21.15.31:6800/3595544994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f35440a6540 con 0x7f3548035890 2024-04-26T17:56:30.876 INFO:teuthology.orchestra.run.smithi031.stdout:Schedule set for path / 2024-04-26T17:56:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.877+0000 7f3561623640 1 -- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3548035890 msgr2=0x7f3548035c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.877+0000 7f3561623640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3548035890 0x7f3548035c70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f354c00a010 tx=0x7f354c005f90 comp rx=0 tx=0).stop 2024-04-26T17:56:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.877+0000 7f3561623640 1 -- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f355c102530 msgr2=0x7f355c103f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.877+0000 7f3561623640 1 --2- 172.21.15.31:0/403804234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f355c102530 0x7f355c103f60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3544005940 tx=0x7f35440042f0 comp rx=0 tx=0).stop 2024-04-26T17:56:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.878+0000 7f355b7fe640 1 -- 172.21.15.31:0/403804234 reap_dead start 2024-04-26T17:56:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.878+0000 7f3561623640 1 -- 172.21.15.31:0/403804234 shutdown_connections 2024-04-26T17:56:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.878+0000 7f3561623640 1 -- 172.21.15.31:0/403804234 >> 172.21.15.31:0/403804234 conn(0x7f355c0fc5a0 msgr2=0x7f355c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.878+0000 7f3561623640 1 -- 172.21.15.31:0/403804234 shutdown_connections 2024-04-26T17:56:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:30.878+0000 7f3561623640 1 -- 172.21.15.31:0/403804234 wait complete. 2024-04-26T17:56:30.889 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-26T17:56:30.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-26T17:56:30.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: Active manager daemon x restarted 2024-04-26T17:56:30.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: Activating manager daemon x 2024-04-26T17:56:30.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T17:56:30.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: mgrmap e21: x(active, starting, since 0.0132248s), standbys: z, y 2024-04-26T17:56:30.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:56:30.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:56:30.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:56:30.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T17:56:30.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T17:56:30.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T17:56:30.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T17:56:30.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T17:56:30.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T17:56:30.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T17:56:30.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T17:56:30.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T17:56:30.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T17:56:30.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T17:56:30.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T17:56:30.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T17:56:30.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T17:56:30.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T17:56:30.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:56:30.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:56:30.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:56:30.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:56:30.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:56:30.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:56:30.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:56:30.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:56:30.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:56:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:56:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:56:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:56:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T17:56:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T17:56:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T17:56:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: Manager daemon x is now available 2024-04-26T17:56:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:56:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T17:56:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T17:56:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: [26/Apr/2024:17:56:30] ENGINE Bus STARTING 2024-04-26T17:56:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:30 smithi031 ceph-mon[29164]: [26/Apr/2024:17:56:30] ENGINE Serving on http://172.21.15.31:8765 2024-04-26T17:56:31.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-26T17:56:31.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: Active manager daemon x restarted 2024-04-26T17:56:31.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: Activating manager daemon x 2024-04-26T17:56:31.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T17:56:31.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: mgrmap e21: x(active, starting, since 0.0132248s), standbys: z, y 2024-04-26T17:56:31.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:56:31.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:56:31.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:56:31.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T17:56:31.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T17:56:31.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T17:56:31.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T17:56:31.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T17:56:31.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T17:56:31.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T17:56:31.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T17:56:31.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T17:56:31.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T17:56:31.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T17:56:31.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T17:56:31.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T17:56:31.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T17:56:31.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T17:56:31.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:56:31.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:56:31.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:56:31.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:56:31.019 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:56:31.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:56:31.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:56:31.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:56:31.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:56:31.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:56:31.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:56:31.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:56:31.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T17:56:31.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T17:56:31.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T17:56:31.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: Manager daemon x is now available 2024-04-26T17:56:31.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:56:31.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T17:56:31.020 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T17:56:31.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: [26/Apr/2024:17:56:30] ENGINE Bus STARTING 2024-04-26T17:56:31.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:30 smithi186 ceph-mon[32797]: [26/Apr/2024:17:56:30] ENGINE Serving on http://172.21.15.31:8765 2024-04-26T17:56:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.036+0000 7f1f41758640 1 -- 172.21.15.31:0/135840536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f3c1029c0 msgr2=0x7f1f3c102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.036+0000 7f1f41758640 1 --2- 172.21.15.31:0/135840536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f3c1029c0 0x7f1f3c102de0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009900 tx=0x7f1f2c02a410 comp rx=0 tx=0).stop 2024-04-26T17:56:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.036+0000 7f1f41758640 1 -- 172.21.15.31:0/135840536 shutdown_connections 2024-04-26T17:56:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.036+0000 7f1f41758640 1 --2- 172.21.15.31:0/135840536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f3c104570 0x7f1f3c10ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.036+0000 7f1f41758640 1 --2- 172.21.15.31:0/135840536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f3c103bb0 0x7f1f3c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.036+0000 7f1f41758640 1 --2- 172.21.15.31:0/135840536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f3c1029c0 0x7f1f3c102de0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.036+0000 7f1f41758640 1 -- 172.21.15.31:0/135840536 >> 172.21.15.31:0/135840536 conn(0x7f1f3c0fdf50 msgr2=0x7f1f3c100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.036+0000 7f1f41758640 1 -- 172.21.15.31:0/135840536 shutdown_connections 2024-04-26T17:56:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.036+0000 7f1f41758640 1 -- 172.21.15.31:0/135840536 wait complete. 2024-04-26T17:56:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.036+0000 7f1f41758640 1 Processor -- start 2024-04-26T17:56:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f41758640 1 -- start start 2024-04-26T17:56:31.038 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-26T17:56:31.038 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: Active manager daemon x restarted 2024-04-26T17:56:31.038 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: Activating manager daemon x 2024-04-26T17:56:31.038 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T17:56:31.038 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: mgrmap e21: x(active, starting, since 0.0132248s), standbys: z, y 2024-04-26T17:56:31.038 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T17:56:31.038 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T17:56:31.038 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T17:56:31.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T17:56:31.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T17:56:31.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T17:56:31.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T17:56:31.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T17:56:31.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T17:56:31.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T17:56:31.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T17:56:31.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T17:56:31.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T17:56:31.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T17:56:31.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T17:56:31.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T17:56:31.040 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T17:56:31.040 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T17:56:31.040 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T17:56:31.040 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T17:56:31.040 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T17:56:31.040 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T17:56:31.040 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T17:56:31.040 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T17:56:31.040 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T17:56:31.040 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T17:56:31.040 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T17:56:31.040 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T17:56:31.040 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T17:56:31.040 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T17:56:31.041 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T17:56:31.041 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T17:56:31.041 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T17:56:31.041 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: Manager daemon x is now available 2024-04-26T17:56:31.041 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:56:31.041 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T17:56:31.041 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T17:56:31.041 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: [26/Apr/2024:17:56:30] ENGINE Bus STARTING 2024-04-26T17:56:31.041 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:30 smithi097 ceph-mon[32419]: [26/Apr/2024:17:56:30] ENGINE Serving on http://172.21.15.31:8765 2024-04-26T17:56:31.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f41758640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f3c1029c0 0x7f1f3c198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:31.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f41758640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f3c103bb0 0x7f1f3c1990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:31.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f41758640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f3c104570 0x7f1f3c1a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:31.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f41758640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3c10e4e0 con 0x7f1f3c103bb0 2024-04-26T17:56:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f41758640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3c10e360 con 0x7f1f3c1029c0 2024-04-26T17:56:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f41758640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3c10e660 con 0x7f1f3c104570 2024-04-26T17:56:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f3a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f3c103bb0 0x7f1f3c1990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f3a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f3c103bb0 0x7f1f3c1990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50372/0 (socket says 172.21.15.31:50372) 2024-04-26T17:56:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f3a7fc640 1 -- 172.21.15.31:0/1925659113 learned_addr learned my addr 172.21.15.31:0/1925659113 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f3affd640 1 --2- 172.21.15.31:0/1925659113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f3c1029c0 0x7f1f3c198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f3b7fe640 1 --2- 172.21.15.31:0/1925659113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f3c104570 0x7f1f3c1a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f3a7fc640 1 -- 172.21.15.31:0/1925659113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f3c104570 msgr2=0x7f1f3c1a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f3a7fc640 1 --2- 172.21.15.31:0/1925659113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f3c104570 0x7f1f3c1a0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f3a7fc640 1 -- 172.21.15.31:0/1925659113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f3c1029c0 msgr2=0x7f1f3c198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f3a7fc640 1 --2- 172.21.15.31:0/1925659113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f3c1029c0 0x7f1f3c198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f3a7fc640 1 -- 172.21.15.31:0/1925659113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f3c1a0780 con 0x7f1f3c103bb0 2024-04-26T17:56:31.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f3a7fc640 1 --2- 172.21.15.31:0/1925659113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f3c103bb0 0x7f1f3c1990a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1f3000e9f0 tx=0x7f1f3000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.037+0000 7f1f23fff640 1 -- 172.21.15.31:0/1925659113 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3000cd80 con 0x7f1f3c103bb0 2024-04-26T17:56:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.038+0000 7f1f41758640 1 -- 172.21.15.31:0/1925659113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f3c1a0a70 con 0x7f1f3c103bb0 2024-04-26T17:56:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.038+0000 7f1f23fff640 1 -- 172.21.15.31:0/1925659113 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f300054d0 con 0x7f1f3c103bb0 2024-04-26T17:56:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.038+0000 7f1f23fff640 1 -- 172.21.15.31:0/1925659113 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30010690 con 0x7f1f3c103bb0 2024-04-26T17:56:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.038+0000 7f1f41758640 1 -- 172.21.15.31:0/1925659113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f3c1a0fc0 con 0x7f1f3c103bb0 2024-04-26T17:56:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.039+0000 7f1f41758640 1 -- 172.21.15.31:0/1925659113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f3c109ce0 con 0x7f1f3c103bb0 2024-04-26T17:56:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.040+0000 7f1f23fff640 1 -- 172.21.15.31:0/1925659113 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f1f30004e30 con 0x7f1f3c103bb0 2024-04-26T17:56:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.040+0000 7f1f23fff640 1 --2- 172.21.15.31:0/1925659113 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1f240a3a60 0x7f1f240a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.041+0000 7f1f3affd640 1 --2- 172.21.15.31:0/1925659113 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1f240a3a60 0x7f1f240a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.041+0000 7f1f23fff640 1 -- 172.21.15.31:0/1925659113 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f1f300d0720 con 0x7f1f3c103bb0 2024-04-26T17:56:31.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.041+0000 7f1f3affd640 1 --2- 172.21.15.31:0/1925659113 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1f240a3a60 0x7f1f240a5f20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009230 tx=0x7f1f2c0314a0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:31.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.045+0000 7f1f23fff640 1 -- 172.21.15.31:0/1925659113 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f30053ff0 con 0x7f1f3c103bb0 2024-04-26T17:56:31.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.270+0000 7f1f41758640 1 -- 172.21.15.31:0/1925659113 --> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f1f3c062e10 con 0x7f1f240a3a60 2024-04-26T17:56:31.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.330+0000 7f1f23fff640 1 -- 172.21.15.31:0/1925659113 <== mgr.24826 v2:172.21.15.31:6800/3595544994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f1f3c062e10 con 0x7f1f240a3a60 2024-04-26T17:56:31.330 INFO:teuthology.orchestra.run.smithi031.stdout:Retention added to path / 2024-04-26T17:56:31.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.332+0000 7f1f41758640 1 -- 172.21.15.31:0/1925659113 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1f240a3a60 msgr2=0x7f1f240a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:31.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.332+0000 7f1f41758640 1 --2- 172.21.15.31:0/1925659113 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1f240a3a60 0x7f1f240a5f20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009230 tx=0x7f1f2c0314a0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.332+0000 7f1f41758640 1 -- 172.21.15.31:0/1925659113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f3c103bb0 msgr2=0x7f1f3c1990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:31.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.332+0000 7f1f41758640 1 --2- 172.21.15.31:0/1925659113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f3c103bb0 0x7f1f3c1990a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1f3000e9f0 tx=0x7f1f3000edf0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.332+0000 7f1f41758640 1 -- 172.21.15.31:0/1925659113 shutdown_connections 2024-04-26T17:56:31.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.332+0000 7f1f41758640 1 --2- 172.21.15.31:0/1925659113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f3c104570 0x7f1f3c1a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.332+0000 7f1f41758640 1 --2- 172.21.15.31:0/1925659113 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1f240a3a60 0x7f1f240a5f20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.332+0000 7f1f41758640 1 --2- 172.21.15.31:0/1925659113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f3c103bb0 0x7f1f3c1990a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.332+0000 7f1f41758640 1 --2- 172.21.15.31:0/1925659113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f3c1029c0 0x7f1f3c198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.332+0000 7f1f41758640 1 -- 172.21.15.31:0/1925659113 >> 172.21.15.31:0/1925659113 conn(0x7f1f3c0fdf50 msgr2=0x7f1f3c106eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:31.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.332+0000 7f1f41758640 1 -- 172.21.15.31:0/1925659113 shutdown_connections 2024-04-26T17:56:31.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.332+0000 7f1f41758640 1 -- 172.21.15.31:0/1925659113 wait complete. 2024-04-26T17:56:31.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-26T17:56:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.487+0000 7f5754f51640 1 -- 172.21.15.31:0/3169412392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5750103b80 msgr2=0x7f5750104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.487+0000 7f5754f51640 1 --2- 172.21.15.31:0/3169412392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5750103b80 0x7f5750104000 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5740009900 tx=0x7f574002a410 comp rx=0 tx=0).stop 2024-04-26T17:56:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.487+0000 7f5754f51640 1 -- 172.21.15.31:0/3169412392 shutdown_connections 2024-04-26T17:56:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.487+0000 7f5754f51640 1 --2- 172.21.15.31:0/3169412392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5750104540 0x7f575010add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.487+0000 7f5754f51640 1 --2- 172.21.15.31:0/3169412392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5750103b80 0x7f5750104000 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.487+0000 7f5754f51640 1 --2- 172.21.15.31:0/3169412392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5750102990 0x7f5750102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.487+0000 7f5754f51640 1 -- 172.21.15.31:0/3169412392 >> 172.21.15.31:0/3169412392 conn(0x7f57500fdf20 msgr2=0x7f5750100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:31.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.487+0000 7f5754f51640 1 -- 172.21.15.31:0/3169412392 shutdown_connections 2024-04-26T17:56:31.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.487+0000 7f5754f51640 1 -- 172.21.15.31:0/3169412392 wait complete. 2024-04-26T17:56:31.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f5754f51640 1 Processor -- start 2024-04-26T17:56:31.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f5754f51640 1 -- start start 2024-04-26T17:56:31.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f5754f51640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5750102990 0x7f5750198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:31.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f5754f51640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5750103b80 0x7f5750199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:31.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f5754f51640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5750104540 0x7f57501a00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:31.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f5754f51640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f575010e4c0 con 0x7f5750104540 2024-04-26T17:56:31.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f5754f51640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f575010e340 con 0x7f5750103b80 2024-04-26T17:56:31.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f5754f51640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f575010e640 con 0x7f5750102990 2024-04-26T17:56:31.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f574ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5750104540 0x7f57501a00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:31.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f574ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5750104540 0x7f57501a00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50400/0 (socket says 172.21.15.31:50400) 2024-04-26T17:56:31.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f574ed9d640 1 -- 172.21.15.31:0/1248221378 learned_addr learned my addr 172.21.15.31:0/1248221378 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:31.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f574e59c640 1 --2- 172.21.15.31:0/1248221378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5750102990 0x7f5750198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:31.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f574dd9b640 1 --2- 172.21.15.31:0/1248221378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5750103b80 0x7f5750199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:31.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f574ed9d640 1 -- 172.21.15.31:0/1248221378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5750102990 msgr2=0x7f5750198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:31.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f574ed9d640 1 --2- 172.21.15.31:0/1248221378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5750102990 0x7f5750198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f574ed9d640 1 -- 172.21.15.31:0/1248221378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5750103b80 msgr2=0x7f5750199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:31.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f574ed9d640 1 --2- 172.21.15.31:0/1248221378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5750103b80 0x7f5750199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.488+0000 7f574ed9d640 1 -- 172.21.15.31:0/1248221378 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57501a0730 con 0x7f5750104540 2024-04-26T17:56:31.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.489+0000 7f574ed9d640 1 --2- 172.21.15.31:0/1248221378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5750104540 0x7f57501a00c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f574800eaa0 tx=0x7f574800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:31.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.489+0000 7f573f7fe640 1 -- 172.21.15.31:0/1248221378 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574800cdb0 con 0x7f5750104540 2024-04-26T17:56:31.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.489+0000 7f573f7fe640 1 -- 172.21.15.31:0/1248221378 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5748005020 con 0x7f5750104540 2024-04-26T17:56:31.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.489+0000 7f5754f51640 1 -- 172.21.15.31:0/1248221378 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57501a0a20 con 0x7f5750104540 2024-04-26T17:56:31.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.489+0000 7f573f7fe640 1 -- 172.21.15.31:0/1248221378 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57480116d0 con 0x7f5750104540 2024-04-26T17:56:31.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.489+0000 7f5754f51640 1 -- 172.21.15.31:0/1248221378 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57501a0f40 con 0x7f5750104540 2024-04-26T17:56:31.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.490+0000 7f5754f51640 1 -- 172.21.15.31:0/1248221378 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5750109cb0 con 0x7f5750104540 2024-04-26T17:56:31.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.491+0000 7f573f7fe640 1 -- 172.21.15.31:0/1248221378 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f57480051d0 con 0x7f5750104540 2024-04-26T17:56:31.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.491+0000 7f573f7fe640 1 --2- 172.21.15.31:0/1248221378 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f57380a3ae0 0x7f57380a5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:31.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.492+0000 7f574e59c640 1 --2- 172.21.15.31:0/1248221378 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f57380a3ae0 0x7f57380a5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:31.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.492+0000 7f573f7fe640 1 -- 172.21.15.31:0/1248221378 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f57480d01d0 con 0x7f5750104540 2024-04-26T17:56:31.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.492+0000 7f574e59c640 1 --2- 172.21.15.31:0/1248221378 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f57380a3ae0 0x7f57380a5fa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5744005fd0 tx=0x7f574400b390 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:31.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.496+0000 7f573f7fe640 1 -- 172.21.15.31:0/1248221378 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5748016030 con 0x7f5750104540 2024-04-26T17:56:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.727+0000 7f5754f51640 1 -- 172.21.15.31:0/1248221378 --> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f5750062d70 con 0x7f57380a3ae0 2024-04-26T17:56:31.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.736+0000 7f573f7fe640 1 -- 172.21.15.31:0/1248221378 <== mgr.24826 v2:172.21.15.31:6800/3595544994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f5750062d70 con 0x7f57380a3ae0 2024-04-26T17:56:31.736 INFO:teuthology.orchestra.run.smithi031.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-26T00:00:00", "created": "2024-04-26T17:56:30", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-26T17:56:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.738+0000 7f5754f51640 1 -- 172.21.15.31:0/1248221378 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f57380a3ae0 msgr2=0x7f57380a5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.738+0000 7f5754f51640 1 --2- 172.21.15.31:0/1248221378 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f57380a3ae0 0x7f57380a5fa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5744005fd0 tx=0x7f574400b390 comp rx=0 tx=0).stop 2024-04-26T17:56:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.738+0000 7f5754f51640 1 -- 172.21.15.31:0/1248221378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5750104540 msgr2=0x7f57501a00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.738+0000 7f5754f51640 1 --2- 172.21.15.31:0/1248221378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5750104540 0x7f57501a00c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f574800eaa0 tx=0x7f574800ef60 comp rx=0 tx=0).stop 2024-04-26T17:56:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.738+0000 7f5754f51640 1 -- 172.21.15.31:0/1248221378 shutdown_connections 2024-04-26T17:56:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.738+0000 7f5754f51640 1 --2- 172.21.15.31:0/1248221378 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f57380a3ae0 0x7f57380a5fa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.738+0000 7f5754f51640 1 --2- 172.21.15.31:0/1248221378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5750104540 0x7f57501a00c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.738+0000 7f5754f51640 1 --2- 172.21.15.31:0/1248221378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5750103b80 0x7f5750199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.738+0000 7f5754f51640 1 --2- 172.21.15.31:0/1248221378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5750102990 0x7f5750198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.738+0000 7f5754f51640 1 -- 172.21.15.31:0/1248221378 >> 172.21.15.31:0/1248221378 conn(0x7f57500fdf20 msgr2=0x7f5750106e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.738+0000 7f5754f51640 1 -- 172.21.15.31:0/1248221378 shutdown_connections 2024-04-26T17:56:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.738+0000 7f5754f51640 1 -- 172.21.15.31:0/1248221378 wait complete. 2024-04-26T17:56:31.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-26T17:56:31.808 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:31 smithi186 ceph-mon[32797]: [26/Apr/2024:17:56:30] ENGINE Serving on https://172.21.15.31:7150 2024-04-26T17:56:31.808 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:31 smithi186 ceph-mon[32797]: [26/Apr/2024:17:56:30] ENGINE Bus STARTED 2024-04-26T17:56:31.809 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:31 smithi186 ceph-mon[32797]: mgrmap e22: x(active, since 1.02084s), standbys: z, y 2024-04-26T17:56:31.809 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:31 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:31.809 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:31 smithi186 ceph-mon[32797]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:31.809 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:31 smithi186 ceph-mon[32797]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:56:31.809 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:31 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:31.809 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:31 smithi186 ceph-mon[32797]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:31.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.880+0000 7f2adda2c640 1 -- 172.21.15.31:0/4229041651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad8104570 msgr2=0x7f2ad810ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:31.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.880+0000 7f2adda2c640 1 --2- 172.21.15.31:0/4229041651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad8104570 0x7f2ad810ae00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0009be0 tx=0x7f2ad002a6c0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.880+0000 7f2adda2c640 1 -- 172.21.15.31:0/4229041651 shutdown_connections 2024-04-26T17:56:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.880+0000 7f2adda2c640 1 --2- 172.21.15.31:0/4229041651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad8104570 0x7f2ad810ae00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.880+0000 7f2adda2c640 1 --2- 172.21.15.31:0/4229041651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ad8103bb0 0x7f2ad8104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.880+0000 7f2adda2c640 1 --2- 172.21.15.31:0/4229041651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ad81029c0 0x7f2ad8102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.880+0000 7f2adda2c640 1 -- 172.21.15.31:0/4229041651 >> 172.21.15.31:0/4229041651 conn(0x7f2ad80fdf50 msgr2=0x7f2ad8100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.880+0000 7f2adda2c640 1 -- 172.21.15.31:0/4229041651 shutdown_connections 2024-04-26T17:56:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.880+0000 7f2adda2c640 1 -- 172.21.15.31:0/4229041651 wait complete. 2024-04-26T17:56:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2adda2c640 1 Processor -- start 2024-04-26T17:56:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2adda2c640 1 -- start start 2024-04-26T17:56:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2adda2c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ad81029c0 0x7f2ad8072500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2adda2c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad8103bb0 0x7f2ad8072a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2adda2c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ad8104570 0x7f2ad8073a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2adda2c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad806cc50 con 0x7f2ad8103bb0 2024-04-26T17:56:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2adda2c640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad806cad0 con 0x7f2ad81029c0 2024-04-26T17:56:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2adda2c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad806cdd0 con 0x7f2ad8104570 2024-04-26T17:56:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2ad6ffd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ad81029c0 0x7f2ad8072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2acffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad8103bb0 0x7f2ad8072a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2ad6ffd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ad81029c0 0x7f2ad8072500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:60140/0 (socket says 172.21.15.31:60140) 2024-04-26T17:56:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2ad6ffd640 1 -- 172.21.15.31:0/2604481919 learned_addr learned my addr 172.21.15.31:0/2604481919 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2ad77fe640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ad8104570 0x7f2ad8073a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2ad6ffd640 1 -- 172.21.15.31:0/2604481919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ad8104570 msgr2=0x7f2ad8073a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2ad6ffd640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ad8104570 0x7f2ad8073a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2ad6ffd640 1 -- 172.21.15.31:0/2604481919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad8103bb0 msgr2=0x7f2ad8072a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2ad6ffd640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad8103bb0 0x7f2ad8072a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2ad6ffd640 1 -- 172.21.15.31:0/2604481919 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad80740f0 con 0x7f2ad81029c0 2024-04-26T17:56:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2ad77fe640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ad8104570 0x7f2ad8073a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:56:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.881+0000 7f2acffff640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad8103bb0 0x7f2ad8072a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:56:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.882+0000 7f2ad6ffd640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ad81029c0 0x7f2ad8072500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800e9c0 tx=0x7f2ac800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.882+0000 7f2adca2a640 1 -- 172.21.15.31:0/2604481919 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac800cd80 con 0x7f2ad81029c0 2024-04-26T17:56:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.882+0000 7f2adca2a640 1 -- 172.21.15.31:0/2604481919 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ac80108e0 con 0x7f2ad81029c0 2024-04-26T17:56:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.882+0000 7f2adda2c640 1 -- 172.21.15.31:0/2604481919 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad80743e0 con 0x7f2ad81029c0 2024-04-26T17:56:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.882+0000 7f2adca2a640 1 -- 172.21.15.31:0/2604481919 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8018840 con 0x7f2ad81029c0 2024-04-26T17:56:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.882+0000 7f2adda2c640 1 -- 172.21.15.31:0/2604481919 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad8074930 con 0x7f2ad81029c0 2024-04-26T17:56:31.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.883+0000 7f2adda2c640 1 -- 172.21.15.31:0/2604481919 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ad8109c80 con 0x7f2ad81029c0 2024-04-26T17:56:31.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.884+0000 7f2adca2a640 1 -- 172.21.15.31:0/2604481919 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f2ac801f020 con 0x7f2ad81029c0 2024-04-26T17:56:31.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.885+0000 7f2adca2a640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2ac40a3dc0 0x7f2ac40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:31.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.885+0000 7f2acffff640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2ac40a3dc0 0x7f2ac40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:31.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.885+0000 7f2adca2a640 1 -- 172.21.15.31:0/2604481919 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f2ac80d0840 con 0x7f2ad81029c0 2024-04-26T17:56:31.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.885+0000 7f2acffff640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2ac40a3dc0 0x7f2ac40a6280 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8073a10 tx=0x7f2ac0009260 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:31.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:31.891+0000 7f2adca2a640 1 -- 172.21.15.31:0/2604481919 <== mon.2 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ac809cf10 con 0x7f2ad81029c0 2024-04-26T17:56:32.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:31 smithi031 ceph-mon[29164]: [26/Apr/2024:17:56:30] ENGINE Serving on https://172.21.15.31:7150 2024-04-26T17:56:32.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:31 smithi031 ceph-mon[29164]: [26/Apr/2024:17:56:30] ENGINE Bus STARTED 2024-04-26T17:56:32.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:31 smithi031 ceph-mon[29164]: mgrmap e22: x(active, since 1.02084s), standbys: z, y 2024-04-26T17:56:32.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:31 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:32.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:31 smithi031 ceph-mon[29164]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:32.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:31 smithi031 ceph-mon[29164]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:56:32.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:31 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:32.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:31 smithi031 ceph-mon[29164]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:32.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:32.134+0000 7f2adda2c640 1 -- 172.21.15.31:0/2604481919 --> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f2ad8062d70 con 0x7f2ac40a3dc0 2024-04-26T17:56:32.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:31 smithi097 ceph-mon[32419]: [26/Apr/2024:17:56:30] ENGINE Serving on https://172.21.15.31:7150 2024-04-26T17:56:32.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:31 smithi097 ceph-mon[32419]: [26/Apr/2024:17:56:30] ENGINE Bus STARTED 2024-04-26T17:56:32.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:31 smithi097 ceph-mon[32419]: mgrmap e22: x(active, since 1.02084s), standbys: z, y 2024-04-26T17:56:32.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:31 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:32.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:31 smithi097 ceph-mon[32419]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:32.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:31 smithi097 ceph-mon[32419]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:56:32.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:31 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:32.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:31 smithi097 ceph-mon[32419]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:32.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:32.141+0000 7f2adca2a640 1 -- 172.21.15.31:0/2604481919 <== mgr.24826 v2:172.21.15.31:6800/3595544994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f2ad8062d70 con 0x7f2ac40a3dc0 2024-04-26T17:56:32.141 INFO:teuthology.orchestra.run.smithi031.stdout:/ 1m 6m3h 2024-04-26T17:56:32.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:32.143+0000 7f2adda2c640 1 -- 172.21.15.31:0/2604481919 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2ac40a3dc0 msgr2=0x7f2ac40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:32.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:32.143+0000 7f2adda2c640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2ac40a3dc0 0x7f2ac40a6280 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8073a10 tx=0x7f2ac0009260 comp rx=0 tx=0).stop 2024-04-26T17:56:32.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:32.143+0000 7f2adda2c640 1 -- 172.21.15.31:0/2604481919 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ad81029c0 msgr2=0x7f2ad8072500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:32.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:32.143+0000 7f2adda2c640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ad81029c0 0x7f2ad8072500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800e9c0 tx=0x7f2ac800edc0 comp rx=0 tx=0).stop 2024-04-26T17:56:32.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:32.144+0000 7f2adda2c640 1 -- 172.21.15.31:0/2604481919 shutdown_connections 2024-04-26T17:56:32.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:32.144+0000 7f2adda2c640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ad8104570 0x7f2ad8073a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:32.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:32.144+0000 7f2adda2c640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2ac40a3dc0 0x7f2ac40a6280 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:32.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:32.144+0000 7f2adda2c640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad8103bb0 0x7f2ad8072a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:32.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:32.144+0000 7f2adda2c640 1 --2- 172.21.15.31:0/2604481919 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ad81029c0 0x7f2ad8072500 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:32.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:32.144+0000 7f2adda2c640 1 -- 172.21.15.31:0/2604481919 >> 172.21.15.31:0/2604481919 conn(0x7f2ad80fdf50 msgr2=0x7f2ad8106e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:32.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:32.144+0000 7f2adda2c640 1 -- 172.21.15.31:0/2604481919 shutdown_connections 2024-04-26T17:56:32.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:32.144+0000 7f2adda2c640 1 -- 172.21.15.31:0/2604481919 wait complete. 2024-04-26T17:56:32.155 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-26T17:56:32.181 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-26T17:56:32.193 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-26T17:56:32.213 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-26T17:56:32.214 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-26T17:56:32.224 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-26T17:56:32.238 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T17:56:32.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T17:56:32.499 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:32.878 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:32 smithi031 ceph-mon[29164]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:32.878 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:32 smithi031 ceph-mon[29164]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:56:32.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:32 smithi031 ceph-mon[29164]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-26T17:56:32.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:32 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:32.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:32 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:32.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:32 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:32.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:32 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:32.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:32 smithi031 ceph-mon[29164]: from='client.24956 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:32.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:32 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:32.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:32 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:33.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:32 smithi097 ceph-mon[32419]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:33.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:32 smithi097 ceph-mon[32419]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:56:33.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:32 smithi097 ceph-mon[32419]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-26T17:56:33.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:32 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:33.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:32 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:33.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:32 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:33.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:32 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:33.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:32 smithi097 ceph-mon[32419]: from='client.24956 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:33.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:32 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:33.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:32 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:33.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:32 smithi186 ceph-mon[32797]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:33.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:32 smithi186 ceph-mon[32797]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:56:33.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:32 smithi186 ceph-mon[32797]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-26T17:56:33.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:32 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:33.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:32 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:33.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:32 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:33.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:32 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:33.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:32 smithi186 ceph-mon[32797]: from='client.24956 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T17:56:33.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:32 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:33.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:32 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.221 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.221 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.221 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.222 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.222 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.222 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.222 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi097 to 176.2M 2024-04-26T17:56:34.222 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: Unable to set osd_memory_target on smithi097 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-04-26T17:56:34.222 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.222 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.222 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.222 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.222 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.222 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.222 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.223 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.223 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.223 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.223 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.223 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.223 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:56:34.223 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:56:34.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.220+0000 7f2ede01f700 1 -- 172.21.15.31:0/1332516760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed81094b0 msgr2=0x7f2ed8109890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:34.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.220+0000 7f2ede01f700 1 --2- 172.21.15.31:0/1332516760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed81094b0 0x7f2ed8109890 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009990 tx=0x7f2ec8009ca0 comp rx=0 tx=0).stop 2024-04-26T17:56:34.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.220+0000 7f2ede01f700 1 -- 172.21.15.31:0/1332516760 shutdown_connections 2024-04-26T17:56:34.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.220+0000 7f2ede01f700 1 --2- 172.21.15.31:0/1332516760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed81023b0 0x7f2ed81102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:34.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.220+0000 7f2ede01f700 1 --2- 172.21.15.31:0/1332516760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed81094b0 0x7f2ed8109890 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:34.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.220+0000 7f2ede01f700 1 --2- 172.21.15.31:0/1332516760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ed8101a10 0x7f2ed8101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:34.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.220+0000 7f2ede01f700 1 -- 172.21.15.31:0/1332516760 >> 172.21.15.31:0/1332516760 conn(0x7f2ed8078990 msgr2=0x7f2ed80ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:34.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.220+0000 7f2ede01f700 1 -- 172.21.15.31:0/1332516760 shutdown_connections 2024-04-26T17:56:34.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.220+0000 7f2ede01f700 1 -- 172.21.15.31:0/1332516760 wait complete. 2024-04-26T17:56:34.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.221+0000 7f2ede01f700 1 Processor -- start 2024-04-26T17:56:34.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.221+0000 7f2ede01f700 1 -- start start 2024-04-26T17:56:34.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ede01f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed8101a10 0x7f2ed819f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:34.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ede01f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed81023b0 0x7f2ed819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:34.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ede01f700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ed81094b0 0x7f2ed8199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:34.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ede01f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed81135a0 con 0x7f2ed8101a10 2024-04-26T17:56:34.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ede01f700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed8113420 con 0x7f2ed81094b0 2024-04-26T17:56:34.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ede01f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed8113720 con 0x7f2ed81023b0 2024-04-26T17:56:34.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed8101a10 0x7f2ed819f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:34.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed8101a10 0x7f2ed819f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50452/0 (socket says 172.21.15.31:50452) 2024-04-26T17:56:34.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ed77fe700 1 -- 172.21.15.31:0/459622800 learned_addr learned my addr 172.21.15.31:0/459622800 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:34.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ecffff700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed81023b0 0x7f2ed819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:34.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ed7fff700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ed81094b0 0x7f2ed8199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:34.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ed77fe700 1 -- 172.21.15.31:0/459622800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed81023b0 msgr2=0x7f2ed819fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:34.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ed77fe700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed81023b0 0x7f2ed819fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:34.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ed77fe700 1 -- 172.21.15.31:0/459622800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ed81094b0 msgr2=0x7f2ed8199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:34.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ed77fe700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ed81094b0 0x7f2ed8199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:34.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ed77fe700 1 -- 172.21.15.31:0/459622800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ed819a060 con 0x7f2ed8101a10 2024-04-26T17:56:34.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ecffff700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed81023b0 0x7f2ed819fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:56:34.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ed7fff700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ed81094b0 0x7f2ed8199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:56:34.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.222+0000 7f2ed77fe700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed8101a10 0x7f2ed819f740 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80095c0 tx=0x7f2ec8005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:34.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.223+0000 7f2ed57fa700 1 -- 172.21.15.31:0/459622800 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec802f070 con 0x7f2ed8101a10 2024-04-26T17:56:34.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.223+0000 7f2ede01f700 1 -- 172.21.15.31:0/459622800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ed819a2f0 con 0x7f2ed8101a10 2024-04-26T17:56:34.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.223+0000 7f2ede01f700 1 -- 172.21.15.31:0/459622800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ed8102b20 con 0x7f2ed8101a10 2024-04-26T17:56:34.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.223+0000 7f2ed57fa700 1 -- 172.21.15.31:0/459622800 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ec800be70 con 0x7f2ed8101a10 2024-04-26T17:56:34.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.223+0000 7f2ed57fa700 1 -- 172.21.15.31:0/459622800 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec802a8f0 con 0x7f2ed8101a10 2024-04-26T17:56:34.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.224+0000 7f2ede01f700 1 -- 172.21.15.31:0/459622800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eb8005330 con 0x7f2ed8101a10 2024-04-26T17:56:34.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.225+0000 7f2ed57fa700 1 -- 172.21.15.31:0/459622800 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f2ec8032070 con 0x7f2ed8101a10 2024-04-26T17:56:34.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.226+0000 7f2ed57fa700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2ec40a3eb0 0x7f2ec40a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:34.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.226+0000 7f2ecffff700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2ec40a3eb0 0x7f2ec40a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:34.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.226+0000 7f2ed57fa700 1 -- 172.21.15.31:0/459622800 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f2ec80d9bd0 con 0x7f2ed8101a10 2024-04-26T17:56:34.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.227+0000 7f2ecffff700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2ec40a3eb0 0x7f2ec40a6370 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8101060 tx=0x7f2ec0012040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:34.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.231+0000 7f2ed57fa700 1 -- 172.21.15.31:0/459622800 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ec8066ff0 con 0x7f2ed8101a10 2024-04-26T17:56:34.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi097 to 176.2M 2024-04-26T17:56:34.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: Unable to set osd_memory_target on smithi097 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-04-26T17:56:34.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:56:34.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:56:34.312 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.312 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.312 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.313 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.313 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.313 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.313 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi097 to 176.2M 2024-04-26T17:56:34.313 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: Unable to set osd_memory_target on smithi097 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-04-26T17:56:34.313 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.313 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.313 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.313 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.313 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.313 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.313 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.313 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:34.313 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.314 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.314 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.314 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T17:56:34.314 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:56:34.314 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:56:34.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.441+0000 7f2ede01f700 1 -- 172.21.15.31:0/459622800 --> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2eb8000bc0 con 0x7f2ec40a3eb0 2024-04-26T17:56:34.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.449+0000 7f2ed57fa700 1 -- 172.21.15.31:0/459622800 <== mgr.24826 v2:172.21.15.31:6800/3595544994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257986 (secure 0 0 0) 0x7f2eb8000bc0 con 0x7f2ec40a3eb0 2024-04-26T17:56:34.453 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.452+0000 7f2ede01f700 1 -- 172.21.15.31:0/459622800 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2ec40a3eb0 msgr2=0x7f2ec40a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.452+0000 7f2ede01f700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2ec40a3eb0 0x7f2ec40a6370 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8101060 tx=0x7f2ec0012040 comp rx=0 tx=0).stop 2024-04-26T17:56:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.452+0000 7f2ede01f700 1 -- 172.21.15.31:0/459622800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed8101a10 msgr2=0x7f2ed819f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.452+0000 7f2ede01f700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed8101a10 0x7f2ed819f740 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80095c0 tx=0x7f2ec8005ef0 comp rx=0 tx=0).stop 2024-04-26T17:56:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.452+0000 7f2ede01f700 1 -- 172.21.15.31:0/459622800 shutdown_connections 2024-04-26T17:56:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.452+0000 7f2ede01f700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2ec40a3eb0 0x7f2ec40a6370 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.452+0000 7f2ede01f700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ed81094b0 0x7f2ed8199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.452+0000 7f2ede01f700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed8101a10 0x7f2ed819f740 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.452+0000 7f2ede01f700 1 --2- 172.21.15.31:0/459622800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed81023b0 0x7f2ed819fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.452+0000 7f2ede01f700 1 -- 172.21.15.31:0/459622800 >> 172.21.15.31:0/459622800 conn(0x7f2ed8078990 msgr2=0x7f2ed8106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.452+0000 7f2ede01f700 1 -- 172.21.15.31:0/459622800 shutdown_connections 2024-04-26T17:56:34.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:34.452+0000 7f2ede01f700 1 -- 172.21.15.31:0/459622800 wait complete. 2024-04-26T17:56:34.456 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-26T17:56:35.121 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T17:56:35.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: Unable to set osd_memory_target on smithi186 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-26T17:56:35.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:56:35.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-26T17:56:35.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T17:56:35.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-26T17:56:35.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-26T17:56:35.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T17:56:35.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:56:35.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:56:35.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: Updating smithi031:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:56:35.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:56:35.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:56:35.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:56:35.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: from='client.15234 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:56:35.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-26T17:56:35.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:35 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.148 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-26T17:56:33.798820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188638,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1388,"num_read_kb":850,"num_write":478,"num_write_kb":4380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365568,"kb_used_data":12224,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576256,"statfs":{"total":1151940427776,"available":1151566086144,"internally_reserved":0,"allocated":12517376,"data_stored":8869744,"data_compressed":186586,"data_compressed_allocated":241664,"data_compressed_original":2883584,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.991832"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627925+0000","last_change":"2024-04-26T17:53:03.616592+0000","last_active":"2024-04-26T17:56:30.627925+0000","last_peered":"2024-04-26T17:56:30.627925+0000","last_clean":"2024-04-26T17:56:30.627925+0000","last_became_active":"2024-04-26T17:53:03.616255+0000","last_became_peered":"2024-04-26T17:53:03.616255+0000","last_unstale":"2024-04-26T17:56:30.627925+0000","last_undegraded":"2024-04-26T17:56:30.627925+0000","last_fullsized":"2024-04-26T17:56:30.627925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:43:31.183679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472977+0000","last_change":"2024-04-26T17:53:03.615402+0000","last_active":"2024-04-26T17:56:30.472977+0000","last_peered":"2024-04-26T17:56:30.472977+0000","last_clean":"2024-04-26T17:56:30.472977+0000","last_became_active":"2024-04-26T17:53:03.615048+0000","last_became_peered":"2024-04-26T17:53:03.615048+0000","last_unstale":"2024-04-26T17:56:30.472977+0000","last_undegraded":"2024-04-26T17:56:30.472977+0000","last_fullsized":"2024-04-26T17:56:30.472977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:05:12.454770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472800+0000","last_change":"2024-04-26T17:53:03.622625+0000","last_active":"2024-04-26T17:56:30.472800+0000","last_peered":"2024-04-26T17:56:30.472800+0000","last_clean":"2024-04-26T17:56:30.472800+0000","last_became_active":"2024-04-26T17:53:03.622305+0000","last_became_peered":"2024-04-26T17:53:03.622305+0000","last_unstale":"2024-04-26T17:56:30.472800+0000","last_undegraded":"2024-04-26T17:56:30.472800+0000","last_fullsized":"2024-04-26T17:56:30.472800+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:06.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.784761+0000","last_change":"2024-04-26T17:53:03.617650+0000","last_active":"2024-04-26T17:56:29.784761+0000","last_peered":"2024-04-26T17:56:29.784761+0000","last_clean":"2024-04-26T17:56:29.784761+0000","last_became_active":"2024-04-26T17:53:03.617220+0000","last_became_peered":"2024-04-26T17:53:03.617220+0000","last_unstale":"2024-04-26T17:56:29.784761+0000","last_undegraded":"2024-04-26T17:56:29.784761+0000","last_fullsized":"2024-04-26T17:56:29.784761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:31.765577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882813+0000","last_change":"2024-04-26T17:53:03.615429+0000","last_active":"2024-04-26T17:56:29.882813+0000","last_peered":"2024-04-26T17:56:29.882813+0000","last_clean":"2024-04-26T17:56:29.882813+0000","last_became_active":"2024-04-26T17:53:03.615079+0000","last_became_peered":"2024-04-26T17:53:03.615079+0000","last_unstale":"2024-04-26T17:56:29.882813+0000","last_undegraded":"2024-04-26T17:56:29.882813+0000","last_fullsized":"2024-04-26T17:56:29.882813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:20.994016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894485+0000","last_change":"2024-04-26T17:53:03.624619+0000","last_active":"2024-04-26T17:56:29.894485+0000","last_peered":"2024-04-26T17:56:29.894485+0000","last_clean":"2024-04-26T17:56:29.894485+0000","last_became_active":"2024-04-26T17:53:03.624251+0000","last_became_peered":"2024-04-26T17:53:03.624251+0000","last_unstale":"2024-04-26T17:56:29.894485+0000","last_undegraded":"2024-04-26T17:56:29.894485+0000","last_fullsized":"2024-04-26T17:56:29.894485+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:50:17.535235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046789+0000","last_change":"2024-04-26T17:53:03.619325+0000","last_active":"2024-04-26T17:56:30.046789+0000","last_peered":"2024-04-26T17:56:30.046789+0000","last_clean":"2024-04-26T17:56:30.046789+0000","last_became_active":"2024-04-26T17:53:03.618989+0000","last_became_peered":"2024-04-26T17:53:03.618989+0000","last_unstale":"2024-04-26T17:56:30.046789+0000","last_undegraded":"2024-04-26T17:56:30.046789+0000","last_fullsized":"2024-04-26T17:56:30.046789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:10.195742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882506+0000","last_change":"2024-04-26T17:53:03.609346+0000","last_active":"2024-04-26T17:56:29.882506+0000","last_peered":"2024-04-26T17:56:29.882506+0000","last_clean":"2024-04-26T17:56:29.882506+0000","last_became_active":"2024-04-26T17:53:03.609021+0000","last_became_peered":"2024-04-26T17:53:03.609021+0000","last_unstale":"2024-04-26T17:56:29.882506+0000","last_undegraded":"2024-04-26T17:56:29.882506+0000","last_fullsized":"2024-04-26T17:56:29.882506+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:44.328599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046813+0000","last_change":"2024-04-26T17:53:03.618573+0000","last_active":"2024-04-26T17:56:30.046813+0000","last_peered":"2024-04-26T17:56:30.046813+0000","last_clean":"2024-04-26T17:56:30.046813+0000","last_became_active":"2024-04-26T17:53:03.618223+0000","last_became_peered":"2024-04-26T17:53:03.618223+0000","last_unstale":"2024-04-26T17:56:30.046813+0000","last_undegraded":"2024-04-26T17:56:30.046813+0000","last_fullsized":"2024-04-26T17:56:30.046813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:16.749205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894921+0000","last_change":"2024-04-26T17:53:03.621530+0000","last_active":"2024-04-26T17:56:29.894921+0000","last_peered":"2024-04-26T17:56:29.894921+0000","last_clean":"2024-04-26T17:56:29.894921+0000","last_became_active":"2024-04-26T17:53:03.621174+0000","last_became_peered":"2024-04-26T17:53:03.621174+0000","last_unstale":"2024-04-26T17:56:29.894921+0000","last_undegraded":"2024-04-26T17:56:29.894921+0000","last_fullsized":"2024-04-26T17:56:29.894921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:31.388232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047169+0000","last_change":"2024-04-26T17:53:03.617682+0000","last_active":"2024-04-26T17:56:30.047169+0000","last_peered":"2024-04-26T17:56:30.047169+0000","last_clean":"2024-04-26T17:56:30.047169+0000","last_became_active":"2024-04-26T17:53:03.617348+0000","last_became_peered":"2024-04-26T17:53:03.617348+0000","last_unstale":"2024-04-26T17:56:30.047169+0000","last_undegraded":"2024-04-26T17:56:30.047169+0000","last_fullsized":"2024-04-26T17:56:30.047169+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:21.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.784690+0000","last_change":"2024-04-26T17:53:03.617955+0000","last_active":"2024-04-26T17:56:29.784690+0000","last_peered":"2024-04-26T17:56:29.784690+0000","last_clean":"2024-04-26T17:56:29.784690+0000","last_became_active":"2024-04-26T17:53:03.616765+0000","last_became_peered":"2024-04-26T17:53:03.616765+0000","last_unstale":"2024-04-26T17:56:29.784690+0000","last_undegraded":"2024-04-26T17:56:29.784690+0000","last_fullsized":"2024-04-26T17:56:29.784690+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:59.292277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.976094+0000","last_change":"2024-04-26T17:53:49.767807+0000","last_active":"2024-04-26T17:56:29.976094+0000","last_peered":"2024-04-26T17:56:29.976094+0000","last_clean":"2024-04-26T17:56:29.976094+0000","last_became_active":"2024-04-26T17:53:49.767404+0000","last_became_peered":"2024-04-26T17:53:49.767404+0000","last_unstale":"2024-04-26T17:56:29.976094+0000","last_undegraded":"2024-04-26T17:56:29.976094+0000","last_fullsized":"2024-04-26T17:56:29.976094+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:20:45.482580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472587+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:56:30.472587+0000","last_peered":"2024-04-26T17:56:30.472587+0000","last_clean":"2024-04-26T17:56:30.472587+0000","last_became_active":"2024-04-26T17:53:03.616623+0000","last_became_peered":"2024-04-26T17:53:03.616623+0000","last_unstale":"2024-04-26T17:56:30.472587+0000","last_undegraded":"2024-04-26T17:56:30.472587+0000","last_fullsized":"2024-04-26T17:56:30.472587+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:04.231549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472552+0000","last_change":"2024-04-26T17:53:03.622147+0000","last_active":"2024-04-26T17:56:30.472552+0000","last_peered":"2024-04-26T17:56:30.472552+0000","last_clean":"2024-04-26T17:56:30.472552+0000","last_became_active":"2024-04-26T17:53:03.621542+0000","last_became_peered":"2024-04-26T17:53:03.621542+0000","last_unstale":"2024-04-26T17:56:30.472552+0000","last_undegraded":"2024-04-26T17:56:30.472552+0000","last_fullsized":"2024-04-26T17:56:30.472552+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:01.746954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.045357+0000","last_change":"2024-04-26T17:53:03.624433+0000","last_active":"2024-04-26T17:56:30.045357+0000","last_peered":"2024-04-26T17:56:30.045357+0000","last_clean":"2024-04-26T17:56:30.045357+0000","last_became_active":"2024-04-26T17:53:03.624135+0000","last_became_peered":"2024-04-26T17:53:03.624135+0000","last_unstale":"2024-04-26T17:56:30.045357+0000","last_undegraded":"2024-04-26T17:56:30.045357+0000","last_fullsized":"2024-04-26T17:56:30.045357+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:32:20.521917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.628025+0000","last_change":"2024-04-26T17:53:03.623607+0000","last_active":"2024-04-26T17:56:30.628025+0000","last_peered":"2024-04-26T17:56:30.628025+0000","last_clean":"2024-04-26T17:56:30.628025+0000","last_became_active":"2024-04-26T17:53:03.621783+0000","last_became_peered":"2024-04-26T17:53:03.621783+0000","last_unstale":"2024-04-26T17:56:30.628025+0000","last_undegraded":"2024-04-26T17:56:30.628025+0000","last_fullsized":"2024-04-26T17:56:30.628025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:38:59.872868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783567+0000","last_change":"2024-04-26T17:53:03.614974+0000","last_active":"2024-04-26T17:56:29.783567+0000","last_peered":"2024-04-26T17:56:29.783567+0000","last_clean":"2024-04-26T17:56:29.783567+0000","last_became_active":"2024-04-26T17:53:03.614619+0000","last_became_peered":"2024-04-26T17:53:03.614619+0000","last_unstale":"2024-04-26T17:56:29.783567+0000","last_undegraded":"2024-04-26T17:56:29.783567+0000","last_fullsized":"2024-04-26T17:56:29.783567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:50.729531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786362+0000","last_change":"2024-04-26T17:53:03.611140+0000","last_active":"2024-04-26T17:56:29.786362+0000","last_peered":"2024-04-26T17:56:29.786362+0000","last_clean":"2024-04-26T17:56:29.786362+0000","last_became_active":"2024-04-26T17:53:03.610549+0000","last_became_peered":"2024-04-26T17:53:03.610549+0000","last_unstale":"2024-04-26T17:56:29.786362+0000","last_undegraded":"2024-04-26T17:56:29.786362+0000","last_fullsized":"2024-04-26T17:56:29.786362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:46:17.178598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783478+0000","last_change":"2024-04-26T17:53:00.585971+0000","last_active":"2024-04-26T17:56:29.783478+0000","last_peered":"2024-04-26T17:56:29.783478+0000","last_clean":"2024-04-26T17:56:29.783478+0000","last_became_active":"2024-04-26T17:53:00.585648+0000","last_became_peered":"2024-04-26T17:53:00.585648+0000","last_unstale":"2024-04-26T17:56:29.783478+0000","last_undegraded":"2024-04-26T17:56:29.783478+0000","last_fullsized":"2024-04-26T17:56:29.783478+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:00:24.082773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783479+0000","last_change":"2024-04-26T17:53:49.770579+0000","last_active":"2024-04-26T17:56:29.783479+0000","last_peered":"2024-04-26T17:56:29.783479+0000","last_clean":"2024-04-26T17:56:29.783479+0000","last_became_active":"2024-04-26T17:53:49.770250+0000","last_became_peered":"2024-04-26T17:53:49.770250+0000","last_unstale":"2024-04-26T17:56:29.783479+0000","last_undegraded":"2024-04-26T17:56:29.783479+0000","last_fullsized":"2024-04-26T17:56:29.783479+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:35.758790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883544+0000","last_change":"2024-04-26T17:53:01.145438+0000","last_active":"2024-04-26T17:56:29.883544+0000","last_peered":"2024-04-26T17:56:29.883544+0000","last_clean":"2024-04-26T17:56:29.883544+0000","last_became_active":"2024-04-26T17:53:01.145030+0000","last_became_peered":"2024-04-26T17:53:01.145030+0000","last_unstale":"2024-04-26T17:56:29.883544+0000","last_undegraded":"2024-04-26T17:56:29.883544+0000","last_fullsized":"2024-04-26T17:56:29.883544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:15.650247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472912+0000","last_change":"2024-04-26T17:53:03.622647+0000","last_active":"2024-04-26T17:56:30.472912+0000","last_peered":"2024-04-26T17:56:30.472912+0000","last_clean":"2024-04-26T17:56:30.472912+0000","last_became_active":"2024-04-26T17:53:03.622298+0000","last_became_peered":"2024-04-26T17:53:03.622298+0000","last_unstale":"2024-04-26T17:56:30.472912+0000","last_undegraded":"2024-04-26T17:56:30.472912+0000","last_fullsized":"2024-04-26T17:56:30.472912+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:37.230539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627763+0000","last_change":"2024-04-26T17:53:49.775405+0000","last_active":"2024-04-26T17:56:30.627763+0000","last_peered":"2024-04-26T17:56:30.627763+0000","last_clean":"2024-04-26T17:56:30.627763+0000","last_became_active":"2024-04-26T17:53:49.770536+0000","last_became_peered":"2024-04-26T17:53:49.770536+0000","last_unstale":"2024-04-26T17:56:30.627763+0000","last_undegraded":"2024-04-26T17:56:30.627763+0000","last_fullsized":"2024-04-26T17:56:30.627763+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:34.643452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.045992+0000","last_change":"2024-04-26T17:53:03.617462+0000","last_active":"2024-04-26T17:56:30.045992+0000","last_peered":"2024-04-26T17:56:30.045992+0000","last_clean":"2024-04-26T17:56:30.045992+0000","last_became_active":"2024-04-26T17:53:03.617115+0000","last_became_peered":"2024-04-26T17:53:03.617115+0000","last_unstale":"2024-04-26T17:56:30.045992+0000","last_undegraded":"2024-04-26T17:56:30.045992+0000","last_fullsized":"2024-04-26T17:56:30.045992+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:11.406146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883071+0000","last_change":"2024-04-26T17:53:00.583378+0000","last_active":"2024-04-26T17:56:29.883071+0000","last_peered":"2024-04-26T17:56:29.883071+0000","last_clean":"2024-04-26T17:56:29.883071+0000","last_became_active":"2024-04-26T17:53:00.583020+0000","last_became_peered":"2024-04-26T17:53:00.583020+0000","last_unstale":"2024-04-26T17:56:29.883071+0000","last_undegraded":"2024-04-26T17:56:29.883071+0000","last_fullsized":"2024-04-26T17:56:29.883071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:24.089996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891803+0000","last_change":"2024-04-26T17:53:03.609557+0000","last_active":"2024-04-26T17:56:29.891803+0000","last_peered":"2024-04-26T17:56:29.891803+0000","last_clean":"2024-04-26T17:56:29.891803+0000","last_became_active":"2024-04-26T17:53:03.609028+0000","last_became_peered":"2024-04-26T17:53:03.609028+0000","last_unstale":"2024-04-26T17:56:29.891803+0000","last_undegraded":"2024-04-26T17:56:29.891803+0000","last_fullsized":"2024-04-26T17:56:29.891803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:39:22.860877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'8","reported_seq":229,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.984224+0000","last_change":"2024-04-26T17:53:00.586683+0000","last_active":"2024-04-26T17:56:29.984224+0000","last_peered":"2024-04-26T17:56:29.984224+0000","last_clean":"2024-04-26T17:56:29.984224+0000","last_became_active":"2024-04-26T17:53:00.586272+0000","last_became_peered":"2024-04-26T17:53:00.586272+0000","last_unstale":"2024-04-26T17:56:29.984224+0000","last_undegraded":"2024-04-26T17:56:29.984224+0000","last_fullsized":"2024-04-26T17:56:29.984224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:20:29.068607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":192,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882791+0000","last_change":"2024-04-26T17:53:03.611073+0000","last_active":"2024-04-26T17:56:29.882791+0000","last_peered":"2024-04-26T17:56:29.882791+0000","last_clean":"2024-04-26T17:56:29.882791+0000","last_became_active":"2024-04-26T17:53:03.610701+0000","last_became_peered":"2024-04-26T17:53:03.610701+0000","last_unstale":"2024-04-26T17:56:29.882791+0000","last_undegraded":"2024-04-26T17:56:29.882791+0000","last_fullsized":"2024-04-26T17:56:29.882791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:01:00.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894392+0000","last_change":"2024-04-26T17:53:00.586082+0000","last_active":"2024-04-26T17:56:29.894392+0000","last_peered":"2024-04-26T17:56:29.894392+0000","last_clean":"2024-04-26T17:56:29.894392+0000","last_became_active":"2024-04-26T17:53:00.585678+0000","last_became_peered":"2024-04-26T17:53:00.585678+0000","last_unstale":"2024-04-26T17:56:29.894392+0000","last_undegraded":"2024-04-26T17:56:29.894392+0000","last_fullsized":"2024-04-26T17:56:29.894392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:55:25.523846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891547+0000","last_change":"2024-04-26T17:53:03.600230+0000","last_active":"2024-04-26T17:56:29.891547+0000","last_peered":"2024-04-26T17:56:29.891547+0000","last_clean":"2024-04-26T17:56:29.891547+0000","last_became_active":"2024-04-26T17:53:03.599713+0000","last_became_peered":"2024-04-26T17:53:03.599713+0000","last_unstale":"2024-04-26T17:56:29.891547+0000","last_undegraded":"2024-04-26T17:56:29.891547+0000","last_fullsized":"2024-04-26T17:56:29.891547+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:45.822561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975577+0000","last_change":"2024-04-26T17:53:00.584588+0000","last_active":"2024-04-26T17:56:29.975577+0000","last_peered":"2024-04-26T17:56:29.975577+0000","last_clean":"2024-04-26T17:56:29.975577+0000","last_became_active":"2024-04-26T17:53:00.584284+0000","last_became_peered":"2024-04-26T17:53:00.584284+0000","last_unstale":"2024-04-26T17:56:29.975577+0000","last_undegraded":"2024-04-26T17:56:29.975577+0000","last_fullsized":"2024-04-26T17:56:29.975577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:08.428904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882589+0000","last_change":"2024-04-26T17:53:03.615237+0000","last_active":"2024-04-26T17:56:29.882589+0000","last_peered":"2024-04-26T17:56:29.882589+0000","last_clean":"2024-04-26T17:56:29.882589+0000","last_became_active":"2024-04-26T17:53:03.614704+0000","last_became_peered":"2024-04-26T17:53:03.614704+0000","last_unstale":"2024-04-26T17:56:29.882589+0000","last_undegraded":"2024-04-26T17:56:29.882589+0000","last_fullsized":"2024-04-26T17:56:29.882589+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:27.251839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891446+0000","last_change":"2024-04-26T17:53:00.589287+0000","last_active":"2024-04-26T17:56:29.891446+0000","last_peered":"2024-04-26T17:56:29.891446+0000","last_clean":"2024-04-26T17:56:29.891446+0000","last_became_active":"2024-04-26T17:53:00.588119+0000","last_became_peered":"2024-04-26T17:53:00.588119+0000","last_unstale":"2024-04-26T17:56:29.891446+0000","last_undegraded":"2024-04-26T17:56:29.891446+0000","last_fullsized":"2024-04-26T17:56:29.891446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:25.389777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783241+0000","last_change":"2024-04-26T17:53:03.608061+0000","last_active":"2024-04-26T17:56:29.783241+0000","last_peered":"2024-04-26T17:56:29.783241+0000","last_clean":"2024-04-26T17:56:29.783241+0000","last_became_active":"2024-04-26T17:53:03.607624+0000","last_became_peered":"2024-04-26T17:53:03.607624+0000","last_unstale":"2024-04-26T17:56:29.783241+0000","last_undegraded":"2024-04-26T17:56:29.783241+0000","last_fullsized":"2024-04-26T17:56:29.783241+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:15.111011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":222,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.983815+0000","last_change":"2024-04-26T17:53:00.588621+0000","last_active":"2024-04-26T17:56:29.983815+0000","last_peered":"2024-04-26T17:56:29.983815+0000","last_clean":"2024-04-26T17:56:29.983815+0000","last_became_active":"2024-04-26T17:53:00.587330+0000","last_became_peered":"2024-04-26T17:53:00.587330+0000","last_unstale":"2024-04-26T17:56:29.983815+0000","last_undegraded":"2024-04-26T17:56:29.983815+0000","last_fullsized":"2024-04-26T17:56:29.983815+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:01:12.178610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":188,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046606+0000","last_change":"2024-04-26T17:53:03.615723+0000","last_active":"2024-04-26T17:56:30.046606+0000","last_peered":"2024-04-26T17:56:30.046606+0000","last_clean":"2024-04-26T17:56:30.046606+0000","last_became_active":"2024-04-26T17:53:03.614931+0000","last_became_peered":"2024-04-26T17:53:03.614931+0000","last_unstale":"2024-04-26T17:56:30.046606+0000","last_undegraded":"2024-04-26T17:56:30.046606+0000","last_fullsized":"2024-04-26T17:56:30.046606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:09.663876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047232+0000","last_change":"2024-04-26T17:53:00.589232+0000","last_active":"2024-04-26T17:56:30.047232+0000","last_peered":"2024-04-26T17:56:30.047232+0000","last_clean":"2024-04-26T17:56:30.047232+0000","last_became_active":"2024-04-26T17:53:00.588863+0000","last_became_peered":"2024-04-26T17:53:00.588863+0000","last_unstale":"2024-04-26T17:56:30.047232+0000","last_undegraded":"2024-04-26T17:56:30.047232+0000","last_fullsized":"2024-04-26T17:56:30.047232+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:14.360782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472784+0000","last_change":"2024-04-26T17:53:03.624923+0000","last_active":"2024-04-26T17:56:30.472784+0000","last_peered":"2024-04-26T17:56:30.472784+0000","last_clean":"2024-04-26T17:56:30.472784+0000","last_became_active":"2024-04-26T17:53:03.624620+0000","last_became_peered":"2024-04-26T17:53:03.624620+0000","last_unstale":"2024-04-26T17:56:30.472784+0000","last_undegraded":"2024-04-26T17:56:30.472784+0000","last_fullsized":"2024-04-26T17:56:30.472784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:50:51.259563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891963+0000","last_change":"2024-04-26T17:53:01.146950+0000","last_active":"2024-04-26T17:56:29.891963+0000","last_peered":"2024-04-26T17:56:29.891963+0000","last_clean":"2024-04-26T17:56:29.891963+0000","last_became_active":"2024-04-26T17:53:01.146591+0000","last_became_peered":"2024-04-26T17:53:01.146591+0000","last_unstale":"2024-04-26T17:56:29.891963+0000","last_undegraded":"2024-04-26T17:56:29.891963+0000","last_fullsized":"2024-04-26T17:56:29.891963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:17.756660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046019+0000","last_change":"2024-04-26T17:53:03.626944+0000","last_active":"2024-04-26T17:56:30.046019+0000","last_peered":"2024-04-26T17:56:30.046019+0000","last_clean":"2024-04-26T17:56:30.046019+0000","last_became_active":"2024-04-26T17:53:03.626594+0000","last_became_peered":"2024-04-26T17:53:03.626594+0000","last_unstale":"2024-04-26T17:56:30.046019+0000","last_undegraded":"2024-04-26T17:56:30.046019+0000","last_fullsized":"2024-04-26T17:56:30.046019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:52:42.997251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'37","reported_seq":453,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.045223+0000","last_change":"2024-04-26T17:53:00.588353+0000","last_active":"2024-04-26T17:56:30.045223+0000","last_peered":"2024-04-26T17:56:30.045223+0000","last_clean":"2024-04-26T17:56:30.045223+0000","last_became_active":"2024-04-26T17:53:00.586351+0000","last_became_peered":"2024-04-26T17:53:00.586351+0000","last_unstale":"2024-04-26T17:56:30.045223+0000","last_undegraded":"2024-04-26T17:56:30.045223+0000","last_fullsized":"2024-04-26T17:56:30.045223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:41.660525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67988,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":387,"num_read_kb":255,"num_write":38,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883217+0000","last_change":"2024-04-26T17:53:03.612052+0000","last_active":"2024-04-26T17:56:29.883217+0000","last_peered":"2024-04-26T17:56:29.883217+0000","last_clean":"2024-04-26T17:56:29.883217+0000","last_became_active":"2024-04-26T17:53:03.611733+0000","last_became_peered":"2024-04-26T17:53:03.611733+0000","last_unstale":"2024-04-26T17:56:29.883217+0000","last_undegraded":"2024-04-26T17:56:29.883217+0000","last_fullsized":"2024-04-26T17:56:29.883217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:05:23.958317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786202+0000","last_change":"2024-04-26T17:53:00.588284+0000","last_active":"2024-04-26T17:56:29.786202+0000","last_peered":"2024-04-26T17:56:29.786202+0000","last_clean":"2024-04-26T17:56:29.786202+0000","last_became_active":"2024-04-26T17:53:00.587942+0000","last_became_peered":"2024-04-26T17:53:00.587942+0000","last_unstale":"2024-04-26T17:56:29.786202+0000","last_undegraded":"2024-04-26T17:56:29.786202+0000","last_fullsized":"2024-04-26T17:56:29.786202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:43.782971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975582+0000","last_change":"2024-04-26T17:53:03.613092+0000","last_active":"2024-04-26T17:56:29.975582+0000","last_peered":"2024-04-26T17:56:29.975582+0000","last_clean":"2024-04-26T17:56:29.975582+0000","last_became_active":"2024-04-26T17:53:03.612773+0000","last_became_peered":"2024-04-26T17:53:03.612773+0000","last_unstale":"2024-04-26T17:56:29.975582+0000","last_undegraded":"2024-04-26T17:56:29.975582+0000","last_fullsized":"2024-04-26T17:56:29.975582+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:41:21.340375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891560+0000","last_change":"2024-04-26T17:53:00.588777+0000","last_active":"2024-04-26T17:56:29.891560+0000","last_peered":"2024-04-26T17:56:29.891560+0000","last_clean":"2024-04-26T17:56:29.891560+0000","last_became_active":"2024-04-26T17:53:00.588380+0000","last_became_peered":"2024-04-26T17:53:00.588380+0000","last_unstale":"2024-04-26T17:56:29.891560+0000","last_undegraded":"2024-04-26T17:56:29.891560+0000","last_fullsized":"2024-04-26T17:56:29.891560+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:44.587381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047058+0000","last_change":"2024-04-26T17:53:03.620344+0000","last_active":"2024-04-26T17:56:30.047058+0000","last_peered":"2024-04-26T17:56:30.047058+0000","last_clean":"2024-04-26T17:56:30.047058+0000","last_became_active":"2024-04-26T17:53:03.619944+0000","last_became_peered":"2024-04-26T17:53:03.619944+0000","last_unstale":"2024-04-26T17:56:30.047058+0000","last_undegraded":"2024-04-26T17:56:30.047058+0000","last_fullsized":"2024-04-26T17:56:30.047058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:08:31.801342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":225,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.976712+0000","last_change":"2024-04-26T17:53:00.587733+0000","last_active":"2024-04-26T17:56:29.976712+0000","last_peered":"2024-04-26T17:56:29.976712+0000","last_clean":"2024-04-26T17:56:29.976712+0000","last_became_active":"2024-04-26T17:53:00.587373+0000","last_became_peered":"2024-04-26T17:53:00.587373+0000","last_unstale":"2024-04-26T17:56:29.976712+0000","last_undegraded":"2024-04-26T17:56:29.976712+0000","last_fullsized":"2024-04-26T17:56:29.976712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:10.978944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1287,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":192,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046569+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:56:30.046569+0000","last_peered":"2024-04-26T17:56:30.046569+0000","last_clean":"2024-04-26T17:56:30.046569+0000","last_became_active":"2024-04-26T17:53:03.616588+0000","last_became_peered":"2024-04-26T17:53:03.616588+0000","last_unstale":"2024-04-26T17:56:30.046569+0000","last_undegraded":"2024-04-26T17:56:30.046569+0000","last_fullsized":"2024-04-26T17:56:30.046569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:46:38.419613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882753+0000","last_change":"2024-04-26T17:53:01.146922+0000","last_active":"2024-04-26T17:56:29.882753+0000","last_peered":"2024-04-26T17:56:29.882753+0000","last_clean":"2024-04-26T17:56:29.882753+0000","last_became_active":"2024-04-26T17:53:01.146520+0000","last_became_peered":"2024-04-26T17:53:01.146520+0000","last_unstale":"2024-04-26T17:56:29.882753+0000","last_undegraded":"2024-04-26T17:56:29.882753+0000","last_fullsized":"2024-04-26T17:56:29.882753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:12:49.238012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891565+0000","last_change":"2024-04-26T17:53:03.608513+0000","last_active":"2024-04-26T17:56:29.891565+0000","last_peered":"2024-04-26T17:56:29.891565+0000","last_clean":"2024-04-26T17:56:29.891565+0000","last_became_active":"2024-04-26T17:53:03.608167+0000","last_became_peered":"2024-04-26T17:53:03.608167+0000","last_unstale":"2024-04-26T17:56:29.891565+0000","last_undegraded":"2024-04-26T17:56:29.891565+0000","last_fullsized":"2024-04-26T17:56:29.891565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:03.002756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883302+0000","last_change":"2024-04-26T17:53:00.583878+0000","last_active":"2024-04-26T17:56:29.883302+0000","last_peered":"2024-04-26T17:56:29.883302+0000","last_clean":"2024-04-26T17:56:29.883302+0000","last_became_active":"2024-04-26T17:53:00.583159+0000","last_became_peered":"2024-04-26T17:53:00.583159+0000","last_unstale":"2024-04-26T17:56:29.883302+0000","last_undegraded":"2024-04-26T17:56:29.883302+0000","last_fullsized":"2024-04-26T17:56:29.883302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:49.850942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627754+0000","last_change":"2024-04-26T17:53:03.614467+0000","last_active":"2024-04-26T17:56:30.627754+0000","last_peered":"2024-04-26T17:56:30.627754+0000","last_clean":"2024-04-26T17:56:30.627754+0000","last_became_active":"2024-04-26T17:53:03.613972+0000","last_became_peered":"2024-04-26T17:53:03.613972+0000","last_unstale":"2024-04-26T17:56:30.627754+0000","last_undegraded":"2024-04-26T17:56:30.627754+0000","last_fullsized":"2024-04-26T17:56:30.627754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:27.259777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783415+0000","last_change":"2024-04-26T17:53:49.767174+0000","last_active":"2024-04-26T17:56:29.783415+0000","last_peered":"2024-04-26T17:56:29.783415+0000","last_clean":"2024-04-26T17:56:29.783415+0000","last_became_active":"2024-04-26T17:53:49.766778+0000","last_became_peered":"2024-04-26T17:53:49.766778+0000","last_unstale":"2024-04-26T17:56:29.783415+0000","last_undegraded":"2024-04-26T17:56:29.783415+0000","last_fullsized":"2024-04-26T17:56:29.783415+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:59.240159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894614+0000","last_change":"2024-04-26T17:53:03.613888+0000","last_active":"2024-04-26T17:56:29.894614+0000","last_peered":"2024-04-26T17:56:29.894614+0000","last_clean":"2024-04-26T17:56:29.894614+0000","last_became_active":"2024-04-26T17:53:03.613499+0000","last_became_peered":"2024-04-26T17:53:03.613499+0000","last_unstale":"2024-04-26T17:56:29.894614+0000","last_undegraded":"2024-04-26T17:56:29.894614+0000","last_fullsized":"2024-04-26T17:56:29.894614+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:15.549419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047067+0000","last_change":"2024-04-26T17:53:00.587803+0000","last_active":"2024-04-26T17:56:30.047067+0000","last_peered":"2024-04-26T17:56:30.047067+0000","last_clean":"2024-04-26T17:56:30.047067+0000","last_became_active":"2024-04-26T17:53:00.587382+0000","last_became_peered":"2024-04-26T17:53:00.587382+0000","last_unstale":"2024-04-26T17:56:30.047067+0000","last_undegraded":"2024-04-26T17:56:30.047067+0000","last_fullsized":"2024-04-26T17:56:30.047067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:38.249478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894610+0000","last_change":"2024-04-26T17:53:49.769008+0000","last_active":"2024-04-26T17:56:29.894610+0000","last_peered":"2024-04-26T17:56:29.894610+0000","last_clean":"2024-04-26T17:56:29.894610+0000","last_became_active":"2024-04-26T17:53:49.768625+0000","last_became_peered":"2024-04-26T17:53:49.768625+0000","last_unstale":"2024-04-26T17:56:29.894610+0000","last_undegraded":"2024-04-26T17:56:29.894610+0000","last_fullsized":"2024-04-26T17:56:29.894610+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:27:46.412625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891725+0000","last_change":"2024-04-26T17:53:03.612899+0000","last_active":"2024-04-26T17:56:29.891725+0000","last_peered":"2024-04-26T17:56:29.891725+0000","last_clean":"2024-04-26T17:56:29.891725+0000","last_became_active":"2024-04-26T17:53:03.612564+0000","last_became_peered":"2024-04-26T17:53:03.612564+0000","last_unstale":"2024-04-26T17:56:29.891725+0000","last_undegraded":"2024-04-26T17:56:29.891725+0000","last_fullsized":"2024-04-26T17:56:29.891725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:07.368256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":222,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975474+0000","last_change":"2024-04-26T17:53:01.145471+0000","last_active":"2024-04-26T17:56:29.975474+0000","last_peered":"2024-04-26T17:56:29.975474+0000","last_clean":"2024-04-26T17:56:29.975474+0000","last_became_active":"2024-04-26T17:53:01.145040+0000","last_became_peered":"2024-04-26T17:53:01.145040+0000","last_unstale":"2024-04-26T17:56:29.975474+0000","last_undegraded":"2024-04-26T17:56:29.975474+0000","last_fullsized":"2024-04-26T17:56:29.975474+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:46:27.437329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":188,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882767+0000","last_change":"2024-04-26T17:53:03.618600+0000","last_active":"2024-04-26T17:56:29.882767+0000","last_peered":"2024-04-26T17:56:29.882767+0000","last_clean":"2024-04-26T17:56:29.882767+0000","last_became_active":"2024-04-26T17:53:03.618139+0000","last_became_peered":"2024-04-26T17:53:03.618139+0000","last_unstale":"2024-04-26T17:56:29.882767+0000","last_undegraded":"2024-04-26T17:56:29.882767+0000","last_fullsized":"2024-04-26T17:56:29.882767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:56.007659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.784624+0000","last_change":"2024-04-26T17:53:00.586386+0000","last_active":"2024-04-26T17:56:29.784624+0000","last_peered":"2024-04-26T17:56:29.784624+0000","last_clean":"2024-04-26T17:56:29.784624+0000","last_became_active":"2024-04-26T17:53:00.585953+0000","last_became_peered":"2024-04-26T17:53:00.585953+0000","last_unstale":"2024-04-26T17:56:29.784624+0000","last_undegraded":"2024-04-26T17:56:29.784624+0000","last_fullsized":"2024-04-26T17:56:29.784624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:39:57.160983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046643+0000","last_change":"2024-04-26T17:51:38.839853+0000","last_active":"2024-04-26T17:56:30.046643+0000","last_peered":"2024-04-26T17:56:30.046643+0000","last_clean":"2024-04-26T17:56:30.046643+0000","last_became_active":"2024-04-26T17:51:38.839318+0000","last_became_peered":"2024-04-26T17:51:38.839318+0000","last_unstale":"2024-04-26T17:56:30.046643+0000","last_undegraded":"2024-04-26T17:56:30.046643+0000","last_fullsized":"2024-04-26T17:56:30.046643+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_clean_scrub_stamp":"2024-04-26T17:48:12.577543+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:08:50.683663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975406+0000","last_change":"2024-04-26T17:53:03.614170+0000","last_active":"2024-04-26T17:56:29.975406+0000","last_peered":"2024-04-26T17:56:29.975406+0000","last_clean":"2024-04-26T17:56:29.975406+0000","last_became_active":"2024-04-26T17:53:03.613836+0000","last_became_peered":"2024-04-26T17:53:03.613836+0000","last_unstale":"2024-04-26T17:56:29.975406+0000","last_undegraded":"2024-04-26T17:56:29.975406+0000","last_fullsized":"2024-04-26T17:56:29.975406+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:48.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046858+0000","last_change":"2024-04-26T17:53:00.587319+0000","last_active":"2024-04-26T17:56:30.046858+0000","last_peered":"2024-04-26T17:56:30.046858+0000","last_clean":"2024-04-26T17:56:30.046858+0000","last_became_active":"2024-04-26T17:53:00.586983+0000","last_became_peered":"2024-04-26T17:53:00.586983+0000","last_unstale":"2024-04-26T17:56:30.046858+0000","last_undegraded":"2024-04-26T17:56:30.046858+0000","last_fullsized":"2024-04-26T17:56:30.046858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:03:43.470013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883218+0000","last_change":"2024-04-26T17:53:01.145672+0000","last_active":"2024-04-26T17:56:29.883218+0000","last_peered":"2024-04-26T17:56:29.883218+0000","last_clean":"2024-04-26T17:56:29.883218+0000","last_became_active":"2024-04-26T17:53:01.145274+0000","last_became_peered":"2024-04-26T17:53:01.145274+0000","last_unstale":"2024-04-26T17:56:29.883218+0000","last_undegraded":"2024-04-26T17:56:29.883218+0000","last_fullsized":"2024-04-26T17:56:29.883218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:21.122352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975691+0000","last_change":"2024-04-26T17:53:03.611232+0000","last_active":"2024-04-26T17:56:29.975691+0000","last_peered":"2024-04-26T17:56:29.975691+0000","last_clean":"2024-04-26T17:56:29.975691+0000","last_became_active":"2024-04-26T17:53:03.610905+0000","last_became_peered":"2024-04-26T17:53:03.610905+0000","last_unstale":"2024-04-26T17:56:29.975691+0000","last_undegraded":"2024-04-26T17:56:29.975691+0000","last_fullsized":"2024-04-26T17:56:29.975691+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:31.426237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'48","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.003263+0000","last_change":"2024-04-26T17:53:00.580220+0000","last_active":"2024-04-26T17:56:30.003263+0000","last_peered":"2024-04-26T17:56:30.003263+0000","last_clean":"2024-04-26T17:56:30.003263+0000","last_became_active":"2024-04-26T17:53:00.579818+0000","last_became_peered":"2024-04-26T17:53:00.579818+0000","last_unstale":"2024-04-26T17:56:30.003263+0000","last_undegraded":"2024-04-26T17:56:30.003263+0000","last_fullsized":"2024-04-26T17:56:30.003263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:15.260489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":87,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883670+0000","last_change":"2024-04-26T17:53:03.610882+0000","last_active":"2024-04-26T17:56:29.883670+0000","last_peered":"2024-04-26T17:56:29.883670+0000","last_clean":"2024-04-26T17:56:29.883670+0000","last_became_active":"2024-04-26T17:53:03.609979+0000","last_became_peered":"2024-04-26T17:53:03.609979+0000","last_unstale":"2024-04-26T17:56:29.883670+0000","last_undegraded":"2024-04-26T17:56:29.883670+0000","last_fullsized":"2024-04-26T17:56:29.883670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:33.167579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.784906+0000","last_change":"2024-04-26T17:53:01.358555+0000","last_active":"2024-04-26T17:56:29.784906+0000","last_peered":"2024-04-26T17:56:29.784906+0000","last_clean":"2024-04-26T17:56:29.784906+0000","last_became_active":"2024-04-26T17:53:01.358138+0000","last_became_peered":"2024-04-26T17:53:01.358138+0000","last_unstale":"2024-04-26T17:56:29.784906+0000","last_undegraded":"2024-04-26T17:56:29.784906+0000","last_fullsized":"2024-04-26T17:56:29.784906+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:14.886609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472981+0000","last_change":"2024-04-26T17:53:03.618816+0000","last_active":"2024-04-26T17:56:30.472981+0000","last_peered":"2024-04-26T17:56:30.472981+0000","last_clean":"2024-04-26T17:56:30.472981+0000","last_became_active":"2024-04-26T17:53:03.618184+0000","last_became_peered":"2024-04-26T17:53:03.618184+0000","last_unstale":"2024-04-26T17:56:30.472981+0000","last_undegraded":"2024-04-26T17:56:30.472981+0000","last_fullsized":"2024-04-26T17:56:30.472981+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:07.514516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.045564+0000","last_change":"2024-04-26T17:53:01.147081+0000","last_active":"2024-04-26T17:56:30.045564+0000","last_peered":"2024-04-26T17:56:30.045564+0000","last_clean":"2024-04-26T17:56:30.045564+0000","last_became_active":"2024-04-26T17:53:01.146698+0000","last_became_peered":"2024-04-26T17:53:01.146698+0000","last_unstale":"2024-04-26T17:56:30.045564+0000","last_undegraded":"2024-04-26T17:56:30.045564+0000","last_fullsized":"2024-04-26T17:56:30.045564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:44:19.927316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882754+0000","last_change":"2024-04-26T17:53:03.612592+0000","last_active":"2024-04-26T17:56:29.882754+0000","last_peered":"2024-04-26T17:56:29.882754+0000","last_clean":"2024-04-26T17:56:29.882754+0000","last_became_active":"2024-04-26T17:53:03.612274+0000","last_became_peered":"2024-04-26T17:53:03.612274+0000","last_unstale":"2024-04-26T17:56:29.882754+0000","last_undegraded":"2024-04-26T17:56:29.882754+0000","last_fullsized":"2024-04-26T17:56:29.882754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:30:48.461947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882651+0000","last_change":"2024-04-26T17:53:00.586430+0000","last_active":"2024-04-26T17:56:29.882651+0000","last_peered":"2024-04-26T17:56:29.882651+0000","last_clean":"2024-04-26T17:56:29.882651+0000","last_became_active":"2024-04-26T17:53:00.586004+0000","last_became_peered":"2024-04-26T17:53:00.586004+0000","last_unstale":"2024-04-26T17:56:29.882651+0000","last_undegraded":"2024-04-26T17:56:29.882651+0000","last_fullsized":"2024-04-26T17:56:29.882651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:47:18.224890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627501+0000","last_change":"2024-04-26T17:53:03.623081+0000","last_active":"2024-04-26T17:56:30.627501+0000","last_peered":"2024-04-26T17:56:30.627501+0000","last_clean":"2024-04-26T17:56:30.627501+0000","last_became_active":"2024-04-26T17:53:03.622761+0000","last_became_peered":"2024-04-26T17:53:03.622761+0000","last_unstale":"2024-04-26T17:56:30.627501+0000","last_undegraded":"2024-04-26T17:56:30.627501+0000","last_fullsized":"2024-04-26T17:56:30.627501+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:39.520075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786022+0000","last_change":"2024-04-26T17:53:00.583013+0000","last_active":"2024-04-26T17:56:29.786022+0000","last_peered":"2024-04-26T17:56:29.786022+0000","last_clean":"2024-04-26T17:56:29.786022+0000","last_became_active":"2024-04-26T17:53:00.582653+0000","last_became_peered":"2024-04-26T17:53:00.582653+0000","last_unstale":"2024-04-26T17:56:29.786022+0000","last_undegraded":"2024-04-26T17:56:29.786022+0000","last_fullsized":"2024-04-26T17:56:29.786022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:09.448621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882786+0000","last_change":"2024-04-26T17:53:03.618232+0000","last_active":"2024-04-26T17:56:29.882786+0000","last_peered":"2024-04-26T17:56:29.882786+0000","last_clean":"2024-04-26T17:56:29.882786+0000","last_became_active":"2024-04-26T17:53:03.617906+0000","last_became_peered":"2024-04-26T17:53:03.617906+0000","last_unstale":"2024-04-26T17:56:29.882786+0000","last_undegraded":"2024-04-26T17:56:29.882786+0000","last_fullsized":"2024-04-26T17:56:29.882786+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:17.357398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882975+0000","last_change":"2024-04-26T17:53:00.586565+0000","last_active":"2024-04-26T17:56:29.882975+0000","last_peered":"2024-04-26T17:56:29.882975+0000","last_clean":"2024-04-26T17:56:29.882975+0000","last_became_active":"2024-04-26T17:53:00.586161+0000","last_became_peered":"2024-04-26T17:53:00.586161+0000","last_unstale":"2024-04-26T17:56:29.882975+0000","last_undegraded":"2024-04-26T17:56:29.882975+0000","last_fullsized":"2024-04-26T17:56:29.882975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:56:25.294055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047092+0000","last_change":"2024-04-26T17:53:03.620340+0000","last_active":"2024-04-26T17:56:30.047092+0000","last_peered":"2024-04-26T17:56:30.047092+0000","last_clean":"2024-04-26T17:56:30.047092+0000","last_became_active":"2024-04-26T17:53:03.619897+0000","last_became_peered":"2024-04-26T17:53:03.619897+0000","last_unstale":"2024-04-26T17:56:30.047092+0000","last_undegraded":"2024-04-26T17:56:30.047092+0000","last_fullsized":"2024-04-26T17:56:30.047092+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:12:40.662070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786265+0000","last_change":"2024-04-26T17:53:01.358131+0000","last_active":"2024-04-26T17:56:29.786265+0000","last_peered":"2024-04-26T17:56:29.786265+0000","last_clean":"2024-04-26T17:56:29.786265+0000","last_became_active":"2024-04-26T17:53:01.357796+0000","last_became_peered":"2024-04-26T17:53:01.357796+0000","last_unstale":"2024-04-26T17:56:29.786265+0000","last_undegraded":"2024-04-26T17:56:29.786265+0000","last_fullsized":"2024-04-26T17:56:29.786265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:47.849873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047364+0000","last_change":"2024-04-26T17:53:03.616133+0000","last_active":"2024-04-26T17:56:30.047364+0000","last_peered":"2024-04-26T17:56:30.047364+0000","last_clean":"2024-04-26T17:56:30.047364+0000","last_became_active":"2024-04-26T17:53:03.615797+0000","last_became_peered":"2024-04-26T17:53:03.615797+0000","last_unstale":"2024-04-26T17:56:30.047364+0000","last_undegraded":"2024-04-26T17:56:30.047364+0000","last_fullsized":"2024-04-26T17:56:30.047364+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:05.698489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'39","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.998526+0000","last_change":"2024-04-26T17:53:00.587696+0000","last_active":"2024-04-26T17:56:29.998526+0000","last_peered":"2024-04-26T17:56:29.998526+0000","last_clean":"2024-04-26T17:56:29.998526+0000","last_became_active":"2024-04-26T17:53:00.587305+0000","last_became_peered":"2024-04-26T17:53:00.587305+0000","last_unstale":"2024-04-26T17:56:29.998526+0000","last_undegraded":"2024-04-26T17:56:29.998526+0000","last_fullsized":"2024-04-26T17:56:29.998526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:39.185753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047031+0000","last_change":"2024-04-26T17:53:03.613076+0000","last_active":"2024-04-26T17:56:30.047031+0000","last_peered":"2024-04-26T17:56:30.047031+0000","last_clean":"2024-04-26T17:56:30.047031+0000","last_became_active":"2024-04-26T17:53:03.612746+0000","last_became_peered":"2024-04-26T17:53:03.612746+0000","last_unstale":"2024-04-26T17:56:30.047031+0000","last_undegraded":"2024-04-26T17:56:30.047031+0000","last_fullsized":"2024-04-26T17:56:30.047031+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:20.920154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975561+0000","last_change":"2024-04-26T17:53:00.582575+0000","last_active":"2024-04-26T17:56:29.975561+0000","last_peered":"2024-04-26T17:56:29.975561+0000","last_clean":"2024-04-26T17:56:29.975561+0000","last_became_active":"2024-04-26T17:53:00.582215+0000","last_became_peered":"2024-04-26T17:53:00.582215+0000","last_unstale":"2024-04-26T17:56:29.975561+0000","last_undegraded":"2024-04-26T17:56:29.975561+0000","last_fullsized":"2024-04-26T17:56:29.975561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:35:12.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047430+0000","last_change":"2024-04-26T17:53:03.616269+0000","last_active":"2024-04-26T17:56:30.047430+0000","last_peered":"2024-04-26T17:56:30.047430+0000","last_clean":"2024-04-26T17:56:30.047430+0000","last_became_active":"2024-04-26T17:53:03.615896+0000","last_became_peered":"2024-04-26T17:53:03.615896+0000","last_unstale":"2024-04-26T17:56:30.047430+0000","last_undegraded":"2024-04-26T17:56:30.047430+0000","last_fullsized":"2024-04-26T17:56:30.047430+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:12.111392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627575+0000","last_change":"2024-04-26T17:53:03.621589+0000","last_active":"2024-04-26T17:56:30.627575+0000","last_peered":"2024-04-26T17:56:30.627575+0000","last_clean":"2024-04-26T17:56:30.627575+0000","last_became_active":"2024-04-26T17:53:03.621211+0000","last_became_peered":"2024-04-26T17:53:03.621211+0000","last_unstale":"2024-04-26T17:56:30.627575+0000","last_undegraded":"2024-04-26T17:56:30.627575+0000","last_fullsized":"2024-04-26T17:56:30.627575+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.487476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786065+0000","last_change":"2024-04-26T17:53:03.624536+0000","last_active":"2024-04-26T17:56:29.786065+0000","last_peered":"2024-04-26T17:56:29.786065+0000","last_clean":"2024-04-26T17:56:29.786065+0000","last_became_active":"2024-04-26T17:53:03.624234+0000","last_became_peered":"2024-04-26T17:53:03.624234+0000","last_unstale":"2024-04-26T17:56:29.786065+0000","last_undegraded":"2024-04-26T17:56:29.786065+0000","last_fullsized":"2024-04-26T17:56:29.786065+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:31:36.322688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786074+0000","last_change":"2024-04-26T17:53:03.620096+0000","last_active":"2024-04-26T17:56:29.786074+0000","last_peered":"2024-04-26T17:56:29.786074+0000","last_clean":"2024-04-26T17:56:29.786074+0000","last_became_active":"2024-04-26T17:53:03.619729+0000","last_became_peered":"2024-04-26T17:53:03.619729+0000","last_unstale":"2024-04-26T17:56:29.786074+0000","last_undegraded":"2024-04-26T17:56:29.786074+0000","last_fullsized":"2024-04-26T17:56:29.786074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:17.987728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472965+0000","last_change":"2024-04-26T17:53:49.768829+0000","last_active":"2024-04-26T17:56:30.472965+0000","last_peered":"2024-04-26T17:56:30.472965+0000","last_clean":"2024-04-26T17:56:30.472965+0000","last_became_active":"2024-04-26T17:53:49.768415+0000","last_became_peered":"2024-04-26T17:53:49.768415+0000","last_unstale":"2024-04-26T17:56:30.472965+0000","last_undegraded":"2024-04-26T17:56:30.472965+0000","last_fullsized":"2024-04-26T17:56:30.472965+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:57.364355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975670+0000","last_change":"2024-04-26T17:53:03.619193+0000","last_active":"2024-04-26T17:56:29.975670+0000","last_peered":"2024-04-26T17:56:29.975670+0000","last_clean":"2024-04-26T17:56:29.975670+0000","last_became_active":"2024-04-26T17:53:03.618776+0000","last_became_peered":"2024-04-26T17:53:03.618776+0000","last_unstale":"2024-04-26T17:56:29.975670+0000","last_undegraded":"2024-04-26T17:56:29.975670+0000","last_fullsized":"2024-04-26T17:56:29.975670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:07:24.811683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627751+0000","last_change":"2024-04-26T17:53:03.621366+0000","last_active":"2024-04-26T17:56:30.627751+0000","last_peered":"2024-04-26T17:56:30.627751+0000","last_clean":"2024-04-26T17:56:30.627751+0000","last_became_active":"2024-04-26T17:53:03.621048+0000","last_became_peered":"2024-04-26T17:53:03.621048+0000","last_unstale":"2024-04-26T17:56:30.627751+0000","last_undegraded":"2024-04-26T17:56:30.627751+0000","last_fullsized":"2024-04-26T17:56:30.627751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:31:06.484880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786409+0000","last_change":"2024-04-26T17:53:03.619383+0000","last_active":"2024-04-26T17:56:29.786409+0000","last_peered":"2024-04-26T17:56:29.786409+0000","last_clean":"2024-04-26T17:56:29.786409+0000","last_became_active":"2024-04-26T17:53:03.619017+0000","last_became_peered":"2024-04-26T17:53:03.619017+0000","last_unstale":"2024-04-26T17:56:29.786409+0000","last_undegraded":"2024-04-26T17:56:29.786409+0000","last_fullsized":"2024-04-26T17:56:29.786409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:20.091959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883534+0000","last_change":"2024-04-26T17:53:49.770695+0000","last_active":"2024-04-26T17:56:29.883534+0000","last_peered":"2024-04-26T17:56:29.883534+0000","last_clean":"2024-04-26T17:56:29.883534+0000","last_became_active":"2024-04-26T17:53:49.770303+0000","last_became_peered":"2024-04-26T17:53:49.770303+0000","last_unstale":"2024-04-26T17:56:29.883534+0000","last_undegraded":"2024-04-26T17:56:29.883534+0000","last_fullsized":"2024-04-26T17:56:29.883534+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:22.830129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891522+0000","last_change":"2024-04-26T17:53:03.605521+0000","last_active":"2024-04-26T17:56:29.891522+0000","last_peered":"2024-04-26T17:56:29.891522+0000","last_clean":"2024-04-26T17:56:29.891522+0000","last_became_active":"2024-04-26T17:53:03.605145+0000","last_became_peered":"2024-04-26T17:53:03.605145+0000","last_unstale":"2024-04-26T17:56:29.891522+0000","last_undegraded":"2024-04-26T17:56:29.891522+0000","last_fullsized":"2024-04-26T17:56:29.891522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:58:09.856021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882572+0000","last_change":"2024-04-26T17:53:03.610866+0000","last_active":"2024-04-26T17:56:29.882572+0000","last_peered":"2024-04-26T17:56:29.882572+0000","last_clean":"2024-04-26T17:56:29.882572+0000","last_became_active":"2024-04-26T17:53:03.610326+0000","last_became_peered":"2024-04-26T17:53:03.610326+0000","last_unstale":"2024-04-26T17:56:29.882572+0000","last_undegraded":"2024-04-26T17:56:29.882572+0000","last_fullsized":"2024-04-26T17:56:29.882572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:20.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046004+0000","last_change":"2024-04-26T17:53:03.615336+0000","last_active":"2024-04-26T17:56:30.046004+0000","last_peered":"2024-04-26T17:56:30.046004+0000","last_clean":"2024-04-26T17:56:30.046004+0000","last_became_active":"2024-04-26T17:53:03.614447+0000","last_became_peered":"2024-04-26T17:53:03.614447+0000","last_unstale":"2024-04-26T17:56:30.046004+0000","last_undegraded":"2024-04-26T17:56:30.046004+0000","last_fullsized":"2024-04-26T17:56:30.046004+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:36:10.091929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.976250+0000","last_change":"2024-04-26T17:53:03.614863+0000","last_active":"2024-04-26T17:56:29.976250+0000","last_peered":"2024-04-26T17:56:29.976250+0000","last_clean":"2024-04-26T17:56:29.976250+0000","last_became_active":"2024-04-26T17:53:03.614520+0000","last_became_peered":"2024-04-26T17:53:03.614520+0000","last_unstale":"2024-04-26T17:56:29.976250+0000","last_undegraded":"2024-04-26T17:56:29.976250+0000","last_fullsized":"2024-04-26T17:56:29.976250+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:51.367785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627720+0000","last_change":"2024-04-26T17:53:03.618521+0000","last_active":"2024-04-26T17:56:30.627720+0000","last_peered":"2024-04-26T17:56:30.627720+0000","last_clean":"2024-04-26T17:56:30.627720+0000","last_became_active":"2024-04-26T17:53:03.617658+0000","last_became_peered":"2024-04-26T17:53:03.617658+0000","last_unstale":"2024-04-26T17:56:30.627720+0000","last_undegraded":"2024-04-26T17:56:30.627720+0000","last_fullsized":"2024-04-26T17:56:30.627720+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:39.813098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106863,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598036,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1252,"num_read_kb":731,"num_write":295,"num_write_kb":1440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":1794108,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383737,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":479802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645374,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":1274981,"data_compressed":44598,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808898,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":882152,"data_compressed":22068,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070535,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":882072,"data_compressed":22068,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332173,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":1082017,"data_compressed":38543,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495698,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":488325,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1064960,"data_stored":556106,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738480,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":480698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1085440,"data_stored":552892,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182496,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1011712,"data_stored":488636,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":489461,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757335,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1089536,"data_stored":1212602,"data_compressed":44699,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133999,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":201865,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T17:56:35.152 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T17:56:35.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T17:56:35.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: Unable to set osd_memory_target on smithi186 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-26T17:56:35.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:56:35.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-26T17:56:35.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T17:56:35.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-26T17:56:35.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-26T17:56:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T17:56:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:56:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:56:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: Updating smithi031:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:56:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:56:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:56:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:56:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: from='client.15234 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:56:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-26T17:56:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.269 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:35 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T17:56:35.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: Unable to set osd_memory_target on smithi186 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-26T17:56:35.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T17:56:35.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-26T17:56:35.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T17:56:35.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-26T17:56:35.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-26T17:56:35.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T17:56:35.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:56:35.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:56:35.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: Updating smithi031:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T17:56:35.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:56:35.273 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:56:35.273 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-04-26T17:56:35.273 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: from='client.15234 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:56:35.273 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-26T17:56:35.273 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.273 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.273 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.273 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.273 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.273 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.273 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:35 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:35.404 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:36.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:36 smithi031 ceph-mon[29164]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:56:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:36 smithi031 ceph-mon[29164]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:56:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:36 smithi031 ceph-mon[29164]: Updating smithi031:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:56:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:36 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:56:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:36 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:56:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:36 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:56:36.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:36 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:36 smithi186 ceph-mon[32797]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:56:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:36 smithi186 ceph-mon[32797]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:56:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:36 smithi186 ceph-mon[32797]: Updating smithi031:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:56:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:36 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:56:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:36 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:56:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:36 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:56:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:36 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:36 smithi097 ceph-mon[32419]: Updating smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:56:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:36 smithi097 ceph-mon[32419]: Updating smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:56:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:36 smithi097 ceph-mon[32419]: Updating smithi031:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T17:56:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:36 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:56:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:36 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:56:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:36 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:56:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:36 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:56:37.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.307+0000 7fcae0694700 1 -- 172.21.15.31:0/863311206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad8103e30 msgr2=0x7fcad8110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:37.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.307+0000 7fcae0694700 1 --2- 172.21.15.31:0/863311206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad8103e30 0x7fcad8110290 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcad000b0a0 tx=0x7fcad000b3b0 comp rx=0 tx=0).stop 2024-04-26T17:56:37.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:37 smithi031 ceph-mon[29164]: pgmap v6: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 183 KiB/s wr, 103 op/s 2024-04-26T17:56:37.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.307+0000 7fcae0694700 1 -- 172.21.15.31:0/863311206 shutdown_connections 2024-04-26T17:56:37.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.307+0000 7fcae0694700 1 --2- 172.21.15.31:0/863311206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad8103e30 0x7fcad8110290 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:37.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.307+0000 7fcae0694700 1 --2- 172.21.15.31:0/863311206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcad81094c0 0x7fcad81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:37.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.307+0000 7fcae0694700 1 --2- 172.21.15.31:0/863311206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcad8103470 0x7fcad81038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:37.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.307+0000 7fcae0694700 1 -- 172.21.15.31:0/863311206 >> 172.21.15.31:0/863311206 conn(0x7fcad807a580 msgr2=0x7fcad807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:37.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.308+0000 7fcae0694700 1 -- 172.21.15.31:0/863311206 shutdown_connections 2024-04-26T17:56:37.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.308+0000 7fcae0694700 1 -- 172.21.15.31:0/863311206 wait complete. 2024-04-26T17:56:37.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.308+0000 7fcae0694700 1 Processor -- start 2024-04-26T17:56:37.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.308+0000 7fcae0694700 1 -- start start 2024-04-26T17:56:37.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcae0694700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad8103470 0x7fcad8075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:37.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcae0694700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcad8103e30 0x7fcad8075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:37.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcae0694700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcad81094c0 0x7fcad80761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:37.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcae0694700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcad81135e0 con 0x7fcad8103470 2024-04-26T17:56:37.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcae0694700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fcad8113460 con 0x7fcad8103e30 2024-04-26T17:56:37.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcae0694700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcad8113760 con 0x7fcad81094c0 2024-04-26T17:56:37.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcade430700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad8103470 0x7fcad8075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:37.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcade430700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad8103470 0x7fcad8075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34322/0 (socket says 172.21.15.31:34322) 2024-04-26T17:56:37.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcaddc2f700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcad8103e30 0x7fcad8075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:37.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcadec31700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcad81094c0 0x7fcad80761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:37.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcade430700 1 -- 172.21.15.31:0/876515817 learned_addr learned my addr 172.21.15.31:0/876515817 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:37.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcaddc2f700 1 -- 172.21.15.31:0/876515817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcad81094c0 msgr2=0x7fcad80761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:37.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcaddc2f700 1 --2- 172.21.15.31:0/876515817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcad81094c0 0x7fcad80761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:37.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcaddc2f700 1 -- 172.21.15.31:0/876515817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad8103470 msgr2=0x7fcad8075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:37.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcaddc2f700 1 --2- 172.21.15.31:0/876515817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad8103470 0x7fcad8075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:37.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcaddc2f700 1 -- 172.21.15.31:0/876515817 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad81a7fe0 con 0x7fcad8103e30 2024-04-26T17:56:37.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcadec31700 1 --2- 172.21.15.31:0/876515817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcad81094c0 0x7fcad80761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:56:37.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcade430700 1 --2- 172.21.15.31:0/876515817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad8103470 0x7fcad8075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:56:37.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.309+0000 7fcaddc2f700 1 --2- 172.21.15.31:0/876515817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcad8103e30 0x7fcad8075c60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcac800d950 tx=0x7fcac800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:37.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.310+0000 7fcacf7fe700 1 -- 172.21.15.31:0/876515817 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8016070 con 0x7fcad8103e30 2024-04-26T17:56:37.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.310+0000 7fcae0694700 1 -- 172.21.15.31:0/876515817 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcad81a82d0 con 0x7fcad8103e30 2024-04-26T17:56:37.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.310+0000 7fcae0694700 1 -- 172.21.15.31:0/876515817 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcad81a8860 con 0x7fcad8103e30 2024-04-26T17:56:37.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.310+0000 7fcacf7fe700 1 -- 172.21.15.31:0/876515817 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcac800fd40 con 0x7fcad8103e30 2024-04-26T17:56:37.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.310+0000 7fcacf7fe700 1 -- 172.21.15.31:0/876515817 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8020060 con 0x7fcad8103e30 2024-04-26T17:56:37.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.311+0000 7fcae0694700 1 -- 172.21.15.31:0/876515817 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcabc005330 con 0x7fcad8103e30 2024-04-26T17:56:37.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.312+0000 7fcacf7fe700 1 -- 172.21.15.31:0/876515817 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fcac8019070 con 0x7fcad8103e30 2024-04-26T17:56:37.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.313+0000 7fcacf7fe700 1 --2- 172.21.15.31:0/876515817 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fcac40a3ec0 0x7fcac40a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:37.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.313+0000 7fcacf7fe700 1 -- 172.21.15.31:0/876515817 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fcac80d0700 con 0x7fcad8103e30 2024-04-26T17:56:37.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.313+0000 7fcade430700 1 --2- 172.21.15.31:0/876515817 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fcac40a3ec0 0x7fcac40a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:37.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.313+0000 7fcade430700 1 --2- 172.21.15.31:0/876515817 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fcac40a3ec0 0x7fcac40a6380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcad40104e0 tx=0x7fcad400b480 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:37.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.317+0000 7fcacf7fe700 1 -- 172.21.15.31:0/876515817 <== mon.2 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcac809cc20 con 0x7fcad8103e30 2024-04-26T17:56:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:37 smithi097 ceph-mon[32419]: pgmap v6: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 183 KiB/s wr, 103 op/s 2024-04-26T17:56:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:37 smithi186 ceph-mon[32797]: pgmap v6: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 183 KiB/s wr, 103 op/s 2024-04-26T17:56:37.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.525+0000 7fcae0694700 1 -- 172.21.15.31:0/876515817 --> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcabc000bc0 con 0x7fcac40a3ec0 2024-04-26T17:56:37.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.533+0000 7fcacf7fe700 1 -- 172.21.15.31:0/876515817 <== mgr.24826 v2:172.21.15.31:6800/3595544994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258026 (secure 0 0 0) 0x7fcabc000bc0 con 0x7fcac40a3ec0 2024-04-26T17:56:37.536 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.535+0000 7fcae0694700 1 -- 172.21.15.31:0/876515817 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fcac40a3ec0 msgr2=0x7fcac40a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.535+0000 7fcae0694700 1 --2- 172.21.15.31:0/876515817 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fcac40a3ec0 0x7fcac40a6380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcad40104e0 tx=0x7fcad400b480 comp rx=0 tx=0).stop 2024-04-26T17:56:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.535+0000 7fcae0694700 1 -- 172.21.15.31:0/876515817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcad8103e30 msgr2=0x7fcad8075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.535+0000 7fcae0694700 1 --2- 172.21.15.31:0/876515817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcad8103e30 0x7fcad8075c60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcac800d950 tx=0x7fcac800dc60 comp rx=0 tx=0).stop 2024-04-26T17:56:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.535+0000 7fcae0694700 1 -- 172.21.15.31:0/876515817 shutdown_connections 2024-04-26T17:56:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.535+0000 7fcae0694700 1 --2- 172.21.15.31:0/876515817 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fcac40a3ec0 0x7fcac40a6380 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.535+0000 7fcae0694700 1 --2- 172.21.15.31:0/876515817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcad81094c0 0x7fcad80761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.535+0000 7fcae0694700 1 --2- 172.21.15.31:0/876515817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcad8103470 0x7fcad8075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.535+0000 7fcae0694700 1 --2- 172.21.15.31:0/876515817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcad8103e30 0x7fcad8075c60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.535+0000 7fcae0694700 1 -- 172.21.15.31:0/876515817 >> 172.21.15.31:0/876515817 conn(0x7fcad807a580 msgr2=0x7fcad8102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.535+0000 7fcae0694700 1 -- 172.21.15.31:0/876515817 shutdown_connections 2024-04-26T17:56:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:37.535+0000 7fcae0694700 1 -- 172.21.15.31:0/876515817 wait complete. 2024-04-26T17:56:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-26T17:56:38.204 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-04-26T17:56:35.799225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1385270,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1759,"num_read_kb":1298,"num_write":624,"num_write_kb":5294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":959,"ondisk_log_size":959,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365680,"kb_used_data":12328,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576144,"statfs":{"total":1151940427776,"available":1151565971456,"internally_reserved":0,"allocated":12623872,"data_stored":9796722,"data_compressed":233188,"data_compressed_allocated":303104,"data_compressed_original":3776512,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":371,"num_read_kb":448,"num_write":146,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.992237"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627925+0000","last_change":"2024-04-26T17:53:03.616592+0000","last_active":"2024-04-26T17:56:30.627925+0000","last_peered":"2024-04-26T17:56:30.627925+0000","last_clean":"2024-04-26T17:56:30.627925+0000","last_became_active":"2024-04-26T17:53:03.616255+0000","last_became_peered":"2024-04-26T17:53:03.616255+0000","last_unstale":"2024-04-26T17:56:30.627925+0000","last_undegraded":"2024-04-26T17:56:30.627925+0000","last_fullsized":"2024-04-26T17:56:30.627925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:43:31.183679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472977+0000","last_change":"2024-04-26T17:53:03.615402+0000","last_active":"2024-04-26T17:56:30.472977+0000","last_peered":"2024-04-26T17:56:30.472977+0000","last_clean":"2024-04-26T17:56:30.472977+0000","last_became_active":"2024-04-26T17:53:03.615048+0000","last_became_peered":"2024-04-26T17:53:03.615048+0000","last_unstale":"2024-04-26T17:56:30.472977+0000","last_undegraded":"2024-04-26T17:56:30.472977+0000","last_fullsized":"2024-04-26T17:56:30.472977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:05:12.454770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472800+0000","last_change":"2024-04-26T17:53:03.622625+0000","last_active":"2024-04-26T17:56:30.472800+0000","last_peered":"2024-04-26T17:56:30.472800+0000","last_clean":"2024-04-26T17:56:30.472800+0000","last_became_active":"2024-04-26T17:53:03.622305+0000","last_became_peered":"2024-04-26T17:53:03.622305+0000","last_unstale":"2024-04-26T17:56:30.472800+0000","last_undegraded":"2024-04-26T17:56:30.472800+0000","last_fullsized":"2024-04-26T17:56:30.472800+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:06.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.784761+0000","last_change":"2024-04-26T17:53:03.617650+0000","last_active":"2024-04-26T17:56:29.784761+0000","last_peered":"2024-04-26T17:56:29.784761+0000","last_clean":"2024-04-26T17:56:29.784761+0000","last_became_active":"2024-04-26T17:53:03.617220+0000","last_became_peered":"2024-04-26T17:53:03.617220+0000","last_unstale":"2024-04-26T17:56:29.784761+0000","last_undegraded":"2024-04-26T17:56:29.784761+0000","last_fullsized":"2024-04-26T17:56:29.784761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:31.765577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882813+0000","last_change":"2024-04-26T17:53:03.615429+0000","last_active":"2024-04-26T17:56:29.882813+0000","last_peered":"2024-04-26T17:56:29.882813+0000","last_clean":"2024-04-26T17:56:29.882813+0000","last_became_active":"2024-04-26T17:53:03.615079+0000","last_became_peered":"2024-04-26T17:53:03.615079+0000","last_unstale":"2024-04-26T17:56:29.882813+0000","last_undegraded":"2024-04-26T17:56:29.882813+0000","last_fullsized":"2024-04-26T17:56:29.882813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:20.994016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894485+0000","last_change":"2024-04-26T17:53:03.624619+0000","last_active":"2024-04-26T17:56:29.894485+0000","last_peered":"2024-04-26T17:56:29.894485+0000","last_clean":"2024-04-26T17:56:29.894485+0000","last_became_active":"2024-04-26T17:53:03.624251+0000","last_became_peered":"2024-04-26T17:53:03.624251+0000","last_unstale":"2024-04-26T17:56:29.894485+0000","last_undegraded":"2024-04-26T17:56:29.894485+0000","last_fullsized":"2024-04-26T17:56:29.894485+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:50:17.535235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046789+0000","last_change":"2024-04-26T17:53:03.619325+0000","last_active":"2024-04-26T17:56:30.046789+0000","last_peered":"2024-04-26T17:56:30.046789+0000","last_clean":"2024-04-26T17:56:30.046789+0000","last_became_active":"2024-04-26T17:53:03.618989+0000","last_became_peered":"2024-04-26T17:53:03.618989+0000","last_unstale":"2024-04-26T17:56:30.046789+0000","last_undegraded":"2024-04-26T17:56:30.046789+0000","last_fullsized":"2024-04-26T17:56:30.046789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:10.195742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882506+0000","last_change":"2024-04-26T17:53:03.609346+0000","last_active":"2024-04-26T17:56:29.882506+0000","last_peered":"2024-04-26T17:56:29.882506+0000","last_clean":"2024-04-26T17:56:29.882506+0000","last_became_active":"2024-04-26T17:53:03.609021+0000","last_became_peered":"2024-04-26T17:53:03.609021+0000","last_unstale":"2024-04-26T17:56:29.882506+0000","last_undegraded":"2024-04-26T17:56:29.882506+0000","last_fullsized":"2024-04-26T17:56:29.882506+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:44.328599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046813+0000","last_change":"2024-04-26T17:53:03.618573+0000","last_active":"2024-04-26T17:56:30.046813+0000","last_peered":"2024-04-26T17:56:30.046813+0000","last_clean":"2024-04-26T17:56:30.046813+0000","last_became_active":"2024-04-26T17:53:03.618223+0000","last_became_peered":"2024-04-26T17:53:03.618223+0000","last_unstale":"2024-04-26T17:56:30.046813+0000","last_undegraded":"2024-04-26T17:56:30.046813+0000","last_fullsized":"2024-04-26T17:56:30.046813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:16.749205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894921+0000","last_change":"2024-04-26T17:53:03.621530+0000","last_active":"2024-04-26T17:56:29.894921+0000","last_peered":"2024-04-26T17:56:29.894921+0000","last_clean":"2024-04-26T17:56:29.894921+0000","last_became_active":"2024-04-26T17:53:03.621174+0000","last_became_peered":"2024-04-26T17:53:03.621174+0000","last_unstale":"2024-04-26T17:56:29.894921+0000","last_undegraded":"2024-04-26T17:56:29.894921+0000","last_fullsized":"2024-04-26T17:56:29.894921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:31.388232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047169+0000","last_change":"2024-04-26T17:53:03.617682+0000","last_active":"2024-04-26T17:56:30.047169+0000","last_peered":"2024-04-26T17:56:30.047169+0000","last_clean":"2024-04-26T17:56:30.047169+0000","last_became_active":"2024-04-26T17:53:03.617348+0000","last_became_peered":"2024-04-26T17:53:03.617348+0000","last_unstale":"2024-04-26T17:56:30.047169+0000","last_undegraded":"2024-04-26T17:56:30.047169+0000","last_fullsized":"2024-04-26T17:56:30.047169+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:21.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.784690+0000","last_change":"2024-04-26T17:53:03.617955+0000","last_active":"2024-04-26T17:56:29.784690+0000","last_peered":"2024-04-26T17:56:29.784690+0000","last_clean":"2024-04-26T17:56:29.784690+0000","last_became_active":"2024-04-26T17:53:03.616765+0000","last_became_peered":"2024-04-26T17:53:03.616765+0000","last_unstale":"2024-04-26T17:56:29.784690+0000","last_undegraded":"2024-04-26T17:56:29.784690+0000","last_fullsized":"2024-04-26T17:56:29.784690+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:59.292277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.976094+0000","last_change":"2024-04-26T17:53:49.767807+0000","last_active":"2024-04-26T17:56:29.976094+0000","last_peered":"2024-04-26T17:56:29.976094+0000","last_clean":"2024-04-26T17:56:29.976094+0000","last_became_active":"2024-04-26T17:53:49.767404+0000","last_became_peered":"2024-04-26T17:53:49.767404+0000","last_unstale":"2024-04-26T17:56:29.976094+0000","last_undegraded":"2024-04-26T17:56:29.976094+0000","last_fullsized":"2024-04-26T17:56:29.976094+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:20:45.482580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472587+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:56:30.472587+0000","last_peered":"2024-04-26T17:56:30.472587+0000","last_clean":"2024-04-26T17:56:30.472587+0000","last_became_active":"2024-04-26T17:53:03.616623+0000","last_became_peered":"2024-04-26T17:53:03.616623+0000","last_unstale":"2024-04-26T17:56:30.472587+0000","last_undegraded":"2024-04-26T17:56:30.472587+0000","last_fullsized":"2024-04-26T17:56:30.472587+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:04.231549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472552+0000","last_change":"2024-04-26T17:53:03.622147+0000","last_active":"2024-04-26T17:56:30.472552+0000","last_peered":"2024-04-26T17:56:30.472552+0000","last_clean":"2024-04-26T17:56:30.472552+0000","last_became_active":"2024-04-26T17:53:03.621542+0000","last_became_peered":"2024-04-26T17:53:03.621542+0000","last_unstale":"2024-04-26T17:56:30.472552+0000","last_undegraded":"2024-04-26T17:56:30.472552+0000","last_fullsized":"2024-04-26T17:56:30.472552+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:01.746954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.045357+0000","last_change":"2024-04-26T17:53:03.624433+0000","last_active":"2024-04-26T17:56:30.045357+0000","last_peered":"2024-04-26T17:56:30.045357+0000","last_clean":"2024-04-26T17:56:30.045357+0000","last_became_active":"2024-04-26T17:53:03.624135+0000","last_became_peered":"2024-04-26T17:53:03.624135+0000","last_unstale":"2024-04-26T17:56:30.045357+0000","last_undegraded":"2024-04-26T17:56:30.045357+0000","last_fullsized":"2024-04-26T17:56:30.045357+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:32:20.521917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.628025+0000","last_change":"2024-04-26T17:53:03.623607+0000","last_active":"2024-04-26T17:56:30.628025+0000","last_peered":"2024-04-26T17:56:30.628025+0000","last_clean":"2024-04-26T17:56:30.628025+0000","last_became_active":"2024-04-26T17:53:03.621783+0000","last_became_peered":"2024-04-26T17:53:03.621783+0000","last_unstale":"2024-04-26T17:56:30.628025+0000","last_undegraded":"2024-04-26T17:56:30.628025+0000","last_fullsized":"2024-04-26T17:56:30.628025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:38:59.872868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783567+0000","last_change":"2024-04-26T17:53:03.614974+0000","last_active":"2024-04-26T17:56:29.783567+0000","last_peered":"2024-04-26T17:56:29.783567+0000","last_clean":"2024-04-26T17:56:29.783567+0000","last_became_active":"2024-04-26T17:53:03.614619+0000","last_became_peered":"2024-04-26T17:53:03.614619+0000","last_unstale":"2024-04-26T17:56:29.783567+0000","last_undegraded":"2024-04-26T17:56:29.783567+0000","last_fullsized":"2024-04-26T17:56:29.783567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:50.729531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786362+0000","last_change":"2024-04-26T17:53:03.611140+0000","last_active":"2024-04-26T17:56:29.786362+0000","last_peered":"2024-04-26T17:56:29.786362+0000","last_clean":"2024-04-26T17:56:29.786362+0000","last_became_active":"2024-04-26T17:53:03.610549+0000","last_became_peered":"2024-04-26T17:53:03.610549+0000","last_unstale":"2024-04-26T17:56:29.786362+0000","last_undegraded":"2024-04-26T17:56:29.786362+0000","last_fullsized":"2024-04-26T17:56:29.786362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:46:17.178598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783478+0000","last_change":"2024-04-26T17:53:00.585971+0000","last_active":"2024-04-26T17:56:29.783478+0000","last_peered":"2024-04-26T17:56:29.783478+0000","last_clean":"2024-04-26T17:56:29.783478+0000","last_became_active":"2024-04-26T17:53:00.585648+0000","last_became_peered":"2024-04-26T17:53:00.585648+0000","last_unstale":"2024-04-26T17:56:29.783478+0000","last_undegraded":"2024-04-26T17:56:29.783478+0000","last_fullsized":"2024-04-26T17:56:29.783478+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:00:24.082773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783479+0000","last_change":"2024-04-26T17:53:49.770579+0000","last_active":"2024-04-26T17:56:29.783479+0000","last_peered":"2024-04-26T17:56:29.783479+0000","last_clean":"2024-04-26T17:56:29.783479+0000","last_became_active":"2024-04-26T17:53:49.770250+0000","last_became_peered":"2024-04-26T17:53:49.770250+0000","last_unstale":"2024-04-26T17:56:29.783479+0000","last_undegraded":"2024-04-26T17:56:29.783479+0000","last_fullsized":"2024-04-26T17:56:29.783479+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:35.758790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883544+0000","last_change":"2024-04-26T17:53:01.145438+0000","last_active":"2024-04-26T17:56:29.883544+0000","last_peered":"2024-04-26T17:56:29.883544+0000","last_clean":"2024-04-26T17:56:29.883544+0000","last_became_active":"2024-04-26T17:53:01.145030+0000","last_became_peered":"2024-04-26T17:53:01.145030+0000","last_unstale":"2024-04-26T17:56:29.883544+0000","last_undegraded":"2024-04-26T17:56:29.883544+0000","last_fullsized":"2024-04-26T17:56:29.883544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:15.650247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472912+0000","last_change":"2024-04-26T17:53:03.622647+0000","last_active":"2024-04-26T17:56:30.472912+0000","last_peered":"2024-04-26T17:56:30.472912+0000","last_clean":"2024-04-26T17:56:30.472912+0000","last_became_active":"2024-04-26T17:53:03.622298+0000","last_became_peered":"2024-04-26T17:53:03.622298+0000","last_unstale":"2024-04-26T17:56:30.472912+0000","last_undegraded":"2024-04-26T17:56:30.472912+0000","last_fullsized":"2024-04-26T17:56:30.472912+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:37.230539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627763+0000","last_change":"2024-04-26T17:53:49.775405+0000","last_active":"2024-04-26T17:56:30.627763+0000","last_peered":"2024-04-26T17:56:30.627763+0000","last_clean":"2024-04-26T17:56:30.627763+0000","last_became_active":"2024-04-26T17:53:49.770536+0000","last_became_peered":"2024-04-26T17:53:49.770536+0000","last_unstale":"2024-04-26T17:56:30.627763+0000","last_undegraded":"2024-04-26T17:56:30.627763+0000","last_fullsized":"2024-04-26T17:56:30.627763+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:34.643452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.045992+0000","last_change":"2024-04-26T17:53:03.617462+0000","last_active":"2024-04-26T17:56:30.045992+0000","last_peered":"2024-04-26T17:56:30.045992+0000","last_clean":"2024-04-26T17:56:30.045992+0000","last_became_active":"2024-04-26T17:53:03.617115+0000","last_became_peered":"2024-04-26T17:53:03.617115+0000","last_unstale":"2024-04-26T17:56:30.045992+0000","last_undegraded":"2024-04-26T17:56:30.045992+0000","last_fullsized":"2024-04-26T17:56:30.045992+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:11.406146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883071+0000","last_change":"2024-04-26T17:53:00.583378+0000","last_active":"2024-04-26T17:56:29.883071+0000","last_peered":"2024-04-26T17:56:29.883071+0000","last_clean":"2024-04-26T17:56:29.883071+0000","last_became_active":"2024-04-26T17:53:00.583020+0000","last_became_peered":"2024-04-26T17:53:00.583020+0000","last_unstale":"2024-04-26T17:56:29.883071+0000","last_undegraded":"2024-04-26T17:56:29.883071+0000","last_fullsized":"2024-04-26T17:56:29.883071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:24.089996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891803+0000","last_change":"2024-04-26T17:53:03.609557+0000","last_active":"2024-04-26T17:56:29.891803+0000","last_peered":"2024-04-26T17:56:29.891803+0000","last_clean":"2024-04-26T17:56:29.891803+0000","last_became_active":"2024-04-26T17:53:03.609028+0000","last_became_peered":"2024-04-26T17:53:03.609028+0000","last_unstale":"2024-04-26T17:56:29.891803+0000","last_undegraded":"2024-04-26T17:56:29.891803+0000","last_fullsized":"2024-04-26T17:56:29.891803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:39:22.860877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'8","reported_seq":235,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:33.990001+0000","last_change":"2024-04-26T17:53:00.586683+0000","last_active":"2024-04-26T17:56:33.990001+0000","last_peered":"2024-04-26T17:56:33.990001+0000","last_clean":"2024-04-26T17:56:33.990001+0000","last_became_active":"2024-04-26T17:53:00.586272+0000","last_became_peered":"2024-04-26T17:53:00.586272+0000","last_unstale":"2024-04-26T17:56:33.990001+0000","last_undegraded":"2024-04-26T17:56:33.990001+0000","last_fullsized":"2024-04-26T17:56:33.990001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:20:29.068607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882791+0000","last_change":"2024-04-26T17:53:03.611073+0000","last_active":"2024-04-26T17:56:29.882791+0000","last_peered":"2024-04-26T17:56:29.882791+0000","last_clean":"2024-04-26T17:56:29.882791+0000","last_became_active":"2024-04-26T17:53:03.610701+0000","last_became_peered":"2024-04-26T17:53:03.610701+0000","last_unstale":"2024-04-26T17:56:29.882791+0000","last_undegraded":"2024-04-26T17:56:29.882791+0000","last_fullsized":"2024-04-26T17:56:29.882791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:01:00.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894392+0000","last_change":"2024-04-26T17:53:00.586082+0000","last_active":"2024-04-26T17:56:29.894392+0000","last_peered":"2024-04-26T17:56:29.894392+0000","last_clean":"2024-04-26T17:56:29.894392+0000","last_became_active":"2024-04-26T17:53:00.585678+0000","last_became_peered":"2024-04-26T17:53:00.585678+0000","last_unstale":"2024-04-26T17:56:29.894392+0000","last_undegraded":"2024-04-26T17:56:29.894392+0000","last_fullsized":"2024-04-26T17:56:29.894392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:55:25.523846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891547+0000","last_change":"2024-04-26T17:53:03.600230+0000","last_active":"2024-04-26T17:56:29.891547+0000","last_peered":"2024-04-26T17:56:29.891547+0000","last_clean":"2024-04-26T17:56:29.891547+0000","last_became_active":"2024-04-26T17:53:03.599713+0000","last_became_peered":"2024-04-26T17:53:03.599713+0000","last_unstale":"2024-04-26T17:56:29.891547+0000","last_undegraded":"2024-04-26T17:56:29.891547+0000","last_fullsized":"2024-04-26T17:56:29.891547+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:45.822561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975577+0000","last_change":"2024-04-26T17:53:00.584588+0000","last_active":"2024-04-26T17:56:29.975577+0000","last_peered":"2024-04-26T17:56:29.975577+0000","last_clean":"2024-04-26T17:56:29.975577+0000","last_became_active":"2024-04-26T17:53:00.584284+0000","last_became_peered":"2024-04-26T17:53:00.584284+0000","last_unstale":"2024-04-26T17:56:29.975577+0000","last_undegraded":"2024-04-26T17:56:29.975577+0000","last_fullsized":"2024-04-26T17:56:29.975577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:08.428904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882589+0000","last_change":"2024-04-26T17:53:03.615237+0000","last_active":"2024-04-26T17:56:29.882589+0000","last_peered":"2024-04-26T17:56:29.882589+0000","last_clean":"2024-04-26T17:56:29.882589+0000","last_became_active":"2024-04-26T17:53:03.614704+0000","last_became_peered":"2024-04-26T17:53:03.614704+0000","last_unstale":"2024-04-26T17:56:29.882589+0000","last_undegraded":"2024-04-26T17:56:29.882589+0000","last_fullsized":"2024-04-26T17:56:29.882589+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:27.251839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891446+0000","last_change":"2024-04-26T17:53:00.589287+0000","last_active":"2024-04-26T17:56:29.891446+0000","last_peered":"2024-04-26T17:56:29.891446+0000","last_clean":"2024-04-26T17:56:29.891446+0000","last_became_active":"2024-04-26T17:53:00.588119+0000","last_became_peered":"2024-04-26T17:53:00.588119+0000","last_unstale":"2024-04-26T17:56:29.891446+0000","last_undegraded":"2024-04-26T17:56:29.891446+0000","last_fullsized":"2024-04-26T17:56:29.891446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:25.389777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783241+0000","last_change":"2024-04-26T17:53:03.608061+0000","last_active":"2024-04-26T17:56:29.783241+0000","last_peered":"2024-04-26T17:56:29.783241+0000","last_clean":"2024-04-26T17:56:29.783241+0000","last_became_active":"2024-04-26T17:53:03.607624+0000","last_became_peered":"2024-04-26T17:53:03.607624+0000","last_unstale":"2024-04-26T17:56:29.783241+0000","last_undegraded":"2024-04-26T17:56:29.783241+0000","last_fullsized":"2024-04-26T17:56:29.783241+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:15.111011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":222,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.983815+0000","last_change":"2024-04-26T17:53:00.588621+0000","last_active":"2024-04-26T17:56:29.983815+0000","last_peered":"2024-04-26T17:56:29.983815+0000","last_clean":"2024-04-26T17:56:29.983815+0000","last_became_active":"2024-04-26T17:53:00.587330+0000","last_became_peered":"2024-04-26T17:53:00.587330+0000","last_unstale":"2024-04-26T17:56:29.983815+0000","last_undegraded":"2024-04-26T17:56:29.983815+0000","last_fullsized":"2024-04-26T17:56:29.983815+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:01:12.178610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":188,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046606+0000","last_change":"2024-04-26T17:53:03.615723+0000","last_active":"2024-04-26T17:56:30.046606+0000","last_peered":"2024-04-26T17:56:30.046606+0000","last_clean":"2024-04-26T17:56:30.046606+0000","last_became_active":"2024-04-26T17:53:03.614931+0000","last_became_peered":"2024-04-26T17:53:03.614931+0000","last_unstale":"2024-04-26T17:56:30.046606+0000","last_undegraded":"2024-04-26T17:56:30.046606+0000","last_fullsized":"2024-04-26T17:56:30.046606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:09.663876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047232+0000","last_change":"2024-04-26T17:53:00.589232+0000","last_active":"2024-04-26T17:56:30.047232+0000","last_peered":"2024-04-26T17:56:30.047232+0000","last_clean":"2024-04-26T17:56:30.047232+0000","last_became_active":"2024-04-26T17:53:00.588863+0000","last_became_peered":"2024-04-26T17:53:00.588863+0000","last_unstale":"2024-04-26T17:56:30.047232+0000","last_undegraded":"2024-04-26T17:56:30.047232+0000","last_fullsized":"2024-04-26T17:56:30.047232+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:14.360782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472784+0000","last_change":"2024-04-26T17:53:03.624923+0000","last_active":"2024-04-26T17:56:30.472784+0000","last_peered":"2024-04-26T17:56:30.472784+0000","last_clean":"2024-04-26T17:56:30.472784+0000","last_became_active":"2024-04-26T17:53:03.624620+0000","last_became_peered":"2024-04-26T17:53:03.624620+0000","last_unstale":"2024-04-26T17:56:30.472784+0000","last_undegraded":"2024-04-26T17:56:30.472784+0000","last_fullsized":"2024-04-26T17:56:30.472784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:50:51.259563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891963+0000","last_change":"2024-04-26T17:53:01.146950+0000","last_active":"2024-04-26T17:56:29.891963+0000","last_peered":"2024-04-26T17:56:29.891963+0000","last_clean":"2024-04-26T17:56:29.891963+0000","last_became_active":"2024-04-26T17:53:01.146591+0000","last_became_peered":"2024-04-26T17:53:01.146591+0000","last_unstale":"2024-04-26T17:56:29.891963+0000","last_undegraded":"2024-04-26T17:56:29.891963+0000","last_fullsized":"2024-04-26T17:56:29.891963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:17.756660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046019+0000","last_change":"2024-04-26T17:53:03.626944+0000","last_active":"2024-04-26T17:56:30.046019+0000","last_peered":"2024-04-26T17:56:30.046019+0000","last_clean":"2024-04-26T17:56:30.046019+0000","last_became_active":"2024-04-26T17:53:03.626594+0000","last_became_peered":"2024-04-26T17:53:03.626594+0000","last_unstale":"2024-04-26T17:56:30.046019+0000","last_undegraded":"2024-04-26T17:56:30.046019+0000","last_fullsized":"2024-04-26T17:56:30.046019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:52:42.997251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'37","reported_seq":453,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.045223+0000","last_change":"2024-04-26T17:53:00.588353+0000","last_active":"2024-04-26T17:56:30.045223+0000","last_peered":"2024-04-26T17:56:30.045223+0000","last_clean":"2024-04-26T17:56:30.045223+0000","last_became_active":"2024-04-26T17:53:00.586351+0000","last_became_peered":"2024-04-26T17:53:00.586351+0000","last_unstale":"2024-04-26T17:56:30.045223+0000","last_undegraded":"2024-04-26T17:56:30.045223+0000","last_fullsized":"2024-04-26T17:56:30.045223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:41.660525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67988,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":387,"num_read_kb":255,"num_write":38,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883217+0000","last_change":"2024-04-26T17:53:03.612052+0000","last_active":"2024-04-26T17:56:29.883217+0000","last_peered":"2024-04-26T17:56:29.883217+0000","last_clean":"2024-04-26T17:56:29.883217+0000","last_became_active":"2024-04-26T17:53:03.611733+0000","last_became_peered":"2024-04-26T17:53:03.611733+0000","last_unstale":"2024-04-26T17:56:29.883217+0000","last_undegraded":"2024-04-26T17:56:29.883217+0000","last_fullsized":"2024-04-26T17:56:29.883217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:05:23.958317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786202+0000","last_change":"2024-04-26T17:53:00.588284+0000","last_active":"2024-04-26T17:56:29.786202+0000","last_peered":"2024-04-26T17:56:29.786202+0000","last_clean":"2024-04-26T17:56:29.786202+0000","last_became_active":"2024-04-26T17:53:00.587942+0000","last_became_peered":"2024-04-26T17:53:00.587942+0000","last_unstale":"2024-04-26T17:56:29.786202+0000","last_undegraded":"2024-04-26T17:56:29.786202+0000","last_fullsized":"2024-04-26T17:56:29.786202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:43.782971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975582+0000","last_change":"2024-04-26T17:53:03.613092+0000","last_active":"2024-04-26T17:56:29.975582+0000","last_peered":"2024-04-26T17:56:29.975582+0000","last_clean":"2024-04-26T17:56:29.975582+0000","last_became_active":"2024-04-26T17:53:03.612773+0000","last_became_peered":"2024-04-26T17:53:03.612773+0000","last_unstale":"2024-04-26T17:56:29.975582+0000","last_undegraded":"2024-04-26T17:56:29.975582+0000","last_fullsized":"2024-04-26T17:56:29.975582+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:41:21.340375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891560+0000","last_change":"2024-04-26T17:53:00.588777+0000","last_active":"2024-04-26T17:56:29.891560+0000","last_peered":"2024-04-26T17:56:29.891560+0000","last_clean":"2024-04-26T17:56:29.891560+0000","last_became_active":"2024-04-26T17:53:00.588380+0000","last_became_peered":"2024-04-26T17:53:00.588380+0000","last_unstale":"2024-04-26T17:56:29.891560+0000","last_undegraded":"2024-04-26T17:56:29.891560+0000","last_fullsized":"2024-04-26T17:56:29.891560+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:44.587381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047058+0000","last_change":"2024-04-26T17:53:03.620344+0000","last_active":"2024-04-26T17:56:30.047058+0000","last_peered":"2024-04-26T17:56:30.047058+0000","last_clean":"2024-04-26T17:56:30.047058+0000","last_became_active":"2024-04-26T17:53:03.619944+0000","last_became_peered":"2024-04-26T17:53:03.619944+0000","last_unstale":"2024-04-26T17:56:30.047058+0000","last_undegraded":"2024-04-26T17:56:30.047058+0000","last_fullsized":"2024-04-26T17:56:30.047058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:08:31.801342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":231,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:33.983294+0000","last_change":"2024-04-26T17:53:00.587733+0000","last_active":"2024-04-26T17:56:33.983294+0000","last_peered":"2024-04-26T17:56:33.983294+0000","last_clean":"2024-04-26T17:56:33.983294+0000","last_became_active":"2024-04-26T17:53:00.587373+0000","last_became_peered":"2024-04-26T17:53:00.587373+0000","last_unstale":"2024-04-26T17:56:33.983294+0000","last_undegraded":"2024-04-26T17:56:33.983294+0000","last_fullsized":"2024-04-26T17:56:33.983294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:10.978944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1287,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046569+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:56:30.046569+0000","last_peered":"2024-04-26T17:56:30.046569+0000","last_clean":"2024-04-26T17:56:30.046569+0000","last_became_active":"2024-04-26T17:53:03.616588+0000","last_became_peered":"2024-04-26T17:53:03.616588+0000","last_unstale":"2024-04-26T17:56:30.046569+0000","last_undegraded":"2024-04-26T17:56:30.046569+0000","last_fullsized":"2024-04-26T17:56:30.046569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:46:38.419613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882753+0000","last_change":"2024-04-26T17:53:01.146922+0000","last_active":"2024-04-26T17:56:29.882753+0000","last_peered":"2024-04-26T17:56:29.882753+0000","last_clean":"2024-04-26T17:56:29.882753+0000","last_became_active":"2024-04-26T17:53:01.146520+0000","last_became_peered":"2024-04-26T17:53:01.146520+0000","last_unstale":"2024-04-26T17:56:29.882753+0000","last_undegraded":"2024-04-26T17:56:29.882753+0000","last_fullsized":"2024-04-26T17:56:29.882753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:12:49.238012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891565+0000","last_change":"2024-04-26T17:53:03.608513+0000","last_active":"2024-04-26T17:56:29.891565+0000","last_peered":"2024-04-26T17:56:29.891565+0000","last_clean":"2024-04-26T17:56:29.891565+0000","last_became_active":"2024-04-26T17:53:03.608167+0000","last_became_peered":"2024-04-26T17:53:03.608167+0000","last_unstale":"2024-04-26T17:56:29.891565+0000","last_undegraded":"2024-04-26T17:56:29.891565+0000","last_fullsized":"2024-04-26T17:56:29.891565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:03.002756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883302+0000","last_change":"2024-04-26T17:53:00.583878+0000","last_active":"2024-04-26T17:56:29.883302+0000","last_peered":"2024-04-26T17:56:29.883302+0000","last_clean":"2024-04-26T17:56:29.883302+0000","last_became_active":"2024-04-26T17:53:00.583159+0000","last_became_peered":"2024-04-26T17:53:00.583159+0000","last_unstale":"2024-04-26T17:56:29.883302+0000","last_undegraded":"2024-04-26T17:56:29.883302+0000","last_fullsized":"2024-04-26T17:56:29.883302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:49.850942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627754+0000","last_change":"2024-04-26T17:53:03.614467+0000","last_active":"2024-04-26T17:56:30.627754+0000","last_peered":"2024-04-26T17:56:30.627754+0000","last_clean":"2024-04-26T17:56:30.627754+0000","last_became_active":"2024-04-26T17:53:03.613972+0000","last_became_peered":"2024-04-26T17:53:03.613972+0000","last_unstale":"2024-04-26T17:56:30.627754+0000","last_undegraded":"2024-04-26T17:56:30.627754+0000","last_fullsized":"2024-04-26T17:56:30.627754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:27.259777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783415+0000","last_change":"2024-04-26T17:53:49.767174+0000","last_active":"2024-04-26T17:56:29.783415+0000","last_peered":"2024-04-26T17:56:29.783415+0000","last_clean":"2024-04-26T17:56:29.783415+0000","last_became_active":"2024-04-26T17:53:49.766778+0000","last_became_peered":"2024-04-26T17:53:49.766778+0000","last_unstale":"2024-04-26T17:56:29.783415+0000","last_undegraded":"2024-04-26T17:56:29.783415+0000","last_fullsized":"2024-04-26T17:56:29.783415+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:59.240159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894614+0000","last_change":"2024-04-26T17:53:03.613888+0000","last_active":"2024-04-26T17:56:29.894614+0000","last_peered":"2024-04-26T17:56:29.894614+0000","last_clean":"2024-04-26T17:56:29.894614+0000","last_became_active":"2024-04-26T17:53:03.613499+0000","last_became_peered":"2024-04-26T17:53:03.613499+0000","last_unstale":"2024-04-26T17:56:29.894614+0000","last_undegraded":"2024-04-26T17:56:29.894614+0000","last_fullsized":"2024-04-26T17:56:29.894614+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:15.549419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047067+0000","last_change":"2024-04-26T17:53:00.587803+0000","last_active":"2024-04-26T17:56:30.047067+0000","last_peered":"2024-04-26T17:56:30.047067+0000","last_clean":"2024-04-26T17:56:30.047067+0000","last_became_active":"2024-04-26T17:53:00.587382+0000","last_became_peered":"2024-04-26T17:53:00.587382+0000","last_unstale":"2024-04-26T17:56:30.047067+0000","last_undegraded":"2024-04-26T17:56:30.047067+0000","last_fullsized":"2024-04-26T17:56:30.047067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:38.249478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894610+0000","last_change":"2024-04-26T17:53:49.769008+0000","last_active":"2024-04-26T17:56:29.894610+0000","last_peered":"2024-04-26T17:56:29.894610+0000","last_clean":"2024-04-26T17:56:29.894610+0000","last_became_active":"2024-04-26T17:53:49.768625+0000","last_became_peered":"2024-04-26T17:53:49.768625+0000","last_unstale":"2024-04-26T17:56:29.894610+0000","last_undegraded":"2024-04-26T17:56:29.894610+0000","last_fullsized":"2024-04-26T17:56:29.894610+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:27:46.412625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891725+0000","last_change":"2024-04-26T17:53:03.612899+0000","last_active":"2024-04-26T17:56:29.891725+0000","last_peered":"2024-04-26T17:56:29.891725+0000","last_clean":"2024-04-26T17:56:29.891725+0000","last_became_active":"2024-04-26T17:53:03.612564+0000","last_became_peered":"2024-04-26T17:53:03.612564+0000","last_unstale":"2024-04-26T17:56:29.891725+0000","last_undegraded":"2024-04-26T17:56:29.891725+0000","last_fullsized":"2024-04-26T17:56:29.891725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:07.368256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":227,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:33.982666+0000","last_change":"2024-04-26T17:53:01.145471+0000","last_active":"2024-04-26T17:56:33.982666+0000","last_peered":"2024-04-26T17:56:33.982666+0000","last_clean":"2024-04-26T17:56:33.982666+0000","last_became_active":"2024-04-26T17:53:01.145040+0000","last_became_peered":"2024-04-26T17:53:01.145040+0000","last_unstale":"2024-04-26T17:56:33.982666+0000","last_undegraded":"2024-04-26T17:56:33.982666+0000","last_fullsized":"2024-04-26T17:56:33.982666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:46:27.437329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":193,"num_read_kb":193,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882767+0000","last_change":"2024-04-26T17:53:03.618600+0000","last_active":"2024-04-26T17:56:29.882767+0000","last_peered":"2024-04-26T17:56:29.882767+0000","last_clean":"2024-04-26T17:56:29.882767+0000","last_became_active":"2024-04-26T17:53:03.618139+0000","last_became_peered":"2024-04-26T17:53:03.618139+0000","last_unstale":"2024-04-26T17:56:29.882767+0000","last_undegraded":"2024-04-26T17:56:29.882767+0000","last_fullsized":"2024-04-26T17:56:29.882767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:56.007659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.784624+0000","last_change":"2024-04-26T17:53:00.586386+0000","last_active":"2024-04-26T17:56:29.784624+0000","last_peered":"2024-04-26T17:56:29.784624+0000","last_clean":"2024-04-26T17:56:29.784624+0000","last_became_active":"2024-04-26T17:53:00.585953+0000","last_became_peered":"2024-04-26T17:53:00.585953+0000","last_unstale":"2024-04-26T17:56:29.784624+0000","last_undegraded":"2024-04-26T17:56:29.784624+0000","last_fullsized":"2024-04-26T17:56:29.784624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:39:57.160983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'148","reported_seq":405,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:31.909946+0000","last_change":"2024-04-26T17:51:38.839853+0000","last_active":"2024-04-26T17:56:31.909946+0000","last_peered":"2024-04-26T17:56:31.909946+0000","last_clean":"2024-04-26T17:56:31.909946+0000","last_became_active":"2024-04-26T17:51:38.839318+0000","last_became_peered":"2024-04-26T17:51:38.839318+0000","last_unstale":"2024-04-26T17:56:31.909946+0000","last_undegraded":"2024-04-26T17:56:31.909946+0000","last_fullsized":"2024-04-26T17:56:31.909946+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_clean_scrub_stamp":"2024-04-26T17:48:12.577543+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:08:50.683663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975406+0000","last_change":"2024-04-26T17:53:03.614170+0000","last_active":"2024-04-26T17:56:29.975406+0000","last_peered":"2024-04-26T17:56:29.975406+0000","last_clean":"2024-04-26T17:56:29.975406+0000","last_became_active":"2024-04-26T17:53:03.613836+0000","last_became_peered":"2024-04-26T17:53:03.613836+0000","last_unstale":"2024-04-26T17:56:29.975406+0000","last_undegraded":"2024-04-26T17:56:29.975406+0000","last_fullsized":"2024-04-26T17:56:29.975406+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:48.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046858+0000","last_change":"2024-04-26T17:53:00.587319+0000","last_active":"2024-04-26T17:56:30.046858+0000","last_peered":"2024-04-26T17:56:30.046858+0000","last_clean":"2024-04-26T17:56:30.046858+0000","last_became_active":"2024-04-26T17:53:00.586983+0000","last_became_peered":"2024-04-26T17:53:00.586983+0000","last_unstale":"2024-04-26T17:56:30.046858+0000","last_undegraded":"2024-04-26T17:56:30.046858+0000","last_fullsized":"2024-04-26T17:56:30.046858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:03:43.470013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883218+0000","last_change":"2024-04-26T17:53:01.145672+0000","last_active":"2024-04-26T17:56:29.883218+0000","last_peered":"2024-04-26T17:56:29.883218+0000","last_clean":"2024-04-26T17:56:29.883218+0000","last_became_active":"2024-04-26T17:53:01.145274+0000","last_became_peered":"2024-04-26T17:53:01.145274+0000","last_unstale":"2024-04-26T17:56:29.883218+0000","last_undegraded":"2024-04-26T17:56:29.883218+0000","last_fullsized":"2024-04-26T17:56:29.883218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:21.122352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975691+0000","last_change":"2024-04-26T17:53:03.611232+0000","last_active":"2024-04-26T17:56:29.975691+0000","last_peered":"2024-04-26T17:56:29.975691+0000","last_clean":"2024-04-26T17:56:29.975691+0000","last_became_active":"2024-04-26T17:53:03.610905+0000","last_became_peered":"2024-04-26T17:53:03.610905+0000","last_unstale":"2024-04-26T17:56:29.975691+0000","last_undegraded":"2024-04-26T17:56:29.975691+0000","last_fullsized":"2024-04-26T17:56:29.975691+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:31.426237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'101","reported_seq":222,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:32.139765+0000","last_change":"2024-04-26T17:53:00.580220+0000","last_active":"2024-04-26T17:56:32.139765+0000","last_peered":"2024-04-26T17:56:32.139765+0000","last_clean":"2024-04-26T17:56:32.139765+0000","last_became_active":"2024-04-26T17:53:00.579818+0000","last_became_peered":"2024-04-26T17:53:00.579818+0000","last_unstale":"2024-04-26T17:56:32.139765+0000","last_undegraded":"2024-04-26T17:56:32.139765+0000","last_fullsized":"2024-04-26T17:56:32.139765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:15.260489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":186,"num_write_kb":1287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883670+0000","last_change":"2024-04-26T17:53:03.610882+0000","last_active":"2024-04-26T17:56:29.883670+0000","last_peered":"2024-04-26T17:56:29.883670+0000","last_clean":"2024-04-26T17:56:29.883670+0000","last_became_active":"2024-04-26T17:53:03.609979+0000","last_became_peered":"2024-04-26T17:53:03.609979+0000","last_unstale":"2024-04-26T17:56:29.883670+0000","last_undegraded":"2024-04-26T17:56:29.883670+0000","last_fullsized":"2024-04-26T17:56:29.883670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:33.167579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.784906+0000","last_change":"2024-04-26T17:53:01.358555+0000","last_active":"2024-04-26T17:56:29.784906+0000","last_peered":"2024-04-26T17:56:29.784906+0000","last_clean":"2024-04-26T17:56:29.784906+0000","last_became_active":"2024-04-26T17:53:01.358138+0000","last_became_peered":"2024-04-26T17:53:01.358138+0000","last_unstale":"2024-04-26T17:56:29.784906+0000","last_undegraded":"2024-04-26T17:56:29.784906+0000","last_fullsized":"2024-04-26T17:56:29.784906+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:14.886609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472981+0000","last_change":"2024-04-26T17:53:03.618816+0000","last_active":"2024-04-26T17:56:30.472981+0000","last_peered":"2024-04-26T17:56:30.472981+0000","last_clean":"2024-04-26T17:56:30.472981+0000","last_became_active":"2024-04-26T17:53:03.618184+0000","last_became_peered":"2024-04-26T17:53:03.618184+0000","last_unstale":"2024-04-26T17:56:30.472981+0000","last_undegraded":"2024-04-26T17:56:30.472981+0000","last_fullsized":"2024-04-26T17:56:30.472981+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:07.514516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.045564+0000","last_change":"2024-04-26T17:53:01.147081+0000","last_active":"2024-04-26T17:56:30.045564+0000","last_peered":"2024-04-26T17:56:30.045564+0000","last_clean":"2024-04-26T17:56:30.045564+0000","last_became_active":"2024-04-26T17:53:01.146698+0000","last_became_peered":"2024-04-26T17:53:01.146698+0000","last_unstale":"2024-04-26T17:56:30.045564+0000","last_undegraded":"2024-04-26T17:56:30.045564+0000","last_fullsized":"2024-04-26T17:56:30.045564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:44:19.927316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882754+0000","last_change":"2024-04-26T17:53:03.612592+0000","last_active":"2024-04-26T17:56:29.882754+0000","last_peered":"2024-04-26T17:56:29.882754+0000","last_clean":"2024-04-26T17:56:29.882754+0000","last_became_active":"2024-04-26T17:53:03.612274+0000","last_became_peered":"2024-04-26T17:53:03.612274+0000","last_unstale":"2024-04-26T17:56:29.882754+0000","last_undegraded":"2024-04-26T17:56:29.882754+0000","last_fullsized":"2024-04-26T17:56:29.882754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:30:48.461947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882651+0000","last_change":"2024-04-26T17:53:00.586430+0000","last_active":"2024-04-26T17:56:29.882651+0000","last_peered":"2024-04-26T17:56:29.882651+0000","last_clean":"2024-04-26T17:56:29.882651+0000","last_became_active":"2024-04-26T17:53:00.586004+0000","last_became_peered":"2024-04-26T17:53:00.586004+0000","last_unstale":"2024-04-26T17:56:29.882651+0000","last_undegraded":"2024-04-26T17:56:29.882651+0000","last_fullsized":"2024-04-26T17:56:29.882651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:47:18.224890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627501+0000","last_change":"2024-04-26T17:53:03.623081+0000","last_active":"2024-04-26T17:56:30.627501+0000","last_peered":"2024-04-26T17:56:30.627501+0000","last_clean":"2024-04-26T17:56:30.627501+0000","last_became_active":"2024-04-26T17:53:03.622761+0000","last_became_peered":"2024-04-26T17:53:03.622761+0000","last_unstale":"2024-04-26T17:56:30.627501+0000","last_undegraded":"2024-04-26T17:56:30.627501+0000","last_fullsized":"2024-04-26T17:56:30.627501+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:39.520075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786022+0000","last_change":"2024-04-26T17:53:00.583013+0000","last_active":"2024-04-26T17:56:29.786022+0000","last_peered":"2024-04-26T17:56:29.786022+0000","last_clean":"2024-04-26T17:56:29.786022+0000","last_became_active":"2024-04-26T17:53:00.582653+0000","last_became_peered":"2024-04-26T17:53:00.582653+0000","last_unstale":"2024-04-26T17:56:29.786022+0000","last_undegraded":"2024-04-26T17:56:29.786022+0000","last_fullsized":"2024-04-26T17:56:29.786022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:09.448621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882786+0000","last_change":"2024-04-26T17:53:03.618232+0000","last_active":"2024-04-26T17:56:29.882786+0000","last_peered":"2024-04-26T17:56:29.882786+0000","last_clean":"2024-04-26T17:56:29.882786+0000","last_became_active":"2024-04-26T17:53:03.617906+0000","last_became_peered":"2024-04-26T17:53:03.617906+0000","last_unstale":"2024-04-26T17:56:29.882786+0000","last_undegraded":"2024-04-26T17:56:29.882786+0000","last_fullsized":"2024-04-26T17:56:29.882786+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:17.357398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882975+0000","last_change":"2024-04-26T17:53:00.586565+0000","last_active":"2024-04-26T17:56:29.882975+0000","last_peered":"2024-04-26T17:56:29.882975+0000","last_clean":"2024-04-26T17:56:29.882975+0000","last_became_active":"2024-04-26T17:53:00.586161+0000","last_became_peered":"2024-04-26T17:53:00.586161+0000","last_unstale":"2024-04-26T17:56:29.882975+0000","last_undegraded":"2024-04-26T17:56:29.882975+0000","last_fullsized":"2024-04-26T17:56:29.882975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:56:25.294055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047092+0000","last_change":"2024-04-26T17:53:03.620340+0000","last_active":"2024-04-26T17:56:30.047092+0000","last_peered":"2024-04-26T17:56:30.047092+0000","last_clean":"2024-04-26T17:56:30.047092+0000","last_became_active":"2024-04-26T17:53:03.619897+0000","last_became_peered":"2024-04-26T17:53:03.619897+0000","last_unstale":"2024-04-26T17:56:30.047092+0000","last_undegraded":"2024-04-26T17:56:30.047092+0000","last_fullsized":"2024-04-26T17:56:30.047092+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:12:40.662070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786265+0000","last_change":"2024-04-26T17:53:01.358131+0000","last_active":"2024-04-26T17:56:29.786265+0000","last_peered":"2024-04-26T17:56:29.786265+0000","last_clean":"2024-04-26T17:56:29.786265+0000","last_became_active":"2024-04-26T17:53:01.357796+0000","last_became_peered":"2024-04-26T17:53:01.357796+0000","last_unstale":"2024-04-26T17:56:29.786265+0000","last_undegraded":"2024-04-26T17:56:29.786265+0000","last_fullsized":"2024-04-26T17:56:29.786265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:47.849873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047364+0000","last_change":"2024-04-26T17:53:03.616133+0000","last_active":"2024-04-26T17:56:30.047364+0000","last_peered":"2024-04-26T17:56:30.047364+0000","last_clean":"2024-04-26T17:56:30.047364+0000","last_became_active":"2024-04-26T17:53:03.615797+0000","last_became_peered":"2024-04-26T17:53:03.615797+0000","last_unstale":"2024-04-26T17:56:30.047364+0000","last_undegraded":"2024-04-26T17:56:30.047364+0000","last_fullsized":"2024-04-26T17:56:30.047364+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:05.698489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'65","reported_seq":125,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:31.733880+0000","last_change":"2024-04-26T17:53:00.587696+0000","last_active":"2024-04-26T17:56:31.733880+0000","last_peered":"2024-04-26T17:56:31.733880+0000","last_clean":"2024-04-26T17:56:31.733880+0000","last_became_active":"2024-04-26T17:53:00.587305+0000","last_became_peered":"2024-04-26T17:53:00.587305+0000","last_unstale":"2024-04-26T17:56:31.733880+0000","last_undegraded":"2024-04-26T17:56:31.733880+0000","last_fullsized":"2024-04-26T17:56:31.733880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:39.185753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047031+0000","last_change":"2024-04-26T17:53:03.613076+0000","last_active":"2024-04-26T17:56:30.047031+0000","last_peered":"2024-04-26T17:56:30.047031+0000","last_clean":"2024-04-26T17:56:30.047031+0000","last_became_active":"2024-04-26T17:53:03.612746+0000","last_became_peered":"2024-04-26T17:53:03.612746+0000","last_unstale":"2024-04-26T17:56:30.047031+0000","last_undegraded":"2024-04-26T17:56:30.047031+0000","last_fullsized":"2024-04-26T17:56:30.047031+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:20.920154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975561+0000","last_change":"2024-04-26T17:53:00.582575+0000","last_active":"2024-04-26T17:56:29.975561+0000","last_peered":"2024-04-26T17:56:29.975561+0000","last_clean":"2024-04-26T17:56:29.975561+0000","last_became_active":"2024-04-26T17:53:00.582215+0000","last_became_peered":"2024-04-26T17:53:00.582215+0000","last_unstale":"2024-04-26T17:56:29.975561+0000","last_undegraded":"2024-04-26T17:56:29.975561+0000","last_fullsized":"2024-04-26T17:56:29.975561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:35:12.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047430+0000","last_change":"2024-04-26T17:53:03.616269+0000","last_active":"2024-04-26T17:56:30.047430+0000","last_peered":"2024-04-26T17:56:30.047430+0000","last_clean":"2024-04-26T17:56:30.047430+0000","last_became_active":"2024-04-26T17:53:03.615896+0000","last_became_peered":"2024-04-26T17:53:03.615896+0000","last_unstale":"2024-04-26T17:56:30.047430+0000","last_undegraded":"2024-04-26T17:56:30.047430+0000","last_fullsized":"2024-04-26T17:56:30.047430+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:12.111392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627575+0000","last_change":"2024-04-26T17:53:03.621589+0000","last_active":"2024-04-26T17:56:30.627575+0000","last_peered":"2024-04-26T17:56:30.627575+0000","last_clean":"2024-04-26T17:56:30.627575+0000","last_became_active":"2024-04-26T17:53:03.621211+0000","last_became_peered":"2024-04-26T17:53:03.621211+0000","last_unstale":"2024-04-26T17:56:30.627575+0000","last_undegraded":"2024-04-26T17:56:30.627575+0000","last_fullsized":"2024-04-26T17:56:30.627575+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.487476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786065+0000","last_change":"2024-04-26T17:53:03.624536+0000","last_active":"2024-04-26T17:56:29.786065+0000","last_peered":"2024-04-26T17:56:29.786065+0000","last_clean":"2024-04-26T17:56:29.786065+0000","last_became_active":"2024-04-26T17:53:03.624234+0000","last_became_peered":"2024-04-26T17:53:03.624234+0000","last_unstale":"2024-04-26T17:56:29.786065+0000","last_undegraded":"2024-04-26T17:56:29.786065+0000","last_fullsized":"2024-04-26T17:56:29.786065+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:31:36.322688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786074+0000","last_change":"2024-04-26T17:53:03.620096+0000","last_active":"2024-04-26T17:56:29.786074+0000","last_peered":"2024-04-26T17:56:29.786074+0000","last_clean":"2024-04-26T17:56:29.786074+0000","last_became_active":"2024-04-26T17:53:03.619729+0000","last_became_peered":"2024-04-26T17:53:03.619729+0000","last_unstale":"2024-04-26T17:56:29.786074+0000","last_undegraded":"2024-04-26T17:56:29.786074+0000","last_fullsized":"2024-04-26T17:56:29.786074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:17.987728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472965+0000","last_change":"2024-04-26T17:53:49.768829+0000","last_active":"2024-04-26T17:56:30.472965+0000","last_peered":"2024-04-26T17:56:30.472965+0000","last_clean":"2024-04-26T17:56:30.472965+0000","last_became_active":"2024-04-26T17:53:49.768415+0000","last_became_peered":"2024-04-26T17:53:49.768415+0000","last_unstale":"2024-04-26T17:56:30.472965+0000","last_undegraded":"2024-04-26T17:56:30.472965+0000","last_fullsized":"2024-04-26T17:56:30.472965+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:57.364355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975670+0000","last_change":"2024-04-26T17:53:03.619193+0000","last_active":"2024-04-26T17:56:29.975670+0000","last_peered":"2024-04-26T17:56:29.975670+0000","last_clean":"2024-04-26T17:56:29.975670+0000","last_became_active":"2024-04-26T17:53:03.618776+0000","last_became_peered":"2024-04-26T17:53:03.618776+0000","last_unstale":"2024-04-26T17:56:29.975670+0000","last_undegraded":"2024-04-26T17:56:29.975670+0000","last_fullsized":"2024-04-26T17:56:29.975670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:07:24.811683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627751+0000","last_change":"2024-04-26T17:53:03.621366+0000","last_active":"2024-04-26T17:56:30.627751+0000","last_peered":"2024-04-26T17:56:30.627751+0000","last_clean":"2024-04-26T17:56:30.627751+0000","last_became_active":"2024-04-26T17:53:03.621048+0000","last_became_peered":"2024-04-26T17:53:03.621048+0000","last_unstale":"2024-04-26T17:56:30.627751+0000","last_undegraded":"2024-04-26T17:56:30.627751+0000","last_fullsized":"2024-04-26T17:56:30.627751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:31:06.484880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786409+0000","last_change":"2024-04-26T17:53:03.619383+0000","last_active":"2024-04-26T17:56:29.786409+0000","last_peered":"2024-04-26T17:56:29.786409+0000","last_clean":"2024-04-26T17:56:29.786409+0000","last_became_active":"2024-04-26T17:53:03.619017+0000","last_became_peered":"2024-04-26T17:53:03.619017+0000","last_unstale":"2024-04-26T17:56:29.786409+0000","last_undegraded":"2024-04-26T17:56:29.786409+0000","last_fullsized":"2024-04-26T17:56:29.786409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:20.091959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883534+0000","last_change":"2024-04-26T17:53:49.770695+0000","last_active":"2024-04-26T17:56:29.883534+0000","last_peered":"2024-04-26T17:56:29.883534+0000","last_clean":"2024-04-26T17:56:29.883534+0000","last_became_active":"2024-04-26T17:53:49.770303+0000","last_became_peered":"2024-04-26T17:53:49.770303+0000","last_unstale":"2024-04-26T17:56:29.883534+0000","last_undegraded":"2024-04-26T17:56:29.883534+0000","last_fullsized":"2024-04-26T17:56:29.883534+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:22.830129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891522+0000","last_change":"2024-04-26T17:53:03.605521+0000","last_active":"2024-04-26T17:56:29.891522+0000","last_peered":"2024-04-26T17:56:29.891522+0000","last_clean":"2024-04-26T17:56:29.891522+0000","last_became_active":"2024-04-26T17:53:03.605145+0000","last_became_peered":"2024-04-26T17:53:03.605145+0000","last_unstale":"2024-04-26T17:56:29.891522+0000","last_undegraded":"2024-04-26T17:56:29.891522+0000","last_fullsized":"2024-04-26T17:56:29.891522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:58:09.856021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882572+0000","last_change":"2024-04-26T17:53:03.610866+0000","last_active":"2024-04-26T17:56:29.882572+0000","last_peered":"2024-04-26T17:56:29.882572+0000","last_clean":"2024-04-26T17:56:29.882572+0000","last_became_active":"2024-04-26T17:53:03.610326+0000","last_became_peered":"2024-04-26T17:53:03.610326+0000","last_unstale":"2024-04-26T17:56:29.882572+0000","last_undegraded":"2024-04-26T17:56:29.882572+0000","last_fullsized":"2024-04-26T17:56:29.882572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:20.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046004+0000","last_change":"2024-04-26T17:53:03.615336+0000","last_active":"2024-04-26T17:56:30.046004+0000","last_peered":"2024-04-26T17:56:30.046004+0000","last_clean":"2024-04-26T17:56:30.046004+0000","last_became_active":"2024-04-26T17:53:03.614447+0000","last_became_peered":"2024-04-26T17:53:03.614447+0000","last_unstale":"2024-04-26T17:56:30.046004+0000","last_undegraded":"2024-04-26T17:56:30.046004+0000","last_fullsized":"2024-04-26T17:56:30.046004+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:36:10.091929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.976250+0000","last_change":"2024-04-26T17:53:03.614863+0000","last_active":"2024-04-26T17:56:29.976250+0000","last_peered":"2024-04-26T17:56:29.976250+0000","last_clean":"2024-04-26T17:56:29.976250+0000","last_became_active":"2024-04-26T17:53:03.614520+0000","last_became_peered":"2024-04-26T17:53:03.614520+0000","last_unstale":"2024-04-26T17:56:29.976250+0000","last_undegraded":"2024-04-26T17:56:29.976250+0000","last_fullsized":"2024-04-26T17:56:29.976250+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:51.367785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627720+0000","last_change":"2024-04-26T17:53:03.618521+0000","last_active":"2024-04-26T17:56:30.627720+0000","last_peered":"2024-04-26T17:56:30.627720+0000","last_clean":"2024-04-26T17:56:30.627720+0000","last_became_active":"2024-04-26T17:53:03.617658+0000","last_became_peered":"2024-04-26T17:53:03.617658+0000","last_unstale":"2024-04-26T17:56:30.627720+0000","last_undegraded":"2024-04-26T17:56:30.627720+0000","last_fullsized":"2024-04-26T17:56:30.627720+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:39.813098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106863,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":794668,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1522,"num_read_kb":968,"num_write":421,"num_write_kb":2354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":626688,"data_stored":2187372,"data_compressed":94284,"data_compressed_allocated":122880,"data_compressed_original":1916928,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383737,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":479802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645374,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":1274981,"data_compressed":44598,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808899,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":882152,"data_compressed":22211,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070535,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":882072,"data_compressed":22068,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332174,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":1082017,"data_compressed":38543,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495699,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":488325,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1064960,"data_stored":556106,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738481,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":489596,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1085440,"data_stored":552892,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182497,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1081344,"data_stored":1210084,"data_compressed":37413,"data_compressed_allocated":49152,"data_compressed_original":708608,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920861,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":489461,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757336,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1114112,"data_stored":1409234,"data_compressed":53745,"data_compressed_allocated":69632,"data_compressed_original":905216,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330631,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":201865,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T17:56:38.208 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T17:56:38.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T17:56:38.458 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:39.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:39 smithi031 ceph-mon[29164]: from='client.24974 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:56:39.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:39 smithi031 ceph-mon[29164]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-26T17:56:39.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:39 smithi097 ceph-mon[32419]: from='client.24974 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:56:39.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:39 smithi097 ceph-mon[32419]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-26T17:56:39.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:39 smithi186 ceph-mon[32797]: from='client.24974 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:56:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:39 smithi186 ceph-mon[32797]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-26T17:56:40.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.141+0000 7fa025840700 1 -- 172.21.15.31:0/2244149234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0201076a0 msgr2=0x7fa02010f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:40.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.141+0000 7fa025840700 1 --2- 172.21.15.31:0/2244149234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0201076a0 0x7fa02010f2b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa01800b0d0 tx=0x7fa01800b3e0 comp rx=0 tx=0).stop 2024-04-26T17:56:40.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.141+0000 7fa025840700 1 -- 172.21.15.31:0/2244149234 shutdown_connections 2024-04-26T17:56:40.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.141+0000 7fa025840700 1 --2- 172.21.15.31:0/2244149234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0201076a0 0x7fa02010f2b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:40.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.141+0000 7fa025840700 1 --2- 172.21.15.31:0/2244149234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa020102d80 0x7fa020103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:40.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.141+0000 7fa025840700 1 --2- 172.21.15.31:0/2244149234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa020103730 0x7fa020106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:40.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.141+0000 7fa025840700 1 -- 172.21.15.31:0/2244149234 >> 172.21.15.31:0/2244149234 conn(0x7fa020078990 msgr2=0x7fa0200ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:40.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.141+0000 7fa025840700 1 -- 172.21.15.31:0/2244149234 shutdown_connections 2024-04-26T17:56:40.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.141+0000 7fa025840700 1 -- 172.21.15.31:0/2244149234 wait complete. 2024-04-26T17:56:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.141+0000 7fa025840700 1 Processor -- start 2024-04-26T17:56:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.142+0000 7fa025840700 1 -- start start 2024-04-26T17:56:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.142+0000 7fa025840700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa020102d80 0x7fa0201a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.142+0000 7fa025840700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa020103730 0x7fa0201a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.142+0000 7fa025840700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0201076a0 0x7fa02010e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.142+0000 7fa025840700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa020101fe0 con 0x7fa020102d80 2024-04-26T17:56:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.142+0000 7fa025840700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa020101e60 con 0x7fa0201076a0 2024-04-26T17:56:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.142+0000 7fa025840700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa020102160 con 0x7fa020103730 2024-04-26T17:56:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.142+0000 7fa01effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa020102d80 0x7fa0201a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.142+0000 7fa01effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa020102d80 0x7fa0201a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34342/0 (socket says 172.21.15.31:34342) 2024-04-26T17:56:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.142+0000 7fa01effd700 1 -- 172.21.15.31:0/1604829864 learned_addr learned my addr 172.21.15.31:0/1604829864 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.142+0000 7fa01f7fe700 1 --2- 172.21.15.31:0/1604829864 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0201076a0 0x7fa02010e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.142+0000 7fa01e7fc700 1 --2- 172.21.15.31:0/1604829864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa020103730 0x7fa0201a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:40.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.143+0000 7fa01effd700 1 -- 172.21.15.31:0/1604829864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa020103730 msgr2=0x7fa0201a5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.143+0000 7fa01effd700 1 --2- 172.21.15.31:0/1604829864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa020103730 0x7fa0201a5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.143+0000 7fa01effd700 1 -- 172.21.15.31:0/1604829864 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0201076a0 msgr2=0x7fa02010e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.143+0000 7fa01effd700 1 --2- 172.21.15.31:0/1604829864 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0201076a0 0x7fa02010e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.143+0000 7fa01effd700 1 -- 172.21.15.31:0/1604829864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa02010e910 con 0x7fa020102d80 2024-04-26T17:56:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.143+0000 7fa01effd700 1 --2- 172.21.15.31:0/1604829864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa020102d80 0x7fa0201a4d00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa01000d950 tx=0x7fa01000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.143+0000 7fa02483e700 1 -- 172.21.15.31:0/1604829864 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa010016070 con 0x7fa020102d80 2024-04-26T17:56:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.143+0000 7fa02483e700 1 -- 172.21.15.31:0/1604829864 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa01000fd40 con 0x7fa020102d80 2024-04-26T17:56:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.143+0000 7fa025840700 1 -- 172.21.15.31:0/1604829864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa02010ec00 con 0x7fa020102d80 2024-04-26T17:56:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.143+0000 7fa02483e700 1 -- 172.21.15.31:0/1604829864 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01000e3b0 con 0x7fa020102d80 2024-04-26T17:56:40.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.143+0000 7fa025840700 1 -- 172.21.15.31:0/1604829864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0201ace00 con 0x7fa020102d80 2024-04-26T17:56:40.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.144+0000 7fa025840700 1 -- 172.21.15.31:0/1604829864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa020103b30 con 0x7fa020102d80 2024-04-26T17:56:40.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.146+0000 7fa02483e700 1 -- 172.21.15.31:0/1604829864 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fa010019070 con 0x7fa020102d80 2024-04-26T17:56:40.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.147+0000 7fa02483e700 1 --2- 172.21.15.31:0/1604829864 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa0080a3ec0 0x7fa0080a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:40.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.147+0000 7fa01e7fc700 1 --2- 172.21.15.31:0/1604829864 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa0080a3ec0 0x7fa0080a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:40.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.148+0000 7fa02483e700 1 -- 172.21.15.31:0/1604829864 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fa0100d0350 con 0x7fa020102d80 2024-04-26T17:56:40.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.148+0000 7fa01e7fc700 1 --2- 172.21.15.31:0/1604829864 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa0080a3ec0 0x7fa0080a6380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa02010f280 tx=0x7fa01400b410 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:40.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.151+0000 7fa02483e700 1 -- 172.21.15.31:0/1604829864 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa01009c8d0 con 0x7fa020102d80 2024-04-26T17:56:40.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.360+0000 7fa025840700 1 -- 172.21.15.31:0/1604829864 --> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa0201059c0 con 0x7fa0080a3ec0 2024-04-26T17:56:40.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.368+0000 7fa02483e700 1 -- 172.21.15.31:0/1604829864 <== mgr.24826 v2:172.21.15.31:6800/3595544994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258039 (secure 0 0 0) 0x7fa0201059c0 con 0x7fa0080a3ec0 2024-04-26T17:56:40.372 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:40.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.372+0000 7fa025840700 1 -- 172.21.15.31:0/1604829864 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa0080a3ec0 msgr2=0x7fa0080a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:40.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.372+0000 7fa025840700 1 --2- 172.21.15.31:0/1604829864 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa0080a3ec0 0x7fa0080a6380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa02010f280 tx=0x7fa01400b410 comp rx=0 tx=0).stop 2024-04-26T17:56:40.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.372+0000 7fa025840700 1 -- 172.21.15.31:0/1604829864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa020102d80 msgr2=0x7fa0201a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:40.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.372+0000 7fa025840700 1 --2- 172.21.15.31:0/1604829864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa020102d80 0x7fa0201a4d00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa01000d950 tx=0x7fa01000dc60 comp rx=0 tx=0).stop 2024-04-26T17:56:40.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.372+0000 7fa025840700 1 -- 172.21.15.31:0/1604829864 shutdown_connections 2024-04-26T17:56:40.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.372+0000 7fa025840700 1 --2- 172.21.15.31:0/1604829864 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa0080a3ec0 0x7fa0080a6380 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:40.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.372+0000 7fa025840700 1 --2- 172.21.15.31:0/1604829864 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0201076a0 0x7fa02010e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:40.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.372+0000 7fa025840700 1 --2- 172.21.15.31:0/1604829864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa020102d80 0x7fa0201a4d00 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:40.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.372+0000 7fa025840700 1 --2- 172.21.15.31:0/1604829864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa020103730 0x7fa0201a5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:40.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.372+0000 7fa025840700 1 -- 172.21.15.31:0/1604829864 >> 172.21.15.31:0/1604829864 conn(0x7fa020078990 msgr2=0x7fa0201052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:40.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.373+0000 7fa025840700 1 -- 172.21.15.31:0/1604829864 shutdown_connections 2024-04-26T17:56:40.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:40.373+0000 7fa025840700 1 -- 172.21.15.31:0/1604829864 wait complete. 2024-04-26T17:56:40.375 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-26T17:56:41.033 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-26T17:56:39.800265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1385270,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1775,"num_read_kb":1313,"num_write":624,"num_write_kb":5294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":959,"ondisk_log_size":959,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365732,"kb_used_data":12372,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576092,"statfs":{"total":1151940427776,"available":1151565918208,"internally_reserved":0,"allocated":12668928,"data_stored":10013226,"data_compressed":245400,"data_compressed_allocated":319488,"data_compressed_original":3969024,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":387,"num_read_kb":463,"num_write":146,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.993277"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627925+0000","last_change":"2024-04-26T17:53:03.616592+0000","last_active":"2024-04-26T17:56:30.627925+0000","last_peered":"2024-04-26T17:56:30.627925+0000","last_clean":"2024-04-26T17:56:30.627925+0000","last_became_active":"2024-04-26T17:53:03.616255+0000","last_became_peered":"2024-04-26T17:53:03.616255+0000","last_unstale":"2024-04-26T17:56:30.627925+0000","last_undegraded":"2024-04-26T17:56:30.627925+0000","last_fullsized":"2024-04-26T17:56:30.627925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:43:31.183679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472977+0000","last_change":"2024-04-26T17:53:03.615402+0000","last_active":"2024-04-26T17:56:30.472977+0000","last_peered":"2024-04-26T17:56:30.472977+0000","last_clean":"2024-04-26T17:56:30.472977+0000","last_became_active":"2024-04-26T17:53:03.615048+0000","last_became_peered":"2024-04-26T17:53:03.615048+0000","last_unstale":"2024-04-26T17:56:30.472977+0000","last_undegraded":"2024-04-26T17:56:30.472977+0000","last_fullsized":"2024-04-26T17:56:30.472977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:05:12.454770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472800+0000","last_change":"2024-04-26T17:53:03.622625+0000","last_active":"2024-04-26T17:56:30.472800+0000","last_peered":"2024-04-26T17:56:30.472800+0000","last_clean":"2024-04-26T17:56:30.472800+0000","last_became_active":"2024-04-26T17:53:03.622305+0000","last_became_peered":"2024-04-26T17:53:03.622305+0000","last_unstale":"2024-04-26T17:56:30.472800+0000","last_undegraded":"2024-04-26T17:56:30.472800+0000","last_fullsized":"2024-04-26T17:56:30.472800+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:06.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.784761+0000","last_change":"2024-04-26T17:53:03.617650+0000","last_active":"2024-04-26T17:56:29.784761+0000","last_peered":"2024-04-26T17:56:29.784761+0000","last_clean":"2024-04-26T17:56:29.784761+0000","last_became_active":"2024-04-26T17:53:03.617220+0000","last_became_peered":"2024-04-26T17:53:03.617220+0000","last_unstale":"2024-04-26T17:56:29.784761+0000","last_undegraded":"2024-04-26T17:56:29.784761+0000","last_fullsized":"2024-04-26T17:56:29.784761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:31.765577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882813+0000","last_change":"2024-04-26T17:53:03.615429+0000","last_active":"2024-04-26T17:56:29.882813+0000","last_peered":"2024-04-26T17:56:29.882813+0000","last_clean":"2024-04-26T17:56:29.882813+0000","last_became_active":"2024-04-26T17:53:03.615079+0000","last_became_peered":"2024-04-26T17:53:03.615079+0000","last_unstale":"2024-04-26T17:56:29.882813+0000","last_undegraded":"2024-04-26T17:56:29.882813+0000","last_fullsized":"2024-04-26T17:56:29.882813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:20.994016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894485+0000","last_change":"2024-04-26T17:53:03.624619+0000","last_active":"2024-04-26T17:56:29.894485+0000","last_peered":"2024-04-26T17:56:29.894485+0000","last_clean":"2024-04-26T17:56:29.894485+0000","last_became_active":"2024-04-26T17:53:03.624251+0000","last_became_peered":"2024-04-26T17:53:03.624251+0000","last_unstale":"2024-04-26T17:56:29.894485+0000","last_undegraded":"2024-04-26T17:56:29.894485+0000","last_fullsized":"2024-04-26T17:56:29.894485+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:50:17.535235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046789+0000","last_change":"2024-04-26T17:53:03.619325+0000","last_active":"2024-04-26T17:56:30.046789+0000","last_peered":"2024-04-26T17:56:30.046789+0000","last_clean":"2024-04-26T17:56:30.046789+0000","last_became_active":"2024-04-26T17:53:03.618989+0000","last_became_peered":"2024-04-26T17:53:03.618989+0000","last_unstale":"2024-04-26T17:56:30.046789+0000","last_undegraded":"2024-04-26T17:56:30.046789+0000","last_fullsized":"2024-04-26T17:56:30.046789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:10.195742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882506+0000","last_change":"2024-04-26T17:53:03.609346+0000","last_active":"2024-04-26T17:56:29.882506+0000","last_peered":"2024-04-26T17:56:29.882506+0000","last_clean":"2024-04-26T17:56:29.882506+0000","last_became_active":"2024-04-26T17:53:03.609021+0000","last_became_peered":"2024-04-26T17:53:03.609021+0000","last_unstale":"2024-04-26T17:56:29.882506+0000","last_undegraded":"2024-04-26T17:56:29.882506+0000","last_fullsized":"2024-04-26T17:56:29.882506+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:44.328599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046813+0000","last_change":"2024-04-26T17:53:03.618573+0000","last_active":"2024-04-26T17:56:30.046813+0000","last_peered":"2024-04-26T17:56:30.046813+0000","last_clean":"2024-04-26T17:56:30.046813+0000","last_became_active":"2024-04-26T17:53:03.618223+0000","last_became_peered":"2024-04-26T17:53:03.618223+0000","last_unstale":"2024-04-26T17:56:30.046813+0000","last_undegraded":"2024-04-26T17:56:30.046813+0000","last_fullsized":"2024-04-26T17:56:30.046813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:16.749205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894921+0000","last_change":"2024-04-26T17:53:03.621530+0000","last_active":"2024-04-26T17:56:29.894921+0000","last_peered":"2024-04-26T17:56:29.894921+0000","last_clean":"2024-04-26T17:56:29.894921+0000","last_became_active":"2024-04-26T17:53:03.621174+0000","last_became_peered":"2024-04-26T17:53:03.621174+0000","last_unstale":"2024-04-26T17:56:29.894921+0000","last_undegraded":"2024-04-26T17:56:29.894921+0000","last_fullsized":"2024-04-26T17:56:29.894921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:31.388232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047169+0000","last_change":"2024-04-26T17:53:03.617682+0000","last_active":"2024-04-26T17:56:30.047169+0000","last_peered":"2024-04-26T17:56:30.047169+0000","last_clean":"2024-04-26T17:56:30.047169+0000","last_became_active":"2024-04-26T17:53:03.617348+0000","last_became_peered":"2024-04-26T17:53:03.617348+0000","last_unstale":"2024-04-26T17:56:30.047169+0000","last_undegraded":"2024-04-26T17:56:30.047169+0000","last_fullsized":"2024-04-26T17:56:30.047169+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:21.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.784690+0000","last_change":"2024-04-26T17:53:03.617955+0000","last_active":"2024-04-26T17:56:29.784690+0000","last_peered":"2024-04-26T17:56:29.784690+0000","last_clean":"2024-04-26T17:56:29.784690+0000","last_became_active":"2024-04-26T17:53:03.616765+0000","last_became_peered":"2024-04-26T17:53:03.616765+0000","last_unstale":"2024-04-26T17:56:29.784690+0000","last_undegraded":"2024-04-26T17:56:29.784690+0000","last_fullsized":"2024-04-26T17:56:29.784690+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:59.292277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.976094+0000","last_change":"2024-04-26T17:53:49.767807+0000","last_active":"2024-04-26T17:56:29.976094+0000","last_peered":"2024-04-26T17:56:29.976094+0000","last_clean":"2024-04-26T17:56:29.976094+0000","last_became_active":"2024-04-26T17:53:49.767404+0000","last_became_peered":"2024-04-26T17:53:49.767404+0000","last_unstale":"2024-04-26T17:56:29.976094+0000","last_undegraded":"2024-04-26T17:56:29.976094+0000","last_fullsized":"2024-04-26T17:56:29.976094+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:20:45.482580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472587+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:56:30.472587+0000","last_peered":"2024-04-26T17:56:30.472587+0000","last_clean":"2024-04-26T17:56:30.472587+0000","last_became_active":"2024-04-26T17:53:03.616623+0000","last_became_peered":"2024-04-26T17:53:03.616623+0000","last_unstale":"2024-04-26T17:56:30.472587+0000","last_undegraded":"2024-04-26T17:56:30.472587+0000","last_fullsized":"2024-04-26T17:56:30.472587+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:04.231549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472552+0000","last_change":"2024-04-26T17:53:03.622147+0000","last_active":"2024-04-26T17:56:30.472552+0000","last_peered":"2024-04-26T17:56:30.472552+0000","last_clean":"2024-04-26T17:56:30.472552+0000","last_became_active":"2024-04-26T17:53:03.621542+0000","last_became_peered":"2024-04-26T17:53:03.621542+0000","last_unstale":"2024-04-26T17:56:30.472552+0000","last_undegraded":"2024-04-26T17:56:30.472552+0000","last_fullsized":"2024-04-26T17:56:30.472552+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:01.746954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.045357+0000","last_change":"2024-04-26T17:53:03.624433+0000","last_active":"2024-04-26T17:56:30.045357+0000","last_peered":"2024-04-26T17:56:30.045357+0000","last_clean":"2024-04-26T17:56:30.045357+0000","last_became_active":"2024-04-26T17:53:03.624135+0000","last_became_peered":"2024-04-26T17:53:03.624135+0000","last_unstale":"2024-04-26T17:56:30.045357+0000","last_undegraded":"2024-04-26T17:56:30.045357+0000","last_fullsized":"2024-04-26T17:56:30.045357+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:32:20.521917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.628025+0000","last_change":"2024-04-26T17:53:03.623607+0000","last_active":"2024-04-26T17:56:30.628025+0000","last_peered":"2024-04-26T17:56:30.628025+0000","last_clean":"2024-04-26T17:56:30.628025+0000","last_became_active":"2024-04-26T17:53:03.621783+0000","last_became_peered":"2024-04-26T17:53:03.621783+0000","last_unstale":"2024-04-26T17:56:30.628025+0000","last_undegraded":"2024-04-26T17:56:30.628025+0000","last_fullsized":"2024-04-26T17:56:30.628025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:38:59.872868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783567+0000","last_change":"2024-04-26T17:53:03.614974+0000","last_active":"2024-04-26T17:56:29.783567+0000","last_peered":"2024-04-26T17:56:29.783567+0000","last_clean":"2024-04-26T17:56:29.783567+0000","last_became_active":"2024-04-26T17:53:03.614619+0000","last_became_peered":"2024-04-26T17:53:03.614619+0000","last_unstale":"2024-04-26T17:56:29.783567+0000","last_undegraded":"2024-04-26T17:56:29.783567+0000","last_fullsized":"2024-04-26T17:56:29.783567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:50.729531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786362+0000","last_change":"2024-04-26T17:53:03.611140+0000","last_active":"2024-04-26T17:56:29.786362+0000","last_peered":"2024-04-26T17:56:29.786362+0000","last_clean":"2024-04-26T17:56:29.786362+0000","last_became_active":"2024-04-26T17:53:03.610549+0000","last_became_peered":"2024-04-26T17:53:03.610549+0000","last_unstale":"2024-04-26T17:56:29.786362+0000","last_undegraded":"2024-04-26T17:56:29.786362+0000","last_fullsized":"2024-04-26T17:56:29.786362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:46:17.178598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783478+0000","last_change":"2024-04-26T17:53:00.585971+0000","last_active":"2024-04-26T17:56:29.783478+0000","last_peered":"2024-04-26T17:56:29.783478+0000","last_clean":"2024-04-26T17:56:29.783478+0000","last_became_active":"2024-04-26T17:53:00.585648+0000","last_became_peered":"2024-04-26T17:53:00.585648+0000","last_unstale":"2024-04-26T17:56:29.783478+0000","last_undegraded":"2024-04-26T17:56:29.783478+0000","last_fullsized":"2024-04-26T17:56:29.783478+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:00:24.082773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783479+0000","last_change":"2024-04-26T17:53:49.770579+0000","last_active":"2024-04-26T17:56:29.783479+0000","last_peered":"2024-04-26T17:56:29.783479+0000","last_clean":"2024-04-26T17:56:29.783479+0000","last_became_active":"2024-04-26T17:53:49.770250+0000","last_became_peered":"2024-04-26T17:53:49.770250+0000","last_unstale":"2024-04-26T17:56:29.783479+0000","last_undegraded":"2024-04-26T17:56:29.783479+0000","last_fullsized":"2024-04-26T17:56:29.783479+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:35.758790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883544+0000","last_change":"2024-04-26T17:53:01.145438+0000","last_active":"2024-04-26T17:56:29.883544+0000","last_peered":"2024-04-26T17:56:29.883544+0000","last_clean":"2024-04-26T17:56:29.883544+0000","last_became_active":"2024-04-26T17:53:01.145030+0000","last_became_peered":"2024-04-26T17:53:01.145030+0000","last_unstale":"2024-04-26T17:56:29.883544+0000","last_undegraded":"2024-04-26T17:56:29.883544+0000","last_fullsized":"2024-04-26T17:56:29.883544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:15.650247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472912+0000","last_change":"2024-04-26T17:53:03.622647+0000","last_active":"2024-04-26T17:56:30.472912+0000","last_peered":"2024-04-26T17:56:30.472912+0000","last_clean":"2024-04-26T17:56:30.472912+0000","last_became_active":"2024-04-26T17:53:03.622298+0000","last_became_peered":"2024-04-26T17:53:03.622298+0000","last_unstale":"2024-04-26T17:56:30.472912+0000","last_undegraded":"2024-04-26T17:56:30.472912+0000","last_fullsized":"2024-04-26T17:56:30.472912+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:37.230539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627763+0000","last_change":"2024-04-26T17:53:49.775405+0000","last_active":"2024-04-26T17:56:30.627763+0000","last_peered":"2024-04-26T17:56:30.627763+0000","last_clean":"2024-04-26T17:56:30.627763+0000","last_became_active":"2024-04-26T17:53:49.770536+0000","last_became_peered":"2024-04-26T17:53:49.770536+0000","last_unstale":"2024-04-26T17:56:30.627763+0000","last_undegraded":"2024-04-26T17:56:30.627763+0000","last_fullsized":"2024-04-26T17:56:30.627763+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:34.643452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.045992+0000","last_change":"2024-04-26T17:53:03.617462+0000","last_active":"2024-04-26T17:56:30.045992+0000","last_peered":"2024-04-26T17:56:30.045992+0000","last_clean":"2024-04-26T17:56:30.045992+0000","last_became_active":"2024-04-26T17:53:03.617115+0000","last_became_peered":"2024-04-26T17:53:03.617115+0000","last_unstale":"2024-04-26T17:56:30.045992+0000","last_undegraded":"2024-04-26T17:56:30.045992+0000","last_fullsized":"2024-04-26T17:56:30.045992+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:11.406146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883071+0000","last_change":"2024-04-26T17:53:00.583378+0000","last_active":"2024-04-26T17:56:29.883071+0000","last_peered":"2024-04-26T17:56:29.883071+0000","last_clean":"2024-04-26T17:56:29.883071+0000","last_became_active":"2024-04-26T17:53:00.583020+0000","last_became_peered":"2024-04-26T17:53:00.583020+0000","last_unstale":"2024-04-26T17:56:29.883071+0000","last_undegraded":"2024-04-26T17:56:29.883071+0000","last_fullsized":"2024-04-26T17:56:29.883071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:03:24.089996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891803+0000","last_change":"2024-04-26T17:53:03.609557+0000","last_active":"2024-04-26T17:56:29.891803+0000","last_peered":"2024-04-26T17:56:29.891803+0000","last_clean":"2024-04-26T17:56:29.891803+0000","last_became_active":"2024-04-26T17:53:03.609028+0000","last_became_peered":"2024-04-26T17:53:03.609028+0000","last_unstale":"2024-04-26T17:56:29.891803+0000","last_undegraded":"2024-04-26T17:56:29.891803+0000","last_fullsized":"2024-04-26T17:56:29.891803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:39:22.860877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'8","reported_seq":235,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:33.990001+0000","last_change":"2024-04-26T17:53:00.586683+0000","last_active":"2024-04-26T17:56:33.990001+0000","last_peered":"2024-04-26T17:56:33.990001+0000","last_clean":"2024-04-26T17:56:33.990001+0000","last_became_active":"2024-04-26T17:53:00.586272+0000","last_became_peered":"2024-04-26T17:53:00.586272+0000","last_unstale":"2024-04-26T17:56:33.990001+0000","last_undegraded":"2024-04-26T17:56:33.990001+0000","last_fullsized":"2024-04-26T17:56:33.990001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:20:29.068607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882791+0000","last_change":"2024-04-26T17:53:03.611073+0000","last_active":"2024-04-26T17:56:29.882791+0000","last_peered":"2024-04-26T17:56:29.882791+0000","last_clean":"2024-04-26T17:56:29.882791+0000","last_became_active":"2024-04-26T17:53:03.610701+0000","last_became_peered":"2024-04-26T17:53:03.610701+0000","last_unstale":"2024-04-26T17:56:29.882791+0000","last_undegraded":"2024-04-26T17:56:29.882791+0000","last_fullsized":"2024-04-26T17:56:29.882791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:01:00.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894392+0000","last_change":"2024-04-26T17:53:00.586082+0000","last_active":"2024-04-26T17:56:29.894392+0000","last_peered":"2024-04-26T17:56:29.894392+0000","last_clean":"2024-04-26T17:56:29.894392+0000","last_became_active":"2024-04-26T17:53:00.585678+0000","last_became_peered":"2024-04-26T17:53:00.585678+0000","last_unstale":"2024-04-26T17:56:29.894392+0000","last_undegraded":"2024-04-26T17:56:29.894392+0000","last_fullsized":"2024-04-26T17:56:29.894392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:55:25.523846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891547+0000","last_change":"2024-04-26T17:53:03.600230+0000","last_active":"2024-04-26T17:56:29.891547+0000","last_peered":"2024-04-26T17:56:29.891547+0000","last_clean":"2024-04-26T17:56:29.891547+0000","last_became_active":"2024-04-26T17:53:03.599713+0000","last_became_peered":"2024-04-26T17:53:03.599713+0000","last_unstale":"2024-04-26T17:56:29.891547+0000","last_undegraded":"2024-04-26T17:56:29.891547+0000","last_fullsized":"2024-04-26T17:56:29.891547+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:45.822561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975577+0000","last_change":"2024-04-26T17:53:00.584588+0000","last_active":"2024-04-26T17:56:29.975577+0000","last_peered":"2024-04-26T17:56:29.975577+0000","last_clean":"2024-04-26T17:56:29.975577+0000","last_became_active":"2024-04-26T17:53:00.584284+0000","last_became_peered":"2024-04-26T17:53:00.584284+0000","last_unstale":"2024-04-26T17:56:29.975577+0000","last_undegraded":"2024-04-26T17:56:29.975577+0000","last_fullsized":"2024-04-26T17:56:29.975577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:08.428904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882589+0000","last_change":"2024-04-26T17:53:03.615237+0000","last_active":"2024-04-26T17:56:29.882589+0000","last_peered":"2024-04-26T17:56:29.882589+0000","last_clean":"2024-04-26T17:56:29.882589+0000","last_became_active":"2024-04-26T17:53:03.614704+0000","last_became_peered":"2024-04-26T17:53:03.614704+0000","last_unstale":"2024-04-26T17:56:29.882589+0000","last_undegraded":"2024-04-26T17:56:29.882589+0000","last_fullsized":"2024-04-26T17:56:29.882589+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:27.251839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891446+0000","last_change":"2024-04-26T17:53:00.589287+0000","last_active":"2024-04-26T17:56:29.891446+0000","last_peered":"2024-04-26T17:56:29.891446+0000","last_clean":"2024-04-26T17:56:29.891446+0000","last_became_active":"2024-04-26T17:53:00.588119+0000","last_became_peered":"2024-04-26T17:53:00.588119+0000","last_unstale":"2024-04-26T17:56:29.891446+0000","last_undegraded":"2024-04-26T17:56:29.891446+0000","last_fullsized":"2024-04-26T17:56:29.891446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:25.389777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783241+0000","last_change":"2024-04-26T17:53:03.608061+0000","last_active":"2024-04-26T17:56:29.783241+0000","last_peered":"2024-04-26T17:56:29.783241+0000","last_clean":"2024-04-26T17:56:29.783241+0000","last_became_active":"2024-04-26T17:53:03.607624+0000","last_became_peered":"2024-04-26T17:53:03.607624+0000","last_unstale":"2024-04-26T17:56:29.783241+0000","last_undegraded":"2024-04-26T17:56:29.783241+0000","last_fullsized":"2024-04-26T17:56:29.783241+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:15.111011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":227,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:33.989483+0000","last_change":"2024-04-26T17:53:00.588621+0000","last_active":"2024-04-26T17:56:33.989483+0000","last_peered":"2024-04-26T17:56:33.989483+0000","last_clean":"2024-04-26T17:56:33.989483+0000","last_became_active":"2024-04-26T17:53:00.587330+0000","last_became_peered":"2024-04-26T17:53:00.587330+0000","last_unstale":"2024-04-26T17:56:33.989483+0000","last_undegraded":"2024-04-26T17:56:33.989483+0000","last_fullsized":"2024-04-26T17:56:33.989483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:01:12.178610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":193,"num_read_kb":193,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046606+0000","last_change":"2024-04-26T17:53:03.615723+0000","last_active":"2024-04-26T17:56:30.046606+0000","last_peered":"2024-04-26T17:56:30.046606+0000","last_clean":"2024-04-26T17:56:30.046606+0000","last_became_active":"2024-04-26T17:53:03.614931+0000","last_became_peered":"2024-04-26T17:53:03.614931+0000","last_unstale":"2024-04-26T17:56:30.046606+0000","last_undegraded":"2024-04-26T17:56:30.046606+0000","last_fullsized":"2024-04-26T17:56:30.046606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:09.663876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047232+0000","last_change":"2024-04-26T17:53:00.589232+0000","last_active":"2024-04-26T17:56:30.047232+0000","last_peered":"2024-04-26T17:56:30.047232+0000","last_clean":"2024-04-26T17:56:30.047232+0000","last_became_active":"2024-04-26T17:53:00.588863+0000","last_became_peered":"2024-04-26T17:53:00.588863+0000","last_unstale":"2024-04-26T17:56:30.047232+0000","last_undegraded":"2024-04-26T17:56:30.047232+0000","last_fullsized":"2024-04-26T17:56:30.047232+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:14.360782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472784+0000","last_change":"2024-04-26T17:53:03.624923+0000","last_active":"2024-04-26T17:56:30.472784+0000","last_peered":"2024-04-26T17:56:30.472784+0000","last_clean":"2024-04-26T17:56:30.472784+0000","last_became_active":"2024-04-26T17:53:03.624620+0000","last_became_peered":"2024-04-26T17:53:03.624620+0000","last_unstale":"2024-04-26T17:56:30.472784+0000","last_undegraded":"2024-04-26T17:56:30.472784+0000","last_fullsized":"2024-04-26T17:56:30.472784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:50:51.259563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891963+0000","last_change":"2024-04-26T17:53:01.146950+0000","last_active":"2024-04-26T17:56:29.891963+0000","last_peered":"2024-04-26T17:56:29.891963+0000","last_clean":"2024-04-26T17:56:29.891963+0000","last_became_active":"2024-04-26T17:53:01.146591+0000","last_became_peered":"2024-04-26T17:53:01.146591+0000","last_unstale":"2024-04-26T17:56:29.891963+0000","last_undegraded":"2024-04-26T17:56:29.891963+0000","last_fullsized":"2024-04-26T17:56:29.891963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:17.756660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046019+0000","last_change":"2024-04-26T17:53:03.626944+0000","last_active":"2024-04-26T17:56:30.046019+0000","last_peered":"2024-04-26T17:56:30.046019+0000","last_clean":"2024-04-26T17:56:30.046019+0000","last_became_active":"2024-04-26T17:53:03.626594+0000","last_became_peered":"2024-04-26T17:53:03.626594+0000","last_unstale":"2024-04-26T17:56:30.046019+0000","last_undegraded":"2024-04-26T17:56:30.046019+0000","last_fullsized":"2024-04-26T17:56:30.046019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:52:42.997251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'37","reported_seq":464,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:34.991009+0000","last_change":"2024-04-26T17:53:00.588353+0000","last_active":"2024-04-26T17:56:34.991009+0000","last_peered":"2024-04-26T17:56:34.991009+0000","last_clean":"2024-04-26T17:56:34.991009+0000","last_became_active":"2024-04-26T17:53:00.586351+0000","last_became_peered":"2024-04-26T17:53:00.586351+0000","last_unstale":"2024-04-26T17:56:34.991009+0000","last_undegraded":"2024-04-26T17:56:34.991009+0000","last_fullsized":"2024-04-26T17:56:34.991009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:41.660525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67988,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":398,"num_read_kb":265,"num_write":38,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883217+0000","last_change":"2024-04-26T17:53:03.612052+0000","last_active":"2024-04-26T17:56:29.883217+0000","last_peered":"2024-04-26T17:56:29.883217+0000","last_clean":"2024-04-26T17:56:29.883217+0000","last_became_active":"2024-04-26T17:53:03.611733+0000","last_became_peered":"2024-04-26T17:53:03.611733+0000","last_unstale":"2024-04-26T17:56:29.883217+0000","last_undegraded":"2024-04-26T17:56:29.883217+0000","last_fullsized":"2024-04-26T17:56:29.883217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:05:23.958317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786202+0000","last_change":"2024-04-26T17:53:00.588284+0000","last_active":"2024-04-26T17:56:29.786202+0000","last_peered":"2024-04-26T17:56:29.786202+0000","last_clean":"2024-04-26T17:56:29.786202+0000","last_became_active":"2024-04-26T17:53:00.587942+0000","last_became_peered":"2024-04-26T17:53:00.587942+0000","last_unstale":"2024-04-26T17:56:29.786202+0000","last_undegraded":"2024-04-26T17:56:29.786202+0000","last_fullsized":"2024-04-26T17:56:29.786202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:43.782971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975582+0000","last_change":"2024-04-26T17:53:03.613092+0000","last_active":"2024-04-26T17:56:29.975582+0000","last_peered":"2024-04-26T17:56:29.975582+0000","last_clean":"2024-04-26T17:56:29.975582+0000","last_became_active":"2024-04-26T17:53:03.612773+0000","last_became_peered":"2024-04-26T17:53:03.612773+0000","last_unstale":"2024-04-26T17:56:29.975582+0000","last_undegraded":"2024-04-26T17:56:29.975582+0000","last_fullsized":"2024-04-26T17:56:29.975582+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:41:21.340375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891560+0000","last_change":"2024-04-26T17:53:00.588777+0000","last_active":"2024-04-26T17:56:29.891560+0000","last_peered":"2024-04-26T17:56:29.891560+0000","last_clean":"2024-04-26T17:56:29.891560+0000","last_became_active":"2024-04-26T17:53:00.588380+0000","last_became_peered":"2024-04-26T17:53:00.588380+0000","last_unstale":"2024-04-26T17:56:29.891560+0000","last_undegraded":"2024-04-26T17:56:29.891560+0000","last_fullsized":"2024-04-26T17:56:29.891560+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:44.587381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047058+0000","last_change":"2024-04-26T17:53:03.620344+0000","last_active":"2024-04-26T17:56:30.047058+0000","last_peered":"2024-04-26T17:56:30.047058+0000","last_clean":"2024-04-26T17:56:30.047058+0000","last_became_active":"2024-04-26T17:53:03.619944+0000","last_became_peered":"2024-04-26T17:53:03.619944+0000","last_unstale":"2024-04-26T17:56:30.047058+0000","last_undegraded":"2024-04-26T17:56:30.047058+0000","last_fullsized":"2024-04-26T17:56:30.047058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:08:31.801342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":231,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:33.983294+0000","last_change":"2024-04-26T17:53:00.587733+0000","last_active":"2024-04-26T17:56:33.983294+0000","last_peered":"2024-04-26T17:56:33.983294+0000","last_clean":"2024-04-26T17:56:33.983294+0000","last_became_active":"2024-04-26T17:53:00.587373+0000","last_became_peered":"2024-04-26T17:53:00.587373+0000","last_unstale":"2024-04-26T17:56:33.983294+0000","last_undegraded":"2024-04-26T17:56:33.983294+0000","last_fullsized":"2024-04-26T17:56:33.983294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:10.978944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1287,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046569+0000","last_change":"2024-04-26T17:53:03.616915+0000","last_active":"2024-04-26T17:56:30.046569+0000","last_peered":"2024-04-26T17:56:30.046569+0000","last_clean":"2024-04-26T17:56:30.046569+0000","last_became_active":"2024-04-26T17:53:03.616588+0000","last_became_peered":"2024-04-26T17:53:03.616588+0000","last_unstale":"2024-04-26T17:56:30.046569+0000","last_undegraded":"2024-04-26T17:56:30.046569+0000","last_fullsized":"2024-04-26T17:56:30.046569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:46:38.419613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882753+0000","last_change":"2024-04-26T17:53:01.146922+0000","last_active":"2024-04-26T17:56:29.882753+0000","last_peered":"2024-04-26T17:56:29.882753+0000","last_clean":"2024-04-26T17:56:29.882753+0000","last_became_active":"2024-04-26T17:53:01.146520+0000","last_became_peered":"2024-04-26T17:53:01.146520+0000","last_unstale":"2024-04-26T17:56:29.882753+0000","last_undegraded":"2024-04-26T17:56:29.882753+0000","last_fullsized":"2024-04-26T17:56:29.882753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:12:49.238012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891565+0000","last_change":"2024-04-26T17:53:03.608513+0000","last_active":"2024-04-26T17:56:29.891565+0000","last_peered":"2024-04-26T17:56:29.891565+0000","last_clean":"2024-04-26T17:56:29.891565+0000","last_became_active":"2024-04-26T17:53:03.608167+0000","last_became_peered":"2024-04-26T17:53:03.608167+0000","last_unstale":"2024-04-26T17:56:29.891565+0000","last_undegraded":"2024-04-26T17:56:29.891565+0000","last_fullsized":"2024-04-26T17:56:29.891565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:03.002756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883302+0000","last_change":"2024-04-26T17:53:00.583878+0000","last_active":"2024-04-26T17:56:29.883302+0000","last_peered":"2024-04-26T17:56:29.883302+0000","last_clean":"2024-04-26T17:56:29.883302+0000","last_became_active":"2024-04-26T17:53:00.583159+0000","last_became_peered":"2024-04-26T17:53:00.583159+0000","last_unstale":"2024-04-26T17:56:29.883302+0000","last_undegraded":"2024-04-26T17:56:29.883302+0000","last_fullsized":"2024-04-26T17:56:29.883302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:32:49.850942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627754+0000","last_change":"2024-04-26T17:53:03.614467+0000","last_active":"2024-04-26T17:56:30.627754+0000","last_peered":"2024-04-26T17:56:30.627754+0000","last_clean":"2024-04-26T17:56:30.627754+0000","last_became_active":"2024-04-26T17:53:03.613972+0000","last_became_peered":"2024-04-26T17:53:03.613972+0000","last_unstale":"2024-04-26T17:56:30.627754+0000","last_undegraded":"2024-04-26T17:56:30.627754+0000","last_fullsized":"2024-04-26T17:56:30.627754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:27.259777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.783415+0000","last_change":"2024-04-26T17:53:49.767174+0000","last_active":"2024-04-26T17:56:29.783415+0000","last_peered":"2024-04-26T17:56:29.783415+0000","last_clean":"2024-04-26T17:56:29.783415+0000","last_became_active":"2024-04-26T17:53:49.766778+0000","last_became_peered":"2024-04-26T17:53:49.766778+0000","last_unstale":"2024-04-26T17:56:29.783415+0000","last_undegraded":"2024-04-26T17:56:29.783415+0000","last_fullsized":"2024-04-26T17:56:29.783415+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:59.240159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894614+0000","last_change":"2024-04-26T17:53:03.613888+0000","last_active":"2024-04-26T17:56:29.894614+0000","last_peered":"2024-04-26T17:56:29.894614+0000","last_clean":"2024-04-26T17:56:29.894614+0000","last_became_active":"2024-04-26T17:53:03.613499+0000","last_became_peered":"2024-04-26T17:53:03.613499+0000","last_unstale":"2024-04-26T17:56:29.894614+0000","last_undegraded":"2024-04-26T17:56:29.894614+0000","last_fullsized":"2024-04-26T17:56:29.894614+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:15.549419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047067+0000","last_change":"2024-04-26T17:53:00.587803+0000","last_active":"2024-04-26T17:56:30.047067+0000","last_peered":"2024-04-26T17:56:30.047067+0000","last_clean":"2024-04-26T17:56:30.047067+0000","last_became_active":"2024-04-26T17:53:00.587382+0000","last_became_peered":"2024-04-26T17:53:00.587382+0000","last_unstale":"2024-04-26T17:56:30.047067+0000","last_undegraded":"2024-04-26T17:56:30.047067+0000","last_fullsized":"2024-04-26T17:56:30.047067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:38.249478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.894610+0000","last_change":"2024-04-26T17:53:49.769008+0000","last_active":"2024-04-26T17:56:29.894610+0000","last_peered":"2024-04-26T17:56:29.894610+0000","last_clean":"2024-04-26T17:56:29.894610+0000","last_became_active":"2024-04-26T17:53:49.768625+0000","last_became_peered":"2024-04-26T17:53:49.768625+0000","last_unstale":"2024-04-26T17:56:29.894610+0000","last_undegraded":"2024-04-26T17:56:29.894610+0000","last_fullsized":"2024-04-26T17:56:29.894610+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:27:46.412625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891725+0000","last_change":"2024-04-26T17:53:03.612899+0000","last_active":"2024-04-26T17:56:29.891725+0000","last_peered":"2024-04-26T17:56:29.891725+0000","last_clean":"2024-04-26T17:56:29.891725+0000","last_became_active":"2024-04-26T17:53:03.612564+0000","last_became_peered":"2024-04-26T17:53:03.612564+0000","last_unstale":"2024-04-26T17:56:29.891725+0000","last_undegraded":"2024-04-26T17:56:29.891725+0000","last_fullsized":"2024-04-26T17:56:29.891725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:07.368256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":227,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:33.982666+0000","last_change":"2024-04-26T17:53:01.145471+0000","last_active":"2024-04-26T17:56:33.982666+0000","last_peered":"2024-04-26T17:56:33.982666+0000","last_clean":"2024-04-26T17:56:33.982666+0000","last_became_active":"2024-04-26T17:53:01.145040+0000","last_became_peered":"2024-04-26T17:53:01.145040+0000","last_unstale":"2024-04-26T17:56:33.982666+0000","last_undegraded":"2024-04-26T17:56:33.982666+0000","last_fullsized":"2024-04-26T17:56:33.982666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:46:27.437329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":193,"num_read_kb":193,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882767+0000","last_change":"2024-04-26T17:53:03.618600+0000","last_active":"2024-04-26T17:56:29.882767+0000","last_peered":"2024-04-26T17:56:29.882767+0000","last_clean":"2024-04-26T17:56:29.882767+0000","last_became_active":"2024-04-26T17:53:03.618139+0000","last_became_peered":"2024-04-26T17:53:03.618139+0000","last_unstale":"2024-04-26T17:56:29.882767+0000","last_undegraded":"2024-04-26T17:56:29.882767+0000","last_fullsized":"2024-04-26T17:56:29.882767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:42:56.007659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.784624+0000","last_change":"2024-04-26T17:53:00.586386+0000","last_active":"2024-04-26T17:56:29.784624+0000","last_peered":"2024-04-26T17:56:29.784624+0000","last_clean":"2024-04-26T17:56:29.784624+0000","last_became_active":"2024-04-26T17:53:00.585953+0000","last_became_peered":"2024-04-26T17:53:00.585953+0000","last_unstale":"2024-04-26T17:56:29.784624+0000","last_undegraded":"2024-04-26T17:56:29.784624+0000","last_fullsized":"2024-04-26T17:56:29.784624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:39:57.160983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"91'148","reported_seq":405,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:31.909946+0000","last_change":"2024-04-26T17:51:38.839853+0000","last_active":"2024-04-26T17:56:31.909946+0000","last_peered":"2024-04-26T17:56:31.909946+0000","last_clean":"2024-04-26T17:56:31.909946+0000","last_became_active":"2024-04-26T17:51:38.839318+0000","last_became_peered":"2024-04-26T17:51:38.839318+0000","last_unstale":"2024-04-26T17:56:31.909946+0000","last_undegraded":"2024-04-26T17:56:31.909946+0000","last_fullsized":"2024-04-26T17:56:31.909946+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:48:12.577543+0000","last_clean_scrub_stamp":"2024-04-26T17:48:12.577543+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:08:50.683663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975406+0000","last_change":"2024-04-26T17:53:03.614170+0000","last_active":"2024-04-26T17:56:29.975406+0000","last_peered":"2024-04-26T17:56:29.975406+0000","last_clean":"2024-04-26T17:56:29.975406+0000","last_became_active":"2024-04-26T17:53:03.613836+0000","last_became_peered":"2024-04-26T17:53:03.613836+0000","last_unstale":"2024-04-26T17:56:29.975406+0000","last_undegraded":"2024-04-26T17:56:29.975406+0000","last_fullsized":"2024-04-26T17:56:29.975406+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:48.237511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046858+0000","last_change":"2024-04-26T17:53:00.587319+0000","last_active":"2024-04-26T17:56:30.046858+0000","last_peered":"2024-04-26T17:56:30.046858+0000","last_clean":"2024-04-26T17:56:30.046858+0000","last_became_active":"2024-04-26T17:53:00.586983+0000","last_became_peered":"2024-04-26T17:53:00.586983+0000","last_unstale":"2024-04-26T17:56:30.046858+0000","last_undegraded":"2024-04-26T17:56:30.046858+0000","last_fullsized":"2024-04-26T17:56:30.046858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:03:43.470013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883218+0000","last_change":"2024-04-26T17:53:01.145672+0000","last_active":"2024-04-26T17:56:29.883218+0000","last_peered":"2024-04-26T17:56:29.883218+0000","last_clean":"2024-04-26T17:56:29.883218+0000","last_became_active":"2024-04-26T17:53:01.145274+0000","last_became_peered":"2024-04-26T17:53:01.145274+0000","last_unstale":"2024-04-26T17:56:29.883218+0000","last_undegraded":"2024-04-26T17:56:29.883218+0000","last_fullsized":"2024-04-26T17:56:29.883218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:21.122352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975691+0000","last_change":"2024-04-26T17:53:03.611232+0000","last_active":"2024-04-26T17:56:29.975691+0000","last_peered":"2024-04-26T17:56:29.975691+0000","last_clean":"2024-04-26T17:56:29.975691+0000","last_became_active":"2024-04-26T17:53:03.610905+0000","last_became_peered":"2024-04-26T17:53:03.610905+0000","last_unstale":"2024-04-26T17:56:29.975691+0000","last_undegraded":"2024-04-26T17:56:29.975691+0000","last_fullsized":"2024-04-26T17:56:29.975691+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:31.426237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'101","reported_seq":222,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:32.139765+0000","last_change":"2024-04-26T17:53:00.580220+0000","last_active":"2024-04-26T17:56:32.139765+0000","last_peered":"2024-04-26T17:56:32.139765+0000","last_clean":"2024-04-26T17:56:32.139765+0000","last_became_active":"2024-04-26T17:53:00.579818+0000","last_became_peered":"2024-04-26T17:53:00.579818+0000","last_unstale":"2024-04-26T17:56:32.139765+0000","last_undegraded":"2024-04-26T17:56:32.139765+0000","last_fullsized":"2024-04-26T17:56:32.139765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:15.260489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":186,"num_write_kb":1287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883670+0000","last_change":"2024-04-26T17:53:03.610882+0000","last_active":"2024-04-26T17:56:29.883670+0000","last_peered":"2024-04-26T17:56:29.883670+0000","last_clean":"2024-04-26T17:56:29.883670+0000","last_became_active":"2024-04-26T17:53:03.609979+0000","last_became_peered":"2024-04-26T17:53:03.609979+0000","last_unstale":"2024-04-26T17:56:29.883670+0000","last_undegraded":"2024-04-26T17:56:29.883670+0000","last_fullsized":"2024-04-26T17:56:29.883670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:33.167579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.784906+0000","last_change":"2024-04-26T17:53:01.358555+0000","last_active":"2024-04-26T17:56:29.784906+0000","last_peered":"2024-04-26T17:56:29.784906+0000","last_clean":"2024-04-26T17:56:29.784906+0000","last_became_active":"2024-04-26T17:53:01.358138+0000","last_became_peered":"2024-04-26T17:53:01.358138+0000","last_unstale":"2024-04-26T17:56:29.784906+0000","last_undegraded":"2024-04-26T17:56:29.784906+0000","last_fullsized":"2024-04-26T17:56:29.784906+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:14.886609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472981+0000","last_change":"2024-04-26T17:53:03.618816+0000","last_active":"2024-04-26T17:56:30.472981+0000","last_peered":"2024-04-26T17:56:30.472981+0000","last_clean":"2024-04-26T17:56:30.472981+0000","last_became_active":"2024-04-26T17:53:03.618184+0000","last_became_peered":"2024-04-26T17:53:03.618184+0000","last_unstale":"2024-04-26T17:56:30.472981+0000","last_undegraded":"2024-04-26T17:56:30.472981+0000","last_fullsized":"2024-04-26T17:56:30.472981+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:07.514516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.045564+0000","last_change":"2024-04-26T17:53:01.147081+0000","last_active":"2024-04-26T17:56:30.045564+0000","last_peered":"2024-04-26T17:56:30.045564+0000","last_clean":"2024-04-26T17:56:30.045564+0000","last_became_active":"2024-04-26T17:53:01.146698+0000","last_became_peered":"2024-04-26T17:53:01.146698+0000","last_unstale":"2024-04-26T17:56:30.045564+0000","last_undegraded":"2024-04-26T17:56:30.045564+0000","last_fullsized":"2024-04-26T17:56:30.045564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:44:19.927316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882754+0000","last_change":"2024-04-26T17:53:03.612592+0000","last_active":"2024-04-26T17:56:29.882754+0000","last_peered":"2024-04-26T17:56:29.882754+0000","last_clean":"2024-04-26T17:56:29.882754+0000","last_became_active":"2024-04-26T17:53:03.612274+0000","last_became_peered":"2024-04-26T17:53:03.612274+0000","last_unstale":"2024-04-26T17:56:29.882754+0000","last_undegraded":"2024-04-26T17:56:29.882754+0000","last_fullsized":"2024-04-26T17:56:29.882754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:30:48.461947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882651+0000","last_change":"2024-04-26T17:53:00.586430+0000","last_active":"2024-04-26T17:56:29.882651+0000","last_peered":"2024-04-26T17:56:29.882651+0000","last_clean":"2024-04-26T17:56:29.882651+0000","last_became_active":"2024-04-26T17:53:00.586004+0000","last_became_peered":"2024-04-26T17:53:00.586004+0000","last_unstale":"2024-04-26T17:56:29.882651+0000","last_undegraded":"2024-04-26T17:56:29.882651+0000","last_fullsized":"2024-04-26T17:56:29.882651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:47:18.224890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627501+0000","last_change":"2024-04-26T17:53:03.623081+0000","last_active":"2024-04-26T17:56:30.627501+0000","last_peered":"2024-04-26T17:56:30.627501+0000","last_clean":"2024-04-26T17:56:30.627501+0000","last_became_active":"2024-04-26T17:53:03.622761+0000","last_became_peered":"2024-04-26T17:53:03.622761+0000","last_unstale":"2024-04-26T17:56:30.627501+0000","last_undegraded":"2024-04-26T17:56:30.627501+0000","last_fullsized":"2024-04-26T17:56:30.627501+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:39.520075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786022+0000","last_change":"2024-04-26T17:53:00.583013+0000","last_active":"2024-04-26T17:56:29.786022+0000","last_peered":"2024-04-26T17:56:29.786022+0000","last_clean":"2024-04-26T17:56:29.786022+0000","last_became_active":"2024-04-26T17:53:00.582653+0000","last_became_peered":"2024-04-26T17:53:00.582653+0000","last_unstale":"2024-04-26T17:56:29.786022+0000","last_undegraded":"2024-04-26T17:56:29.786022+0000","last_fullsized":"2024-04-26T17:56:29.786022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:09.448621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882786+0000","last_change":"2024-04-26T17:53:03.618232+0000","last_active":"2024-04-26T17:56:29.882786+0000","last_peered":"2024-04-26T17:56:29.882786+0000","last_clean":"2024-04-26T17:56:29.882786+0000","last_became_active":"2024-04-26T17:53:03.617906+0000","last_became_peered":"2024-04-26T17:53:03.617906+0000","last_unstale":"2024-04-26T17:56:29.882786+0000","last_undegraded":"2024-04-26T17:56:29.882786+0000","last_fullsized":"2024-04-26T17:56:29.882786+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:17.357398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882975+0000","last_change":"2024-04-26T17:53:00.586565+0000","last_active":"2024-04-26T17:56:29.882975+0000","last_peered":"2024-04-26T17:56:29.882975+0000","last_clean":"2024-04-26T17:56:29.882975+0000","last_became_active":"2024-04-26T17:53:00.586161+0000","last_became_peered":"2024-04-26T17:53:00.586161+0000","last_unstale":"2024-04-26T17:56:29.882975+0000","last_undegraded":"2024-04-26T17:56:29.882975+0000","last_fullsized":"2024-04-26T17:56:29.882975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T19:56:25.294055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047092+0000","last_change":"2024-04-26T17:53:03.620340+0000","last_active":"2024-04-26T17:56:30.047092+0000","last_peered":"2024-04-26T17:56:30.047092+0000","last_clean":"2024-04-26T17:56:30.047092+0000","last_became_active":"2024-04-26T17:53:03.619897+0000","last_became_peered":"2024-04-26T17:53:03.619897+0000","last_unstale":"2024-04-26T17:56:30.047092+0000","last_undegraded":"2024-04-26T17:56:30.047092+0000","last_fullsized":"2024-04-26T17:56:30.047092+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:12:40.662070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786265+0000","last_change":"2024-04-26T17:53:01.358131+0000","last_active":"2024-04-26T17:56:29.786265+0000","last_peered":"2024-04-26T17:56:29.786265+0000","last_clean":"2024-04-26T17:56:29.786265+0000","last_became_active":"2024-04-26T17:53:01.357796+0000","last_became_peered":"2024-04-26T17:53:01.357796+0000","last_unstale":"2024-04-26T17:56:29.786265+0000","last_undegraded":"2024-04-26T17:56:29.786265+0000","last_fullsized":"2024-04-26T17:56:29.786265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:47.849873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047364+0000","last_change":"2024-04-26T17:53:03.616133+0000","last_active":"2024-04-26T17:56:30.047364+0000","last_peered":"2024-04-26T17:56:30.047364+0000","last_clean":"2024-04-26T17:56:30.047364+0000","last_became_active":"2024-04-26T17:53:03.615797+0000","last_became_peered":"2024-04-26T17:53:03.615797+0000","last_unstale":"2024-04-26T17:56:30.047364+0000","last_undegraded":"2024-04-26T17:56:30.047364+0000","last_fullsized":"2024-04-26T17:56:30.047364+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:05.698489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'65","reported_seq":125,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:31.733880+0000","last_change":"2024-04-26T17:53:00.587696+0000","last_active":"2024-04-26T17:56:31.733880+0000","last_peered":"2024-04-26T17:56:31.733880+0000","last_clean":"2024-04-26T17:56:31.733880+0000","last_became_active":"2024-04-26T17:53:00.587305+0000","last_became_peered":"2024-04-26T17:53:00.587305+0000","last_unstale":"2024-04-26T17:56:31.733880+0000","last_undegraded":"2024-04-26T17:56:31.733880+0000","last_fullsized":"2024-04-26T17:56:31.733880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:39.185753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047031+0000","last_change":"2024-04-26T17:53:03.613076+0000","last_active":"2024-04-26T17:56:30.047031+0000","last_peered":"2024-04-26T17:56:30.047031+0000","last_clean":"2024-04-26T17:56:30.047031+0000","last_became_active":"2024-04-26T17:53:03.612746+0000","last_became_peered":"2024-04-26T17:53:03.612746+0000","last_unstale":"2024-04-26T17:56:30.047031+0000","last_undegraded":"2024-04-26T17:56:30.047031+0000","last_fullsized":"2024-04-26T17:56:30.047031+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:20.920154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975561+0000","last_change":"2024-04-26T17:53:00.582575+0000","last_active":"2024-04-26T17:56:29.975561+0000","last_peered":"2024-04-26T17:56:29.975561+0000","last_clean":"2024-04-26T17:56:29.975561+0000","last_became_active":"2024-04-26T17:53:00.582215+0000","last_became_peered":"2024-04-26T17:53:00.582215+0000","last_unstale":"2024-04-26T17:56:29.975561+0000","last_undegraded":"2024-04-26T17:56:29.975561+0000","last_fullsized":"2024-04-26T17:56:29.975561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:52:59.560312+0000","last_clean_scrub_stamp":"2024-04-26T17:52:59.560312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:35:12.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.047430+0000","last_change":"2024-04-26T17:53:03.616269+0000","last_active":"2024-04-26T17:56:30.047430+0000","last_peered":"2024-04-26T17:56:30.047430+0000","last_clean":"2024-04-26T17:56:30.047430+0000","last_became_active":"2024-04-26T17:53:03.615896+0000","last_became_peered":"2024-04-26T17:53:03.615896+0000","last_unstale":"2024-04-26T17:56:30.047430+0000","last_undegraded":"2024-04-26T17:56:30.047430+0000","last_fullsized":"2024-04-26T17:56:30.047430+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:30:12.111392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627575+0000","last_change":"2024-04-26T17:53:03.621589+0000","last_active":"2024-04-26T17:56:30.627575+0000","last_peered":"2024-04-26T17:56:30.627575+0000","last_clean":"2024-04-26T17:56:30.627575+0000","last_became_active":"2024-04-26T17:53:03.621211+0000","last_became_peered":"2024-04-26T17:53:03.621211+0000","last_unstale":"2024-04-26T17:56:30.627575+0000","last_undegraded":"2024-04-26T17:56:30.627575+0000","last_fullsized":"2024-04-26T17:56:30.627575+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.487476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786065+0000","last_change":"2024-04-26T17:53:03.624536+0000","last_active":"2024-04-26T17:56:29.786065+0000","last_peered":"2024-04-26T17:56:29.786065+0000","last_clean":"2024-04-26T17:56:29.786065+0000","last_became_active":"2024-04-26T17:53:03.624234+0000","last_became_peered":"2024-04-26T17:53:03.624234+0000","last_unstale":"2024-04-26T17:56:29.786065+0000","last_undegraded":"2024-04-26T17:56:29.786065+0000","last_fullsized":"2024-04-26T17:56:29.786065+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:31:36.322688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786074+0000","last_change":"2024-04-26T17:53:03.620096+0000","last_active":"2024-04-26T17:56:29.786074+0000","last_peered":"2024-04-26T17:56:29.786074+0000","last_clean":"2024-04-26T17:56:29.786074+0000","last_became_active":"2024-04-26T17:53:03.619729+0000","last_became_peered":"2024-04-26T17:53:03.619729+0000","last_unstale":"2024-04-26T17:56:29.786074+0000","last_undegraded":"2024-04-26T17:56:29.786074+0000","last_fullsized":"2024-04-26T17:56:29.786074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:17.987728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.472965+0000","last_change":"2024-04-26T17:53:49.768829+0000","last_active":"2024-04-26T17:56:30.472965+0000","last_peered":"2024-04-26T17:56:30.472965+0000","last_clean":"2024-04-26T17:56:30.472965+0000","last_became_active":"2024-04-26T17:53:49.768415+0000","last_became_peered":"2024-04-26T17:53:49.768415+0000","last_unstale":"2024-04-26T17:56:30.472965+0000","last_undegraded":"2024-04-26T17:56:30.472965+0000","last_fullsized":"2024-04-26T17:56:30.472965+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:57.364355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.975670+0000","last_change":"2024-04-26T17:53:03.619193+0000","last_active":"2024-04-26T17:56:29.975670+0000","last_peered":"2024-04-26T17:56:29.975670+0000","last_clean":"2024-04-26T17:56:29.975670+0000","last_became_active":"2024-04-26T17:53:03.618776+0000","last_became_peered":"2024-04-26T17:53:03.618776+0000","last_unstale":"2024-04-26T17:56:29.975670+0000","last_undegraded":"2024-04-26T17:56:29.975670+0000","last_fullsized":"2024-04-26T17:56:29.975670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:07:24.811683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627751+0000","last_change":"2024-04-26T17:53:03.621366+0000","last_active":"2024-04-26T17:56:30.627751+0000","last_peered":"2024-04-26T17:56:30.627751+0000","last_clean":"2024-04-26T17:56:30.627751+0000","last_became_active":"2024-04-26T17:53:03.621048+0000","last_became_peered":"2024-04-26T17:53:03.621048+0000","last_unstale":"2024-04-26T17:56:30.627751+0000","last_undegraded":"2024-04-26T17:56:30.627751+0000","last_fullsized":"2024-04-26T17:56:30.627751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:31:06.484880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.786409+0000","last_change":"2024-04-26T17:53:03.619383+0000","last_active":"2024-04-26T17:56:29.786409+0000","last_peered":"2024-04-26T17:56:29.786409+0000","last_clean":"2024-04-26T17:56:29.786409+0000","last_became_active":"2024-04-26T17:53:03.619017+0000","last_became_peered":"2024-04-26T17:53:03.619017+0000","last_unstale":"2024-04-26T17:56:29.786409+0000","last_undegraded":"2024-04-26T17:56:29.786409+0000","last_fullsized":"2024-04-26T17:56:29.786409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:40:20.091959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.883534+0000","last_change":"2024-04-26T17:53:49.770695+0000","last_active":"2024-04-26T17:56:29.883534+0000","last_peered":"2024-04-26T17:56:29.883534+0000","last_clean":"2024-04-26T17:56:29.883534+0000","last_became_active":"2024-04-26T17:53:49.770303+0000","last_became_peered":"2024-04-26T17:53:49.770303+0000","last_unstale":"2024-04-26T17:56:29.883534+0000","last_undegraded":"2024-04-26T17:56:29.883534+0000","last_fullsized":"2024-04-26T17:56:29.883534+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:22.830129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.891522+0000","last_change":"2024-04-26T17:53:03.605521+0000","last_active":"2024-04-26T17:56:29.891522+0000","last_peered":"2024-04-26T17:56:29.891522+0000","last_clean":"2024-04-26T17:56:29.891522+0000","last_became_active":"2024-04-26T17:53:03.605145+0000","last_became_peered":"2024-04-26T17:53:03.605145+0000","last_unstale":"2024-04-26T17:56:29.891522+0000","last_undegraded":"2024-04-26T17:56:29.891522+0000","last_fullsized":"2024-04-26T17:56:29.891522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:58:09.856021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.882572+0000","last_change":"2024-04-26T17:53:03.610866+0000","last_active":"2024-04-26T17:56:29.882572+0000","last_peered":"2024-04-26T17:56:29.882572+0000","last_clean":"2024-04-26T17:56:29.882572+0000","last_became_active":"2024-04-26T17:53:03.610326+0000","last_became_peered":"2024-04-26T17:53:03.610326+0000","last_unstale":"2024-04-26T17:56:29.882572+0000","last_undegraded":"2024-04-26T17:56:29.882572+0000","last_fullsized":"2024-04-26T17:56:29.882572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:20.042415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.046004+0000","last_change":"2024-04-26T17:53:03.615336+0000","last_active":"2024-04-26T17:56:30.046004+0000","last_peered":"2024-04-26T17:56:30.046004+0000","last_clean":"2024-04-26T17:56:30.046004+0000","last_became_active":"2024-04-26T17:53:03.614447+0000","last_became_peered":"2024-04-26T17:53:03.614447+0000","last_unstale":"2024-04-26T17:56:30.046004+0000","last_undegraded":"2024-04-26T17:56:30.046004+0000","last_fullsized":"2024-04-26T17:56:30.046004+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T18:36:10.091929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:29.976250+0000","last_change":"2024-04-26T17:53:03.614863+0000","last_active":"2024-04-26T17:56:29.976250+0000","last_peered":"2024-04-26T17:56:29.976250+0000","last_clean":"2024-04-26T17:56:29.976250+0000","last_became_active":"2024-04-26T17:53:03.614520+0000","last_became_peered":"2024-04-26T17:53:03.614520+0000","last_unstale":"2024-04-26T17:56:29.976250+0000","last_undegraded":"2024-04-26T17:56:29.976250+0000","last_fullsized":"2024-04-26T17:56:29.976250+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:51.367785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T17:56:30.627720+0000","last_change":"2024-04-26T17:53:03.618521+0000","last_active":"2024-04-26T17:56:30.627720+0000","last_peered":"2024-04-26T17:56:30.627720+0000","last_clean":"2024-04-26T17:56:30.627720+0000","last_became_active":"2024-04-26T17:53:03.617658+0000","last_became_peered":"2024-04-26T17:53:03.617658+0000","last_unstale":"2024-04-26T17:56:30.627720+0000","last_undegraded":"2024-04-26T17:56:30.627720+0000","last_fullsized":"2024-04-26T17:56:30.627720+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T17:53:02.586248+0000","last_clean_scrub_stamp":"2024-04-26T17:53:02.586248+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:39.813098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106863,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":794668,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1538,"num_read_kb":983,"num_write":421,"num_write_kb":2354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2384004,"data_compressed":103473,"data_compressed_allocated":135168,"data_compressed_original":2101248,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383738,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":488700,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645375,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28192,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716960,"statfs":{"total":95995035648,"available":95966167040,"internally_reserved":0,"allocated":1196032,"data_stored":1477100,"data_compressed":53745,"data_compressed_allocated":69632,"data_compressed_original":905216,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808899,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":882152,"data_compressed":22211,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070536,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":882072,"data_compressed":22211,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332174,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":1082017,"data_compressed":38543,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495699,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":488325,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1064960,"data_stored":556106,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738481,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":489596,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411432,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":558379,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182497,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1081344,"data_stored":1210084,"data_compressed":37413,"data_compressed_allocated":49152,"data_compressed_original":708608,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920861,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":489461,"data_compressed":2922,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757336,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1114112,"data_stored":1409234,"data_compressed":53745,"data_compressed_allocated":69632,"data_compressed_original":905216,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35621,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330631,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":398497,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T17:56:41.037 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:56:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:41 smithi186 ceph-mon[32797]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-26T17:56:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:41 smithi186 ceph-mon[32797]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:56:41.291 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:41.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:41 smithi031 ceph-mon[29164]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-26T17:56:41.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:41 smithi031 ceph-mon[29164]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:56:41.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:41 smithi097 ceph-mon[32419]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-26T17:56:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:41 smithi097 ceph-mon[32419]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T17:56:42.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.893+0000 7f97c95bb700 1 -- 172.21.15.31:0/2633259847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c410fe70 msgr2=0x7f97c4112260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:42.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.893+0000 7f97c95bb700 1 --2- 172.21.15.31:0/2633259847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c410fe70 0x7f97c4112260 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00b0a0 tx=0x7f97bc00b3b0 comp rx=0 tx=0).stop 2024-04-26T17:56:42.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.893+0000 7f97c95bb700 1 -- 172.21.15.31:0/2633259847 shutdown_connections 2024-04-26T17:56:42.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.893+0000 7f97c95bb700 1 --2- 172.21.15.31:0/2633259847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c410fe70 0x7f97c4112260 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:42.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.893+0000 7f97c95bb700 1 --2- 172.21.15.31:0/2633259847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97c406bbd0 0x7f97c406bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:42.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.893+0000 7f97c95bb700 1 --2- 172.21.15.31:0/2633259847 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97c406c4f0 0x7f97c410f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:42.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.893+0000 7f97c95bb700 1 -- 172.21.15.31:0/2633259847 >> 172.21.15.31:0/2633259847 conn(0x7f97c40793c0 msgr2=0x7f97c40797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:42.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.893+0000 7f97c95bb700 1 -- 172.21.15.31:0/2633259847 shutdown_connections 2024-04-26T17:56:42.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.893+0000 7f97c95bb700 1 -- 172.21.15.31:0/2633259847 wait complete. 2024-04-26T17:56:42.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.894+0000 7f97c95bb700 1 Processor -- start 2024-04-26T17:56:42.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.894+0000 7f97c95bb700 1 -- start start 2024-04-26T17:56:42.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c95bb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c406bbd0 0x7f97c419bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:42.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c95bb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97c406c4f0 0x7f97c419c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:42.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c95bb700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97c410fe70 0x7f97c41a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:42.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c95bb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f97c4115820 con 0x7f97c406bbd0 2024-04-26T17:56:42.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c95bb700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41156a0 con 0x7f97c410fe70 2024-04-26T17:56:42.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c95bb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41159a0 con 0x7f97c406c4f0 2024-04-26T17:56:42.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c406bbd0 0x7f97c419bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:42.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c406bbd0 0x7f97c419bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34368/0 (socket says 172.21.15.31:34368) 2024-04-26T17:56:42.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c2ffd700 1 -- 172.21.15.31:0/3349270651 learned_addr learned my addr 172.21.15.31:0/3349270651 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:42.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c27fc700 1 --2- 172.21.15.31:0/3349270651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97c406c4f0 0x7f97c419c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:42.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c37fe700 1 --2- 172.21.15.31:0/3349270651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97c410fe70 0x7f97c41a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:42.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c2ffd700 1 -- 172.21.15.31:0/3349270651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97c406c4f0 msgr2=0x7f97c419c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:42.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c2ffd700 1 --2- 172.21.15.31:0/3349270651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97c406c4f0 0x7f97c419c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:42.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c2ffd700 1 -- 172.21.15.31:0/3349270651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97c410fe70 msgr2=0x7f97c41a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:42.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c2ffd700 1 --2- 172.21.15.31:0/3349270651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97c410fe70 0x7f97c41a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:42.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c2ffd700 1 -- 172.21.15.31:0/3349270651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c41a0d50 con 0x7f97c406bbd0 2024-04-26T17:56:42.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c2ffd700 1 --2- 172.21.15.31:0/3349270651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c406bbd0 0x7f97c419bd90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f97b400d980 tx=0x7f97b400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:42.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97abfff700 1 -- 172.21.15.31:0/3349270651 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4017070 con 0x7f97c406bbd0 2024-04-26T17:56:42.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c95bb700 1 -- 172.21.15.31:0/3349270651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c41a1040 con 0x7f97c406bbd0 2024-04-26T17:56:42.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97abfff700 1 -- 172.21.15.31:0/3349270651 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97b4004830 con 0x7f97c406bbd0 2024-04-26T17:56:42.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97abfff700 1 -- 172.21.15.31:0/3349270651 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4005440 con 0x7f97c406bbd0 2024-04-26T17:56:42.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.895+0000 7f97c95bb700 1 -- 172.21.15.31:0/3349270651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c41a88c0 con 0x7f97c406bbd0 2024-04-26T17:56:42.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.896+0000 7f97c95bb700 1 -- 172.21.15.31:0/3349270651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c406a110 con 0x7f97c406bbd0 2024-04-26T17:56:42.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.898+0000 7f97abfff700 1 -- 172.21.15.31:0/3349270651 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f97b40055e0 con 0x7f97c406bbd0 2024-04-26T17:56:42.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.898+0000 7f97abfff700 1 --2- 172.21.15.31:0/3349270651 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f97ac0a3e40 0x7f97ac0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:42.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.898+0000 7f97c27fc700 1 --2- 172.21.15.31:0/3349270651 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f97ac0a3e40 0x7f97ac0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:42.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.898+0000 7f97abfff700 1 -- 172.21.15.31:0/3349270651 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f97b40d0cc0 con 0x7f97c406bbd0 2024-04-26T17:56:42.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.899+0000 7f97c27fc700 1 --2- 172.21.15.31:0/3349270651 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f97ac0a3e40 0x7f97ac0a6300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f97b8005fd0 tx=0x7f97b8005db0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:42.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:42.902+0000 7f97abfff700 1 -- 172.21.15.31:0/3349270651 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97b4020020 con 0x7f97c406bbd0 2024-04-26T17:56:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:43.168+0000 7f97c95bb700 1 -- 172.21.15.31:0/3349270651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97c419d0a0 con 0x7f97c406bbd0 2024-04-26T17:56:43.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:43.169+0000 7f97abfff700 1 -- 172.21.15.31:0/3349270651 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11871 (secure 0 0 0) 0x7f97b409d240 con 0x7f97c406bbd0 2024-04-26T17:56:43.168 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:43.168 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T17:56:28.628701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24446,24457,24473]},"id":1}]} 2024-04-26T17:56:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:43.171+0000 7f97c95bb700 1 -- 172.21.15.31:0/3349270651 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f97ac0a3e40 msgr2=0x7f97ac0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:43.171+0000 7f97c95bb700 1 --2- 172.21.15.31:0/3349270651 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f97ac0a3e40 0x7f97ac0a6300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f97b8005fd0 tx=0x7f97b8005db0 comp rx=0 tx=0).stop 2024-04-26T17:56:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:43.171+0000 7f97c95bb700 1 -- 172.21.15.31:0/3349270651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c406bbd0 msgr2=0x7f97c419bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:43.171+0000 7f97c95bb700 1 --2- 172.21.15.31:0/3349270651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c406bbd0 0x7f97c419bd90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f97b400d980 tx=0x7f97b400dd40 comp rx=0 tx=0).stop 2024-04-26T17:56:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:43.171+0000 7f97c95bb700 1 -- 172.21.15.31:0/3349270651 shutdown_connections 2024-04-26T17:56:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:43.171+0000 7f97c95bb700 1 --2- 172.21.15.31:0/3349270651 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f97ac0a3e40 0x7f97ac0a6300 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:43.171+0000 7f97c95bb700 1 --2- 172.21.15.31:0/3349270651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97c410fe70 0x7f97c41a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:43.171+0000 7f97c95bb700 1 --2- 172.21.15.31:0/3349270651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c406bbd0 0x7f97c419bd90 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:43.171+0000 7f97c95bb700 1 --2- 172.21.15.31:0/3349270651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97c406c4f0 0x7f97c419c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:43.171+0000 7f97c95bb700 1 -- 172.21.15.31:0/3349270651 >> 172.21.15.31:0/3349270651 conn(0x7f97c40793c0 msgr2=0x7f97c4111420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:43.171+0000 7f97c95bb700 1 -- 172.21.15.31:0/3349270651 shutdown_connections 2024-04-26T17:56:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:43.171+0000 7f97c95bb700 1 -- 172.21.15.31:0/3349270651 wait complete. 2024-04-26T17:56:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-26T17:56:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:43 smithi097 ceph-mon[32419]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 51 op/s 2024-04-26T17:56:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:43 smithi186 ceph-mon[32797]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 51 op/s 2024-04-26T17:56:43.543 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:43 smithi031 ceph-mon[29164]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 51 op/s 2024-04-26T17:56:43.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:56:44.100 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:44.345 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:44 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3349270651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:44.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:44 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3349270651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:44.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:44 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3349270651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:45.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:45 smithi031 ceph-mon[29164]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-26T17:56:45.339 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:45 smithi097 ceph-mon[32419]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-26T17:56:45.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.364+0000 7f3f88083640 1 -- 172.21.15.31:0/3408003198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f80103be0 msgr2=0x7f3f80103fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:45.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.364+0000 7f3f88083640 1 --2- 172.21.15.31:0/3408003198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f80103be0 0x7f3f80103fc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c009900 tx=0x7f3f7c02a480 comp rx=0 tx=0).stop 2024-04-26T17:56:45.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.364+0000 7f3f88083640 1 -- 172.21.15.31:0/3408003198 shutdown_connections 2024-04-26T17:56:45.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.364+0000 7f3f88083640 1 --2- 172.21.15.31:0/3408003198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f8010cfb0 0x7f3f8010f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.364+0000 7f3f88083640 1 --2- 172.21.15.31:0/3408003198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f80104500 0x7f3f8010c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.364+0000 7f3f88083640 1 --2- 172.21.15.31:0/3408003198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f80103be0 0x7f3f80103fc0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.364+0000 7f3f88083640 1 -- 172.21.15.31:0/3408003198 >> 172.21.15.31:0/3408003198 conn(0x7f3f80077bb0 msgr2=0x7f3f80077fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:45.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.364+0000 7f3f88083640 1 -- 172.21.15.31:0/3408003198 shutdown_connections 2024-04-26T17:56:45.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.365+0000 7f3f88083640 1 -- 172.21.15.31:0/3408003198 wait complete. 2024-04-26T17:56:45.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.365+0000 7f3f88083640 1 Processor -- start 2024-04-26T17:56:45.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.365+0000 7f3f88083640 1 -- start start 2024-04-26T17:56:45.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.365+0000 7f3f88083640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f80103be0 0x7f3f801990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.365+0000 7f3f88083640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f80104500 0x7f3f801995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.365+0000 7f3f88083640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f8010cfb0 0x7f3f8019d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.365+0000 7f3f88083640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3f80112bf0 con 0x7f3f80104500 2024-04-26T17:56:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.365+0000 7f3f88083640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3f80112a70 con 0x7f3f8010cfb0 2024-04-26T17:56:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.365+0000 7f3f88083640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3f80112d70 con 0x7f3f80103be0 2024-04-26T17:56:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.365+0000 7f3f8561e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f80104500 0x7f3f801995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.365+0000 7f3f8561e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f80104500 0x7f3f801995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37916/0 (socket says 172.21.15.31:37916) 2024-04-26T17:56:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.365+0000 7f3f8561e640 1 -- 172.21.15.31:0/2956673663 learned_addr learned my addr 172.21.15.31:0/2956673663 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.366+0000 7f3f86620640 1 --2- 172.21.15.31:0/2956673663 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f8010cfb0 0x7f3f8019d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.366+0000 7f3f85e1f640 1 --2- 172.21.15.31:0/2956673663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f80103be0 0x7f3f801990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.366+0000 7f3f8561e640 1 -- 172.21.15.31:0/2956673663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f80103be0 msgr2=0x7f3f801990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.366+0000 7f3f8561e640 1 --2- 172.21.15.31:0/2956673663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f80103be0 0x7f3f801990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.366+0000 7f3f8561e640 1 -- 172.21.15.31:0/2956673663 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f8010cfb0 msgr2=0x7f3f8019d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.366+0000 7f3f8561e640 1 --2- 172.21.15.31:0/2956673663 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f8010cfb0 0x7f3f8019d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.366+0000 7f3f8561e640 1 -- 172.21.15.31:0/2956673663 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8019e050 con 0x7f3f80104500 2024-04-26T17:56:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.366+0000 7f3f8561e640 1 --2- 172.21.15.31:0/2956673663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f80104500 0x7f3f801995f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3f7000d950 tx=0x7f3f7000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.366+0000 7f3f76ffd640 1 -- 172.21.15.31:0/2956673663 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f70017030 con 0x7f3f80104500 2024-04-26T17:56:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.366+0000 7f3f88083640 1 -- 172.21.15.31:0/2956673663 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f8019e340 con 0x7f3f80104500 2024-04-26T17:56:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.366+0000 7f3f76ffd640 1 -- 172.21.15.31:0/2956673663 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f70004020 con 0x7f3f80104500 2024-04-26T17:56:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.366+0000 7f3f76ffd640 1 -- 172.21.15.31:0/2956673663 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f700047a0 con 0x7f3f80104500 2024-04-26T17:56:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.366+0000 7f3f88083640 1 -- 172.21.15.31:0/2956673663 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f801a5bd0 con 0x7f3f80104500 2024-04-26T17:56:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.368+0000 7f3f76ffd640 1 -- 172.21.15.31:0/2956673663 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f3f70009a30 con 0x7f3f80104500 2024-04-26T17:56:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.369+0000 7f3f8ada6740 1 Processor -- start 2024-04-26T17:56:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.369+0000 7f3f8ada6740 1 -- start start 2024-04-26T17:56:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.369+0000 7f3f8ada6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 0x55cbf32361e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.369+0000 7f3f76ffd640 1 --2- 172.21.15.31:0/2956673663 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3f6c0a3e60 0x7f3f6c0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.369+0000 7f3f8ada6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3244fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.369+0000 7f3f8ada6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf32454f0 0x55cbf32478e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.369+0000 7f3f8ada6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cbf319c460 con 0x55cbf3236720 2024-04-26T17:56:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.369+0000 7f3f8ada6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55cbf3156590 con 0x55cbf3235e00 2024-04-26T17:56:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.369+0000 7f3f8ada6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cbf3198420 con 0x55cbf32454f0 2024-04-26T17:56:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f8561e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3244fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f8561e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3244fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37928/0 (socket says 172.21.15.31:37928) 2024-04-26T17:56:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f8561e640 1 -- 172.21.15.31:0/3505537630 learned_addr learned my addr 172.21.15.31:0/3505537630 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f85e1f640 1 --2- 172.21.15.31:0/2956673663 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3f6c0a3e60 0x7f3f6c0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f86620640 1 --2- 172.21.15.31:0/3505537630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 0x55cbf32361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f76ffd640 1 -- 172.21.15.31:0/2956673663 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f3f700d0370 con 0x7f3f80104500 2024-04-26T17:56:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f86620640 1 --2- 172.21.15.31:0/3505537630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf32454f0 0x55cbf32478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f8561e640 1 -- 172.21.15.31:0/3505537630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf32454f0 msgr2=0x55cbf32478e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f8561e640 1 --2- 172.21.15.31:0/3505537630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf32454f0 0x55cbf32478e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f8561e640 1 -- 172.21.15.31:0/3505537630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 msgr2=0x55cbf32361e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f8561e640 1 --2- 172.21.15.31:0/3505537630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 0x55cbf32361e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f8561e640 1 -- 172.21.15.31:0/3505537630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbf31630d0 con 0x55cbf3236720 2024-04-26T17:56:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f8561e640 1 --2- 172.21.15.31:0/3505537630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3244fb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3f7009be90 tx=0x7f3f7009bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afc9e0961e4a6edd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f85e1f640 1 --2- 172.21.15.31:0/2956673663 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3f6c0a3e60 0x7f3f6c0a6320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c031440 tx=0x7f3f7c005b50 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f6bfff640 1 -- 172.21.15.31:0/3505537630 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7009b5e0 con 0x55cbf3236720 2024-04-26T17:56:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f6bfff640 1 -- 172.21.15.31:0/3505537630 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f7009c890 con 0x55cbf3236720 2024-04-26T17:56:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.370+0000 7f3f6bfff640 1 -- 172.21.15.31:0/3505537630 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f700041d0 con 0x55cbf3236720 2024-04-26T17:56:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.371+0000 7f3f8ada6740 1 -- 172.21.15.31:0/3505537630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 msgr2=0x55cbf3244fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.371+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/3505537630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3244fb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3f7009be90 tx=0x7f3f7009bcd0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.371+0000 7f3f8ada6740 1 -- 172.21.15.31:0/3505537630 shutdown_connections 2024-04-26T17:56:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.371+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/3505537630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf32454f0 0x55cbf32478e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.371+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/3505537630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3244fb0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.371+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/3505537630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 0x55cbf32361e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.371+0000 7f3f8ada6740 1 -- 172.21.15.31:0/3505537630 >> 172.21.15.31:0/3505537630 conn(0x55cbf3232410 msgr2=0x55cbf32327f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.371+0000 7f3f8ada6740 1 -- 172.21.15.31:0/3505537630 shutdown_connections 2024-04-26T17:56:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.371+0000 7f3f8ada6740 1 -- 172.21.15.31:0/3505537630 wait complete. 2024-04-26T17:56:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8ada6740 1 -- start start 2024-04-26T17:56:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8ada6740 1 Processor -- start 2024-04-26T17:56:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8ada6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf3235e00 0x55cbf3242420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8ada6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3242960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8ada6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf32454f0 0x55cbf3242ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8ada6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cbf32480c0 con 0x55cbf3236720 2024-04-26T17:56:45.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8ada6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55cbf3123b60 con 0x55cbf32454f0 2024-04-26T17:56:45.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8ada6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cbf3248240 con 0x55cbf3235e00 2024-04-26T17:56:45.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8ada6740 10 client.0 _ll_register_callbacks cb 0x55cbf31482b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:56:45.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8561e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3242960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8561e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3242960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37936/0 (socket says 172.21.15.31:37936) 2024-04-26T17:56:45.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8561e640 1 -- 172.21.15.31:0/2396907896 learned_addr learned my addr 172.21.15.31:0/2396907896 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:45.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f86620640 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf32454f0 0x55cbf3242ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f85e1f640 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf3235e00 0x55cbf3242420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8561e640 1 -- 172.21.15.31:0/2396907896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf3235e00 msgr2=0x55cbf3242420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:45.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8561e640 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf3235e00 0x55cbf3242420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8561e640 1 -- 172.21.15.31:0/2396907896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf32454f0 msgr2=0x55cbf3242ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.372+0000 7f3f8561e640 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf32454f0 0x55cbf3242ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f8561e640 1 -- 172.21.15.31:0/2396907896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbf2eae210 con 0x55cbf3236720 2024-04-26T17:56:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f85e1f640 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf3235e00 0x55cbf3242420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:56:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f86620640 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf32454f0 0x55cbf3242ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:56:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f8561e640 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3242960 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3f70005fd0 tx=0x7f3f700984b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41142f5ca4134f97 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f697fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:56:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f697fa640 1 -- 172.21.15.31:0/2396907896 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7009c190 con 0x55cbf3236720 2024-04-26T17:56:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f697fa640 1 -- 172.21.15.31:0/2396907896 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f3f700041d0 con 0x55cbf3236720 2024-04-26T17:56:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f8ada6740 1 -- 172.21.15.31:0/2396907896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f7009c890 con 0x55cbf3236720 2024-04-26T17:56:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f697fa640 1 -- 172.21.15.31:0/2396907896 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3f700a0540 con 0x55cbf3236720 2024-04-26T17:56:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f8ada6740 10 client.15264 fetch_fsmap learned FSMap version 26 2024-04-26T17:56:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f8ada6740 1 -- 172.21.15.31:0/2396907896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cbf2f7a620 con 0x55cbf3236720 2024-04-26T17:56:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f697fa640 1 -- 172.21.15.31:0/2396907896 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f7009c890 con 0x55cbf3236720 2024-04-26T17:56:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f697fa640 1 -- 172.21.15.31:0/2396907896 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3f700a00d0 con 0x55cbf3236720 2024-04-26T17:56:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f8ada6740 10 client.15264 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T17:56:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f8ada6740 10 client.15264 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:56:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f8ada6740 20 client.15264 populate_metadata read hostname 'smithi031' 2024-04-26T17:56:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55cbf32554c0 0x55cbf32578b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f8ada6740 4 client.15264 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:56:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.373+0000 7f3f8ada6740 1 -- 172.21.15.31:0/2396907896 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbf31aa070 con 0x55cbf32554c0 2024-04-26T17:56:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.374+0000 7f3f85e1f640 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55cbf32554c0 0x55cbf32578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.374+0000 7f3f697fa640 10 client.15264 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:56:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.374+0000 7f3f85e1f640 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55cbf32554c0 0x55cbf32578b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b6b06bbd1639b20 server_cookie=fed968266b27a62d in_seq=0 out_seq=0 2024-04-26T17:56:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f697fa640 1 -- 172.21.15.31:0/2396907896 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cbf31aa070 con 0x55cbf32554c0 2024-04-26T17:56:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f697fa640 10 client.15264 handle_command_reply: tid=0 2024-04-26T17:56:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f8ada6740 1 client.15264 shutdown 2024-04-26T17:56:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f8ada6740 1 -- 172.21.15.31:0/2396907896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 msgr2=0x55cbf3242960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3242960 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3f70005fd0 tx=0x7f3f700984b0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f8ada6740 1 -- 172.21.15.31:0/2396907896 shutdown_connections 2024-04-26T17:56:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55cbf32554c0 0x55cbf32578b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf32454f0 0x55cbf3242ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3242960 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/2396907896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf3235e00 0x55cbf3242420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f697fa640 0 client.15264 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:56:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f8ada6740 1 -- 172.21.15.31:0/2396907896 >> 172.21.15.31:0/2396907896 conn(0x55cbf3232410 msgr2=0x55cbf3238280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f8ada6740 1 -- 172.21.15.31:0/2396907896 shutdown_connections 2024-04-26T17:56:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f8ada6740 1 -- 172.21.15.31:0/2396907896 wait complete. 2024-04-26T17:56:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.375+0000 7f3f8ada6740 20 client.15264 trim_cache size 0 max 16384 2024-04-26T17:56:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.390+0000 7f3f8ada6740 1 Processor -- start 2024-04-26T17:56:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.390+0000 7f3f8ada6740 1 -- start start 2024-04-26T17:56:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.390+0000 7f3f8ada6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 0x55cbf325a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.390+0000 7f3f8ada6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf3236720 0x55cbf325ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.390+0000 7f3f8ada6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf32454f0 0x55cbf323f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.390+0000 7f3f8ada6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801aeb90 con 0x55cbf32454f0 2024-04-26T17:56:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.390+0000 7f3f8ada6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801ac460 con 0x55cbf3235e00 2024-04-26T17:56:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.390+0000 7f3f8ada6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801aa2d0 con 0x55cbf3236720 2024-04-26T17:56:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.390+0000 7f3f86620640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf32454f0 0x55cbf323f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.390+0000 7f3f86620640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf32454f0 0x55cbf323f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37944/0 (socket says 172.21.15.31:37944) 2024-04-26T17:56:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.390+0000 7f3f86620640 1 -- 172.21.15.31:0/967279649 learned_addr learned my addr 172.21.15.31:0/967279649 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.390+0000 7f3f8561e640 1 --2- 172.21.15.31:0/967279649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf3236720 0x55cbf325ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.390+0000 7f3f85e1f640 1 --2- 172.21.15.31:0/967279649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 0x55cbf325a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.391+0000 7f3f86620640 1 -- 172.21.15.31:0/967279649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf3236720 msgr2=0x55cbf325ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.391+0000 7f3f86620640 1 --2- 172.21.15.31:0/967279649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf3236720 0x55cbf325ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.391+0000 7f3f86620640 1 -- 172.21.15.31:0/967279649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 msgr2=0x55cbf325a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.391+0000 7f3f86620640 1 --2- 172.21.15.31:0/967279649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 0x55cbf325a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.391+0000 7f3f86620640 1 -- 172.21.15.31:0/967279649 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbf3236550 con 0x55cbf32454f0 2024-04-26T17:56:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.391+0000 7f3f86620640 1 --2- 172.21.15.31:0/967279649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf32454f0 0x55cbf323f680 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3f780111d0 tx=0x7f3f78009dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace516c7b4e36ab4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.391+0000 7f3f6b7fe640 1 -- 172.21.15.31:0/967279649 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f78003970 con 0x55cbf32454f0 2024-04-26T17:56:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.391+0000 7f3f6b7fe640 1 -- 172.21.15.31:0/967279649 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f78004030 con 0x55cbf32454f0 2024-04-26T17:56:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.391+0000 7f3f6b7fe640 1 -- 172.21.15.31:0/967279649 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f78010690 con 0x55cbf32454f0 2024-04-26T17:56:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.391+0000 7f3f8ada6740 1 -- 172.21.15.31:0/967279649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf32454f0 msgr2=0x55cbf323f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.391+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/967279649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf32454f0 0x55cbf323f680 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3f780111d0 tx=0x7f3f78009dd0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 -- 172.21.15.31:0/967279649 shutdown_connections 2024-04-26T17:56:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/967279649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf32454f0 0x55cbf323f680 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/967279649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf3236720 0x55cbf325ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/967279649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 0x55cbf325a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 -- 172.21.15.31:0/967279649 >> 172.21.15.31:0/967279649 conn(0x55cbf3232410 msgr2=0x55cbf3238280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 -- 172.21.15.31:0/967279649 shutdown_connections 2024-04-26T17:56:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 -- 172.21.15.31:0/967279649 wait complete. 2024-04-26T17:56:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 -- start start 2024-04-26T17:56:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 Processor -- start 2024-04-26T17:56:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 0x55cbf3258520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3258a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf32454f0 0x55cbf3258fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801aa920 con 0x55cbf3236720 2024-04-26T17:56:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801abd90 con 0x55cbf3235e00 2024-04-26T17:56:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801aaff0 con 0x55cbf32454f0 2024-04-26T17:56:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8ada6740 10 client.0 _ll_register_callbacks cb 0x7f3f78009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:56:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8561e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3258a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8561e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3258a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37956/0 (socket says 172.21.15.31:37956) 2024-04-26T17:56:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.392+0000 7f3f8561e640 1 -- 172.21.15.31:0/3593539129 learned_addr learned my addr 172.21.15.31:0/3593539129 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f85e1f640 1 --2- 172.21.15.31:0/3593539129 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 0x55cbf3258520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f86620640 1 --2- 172.21.15.31:0/3593539129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf32454f0 0x55cbf3258fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f8561e640 1 -- 172.21.15.31:0/3593539129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf32454f0 msgr2=0x55cbf3258fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f8561e640 1 --2- 172.21.15.31:0/3593539129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf32454f0 0x55cbf3258fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f8561e640 1 -- 172.21.15.31:0/3593539129 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 msgr2=0x55cbf3258520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f8561e640 1 --2- 172.21.15.31:0/3593539129 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 0x55cbf3258520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f8561e640 1 -- 172.21.15.31:0/3593539129 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbf32ba8a0 con 0x55cbf3236720 2024-04-26T17:56:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f86620640 1 --2- 172.21.15.31:0/3593539129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf32454f0 0x55cbf3258fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:56:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f8561e640 1 --2- 172.21.15.31:0/3593539129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3258a60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3f7009c120 tx=0x7f3f70009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892615e6223cb43 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f697fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:56:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f697fa640 1 -- 172.21.15.31:0/3593539129 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7009c890 con 0x55cbf3236720 2024-04-26T17:56:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f697fa640 1 -- 172.21.15.31:0/3593539129 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f3f7009b050 con 0x55cbf3236720 2024-04-26T17:56:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f8ada6740 1 -- 172.21.15.31:0/3593539129 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f78004030 con 0x55cbf3236720 2024-04-26T17:56:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f697fa640 1 -- 172.21.15.31:0/3593539129 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3f7007e020 con 0x55cbf3236720 2024-04-26T17:56:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f8ada6740 10 client.15270 fetch_fsmap learned FSMap version 26 2024-04-26T17:56:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f8ada6740 1 -- 172.21.15.31:0/3593539129 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cbf32bbb50 con 0x55cbf3236720 2024-04-26T17:56:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.393+0000 7f3f697fa640 1 -- 172.21.15.31:0/3593539129 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f78004030 con 0x55cbf3236720 2024-04-26T17:56:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.394+0000 7f3f697fa640 1 -- 172.21.15.31:0/3593539129 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3f70098800 con 0x55cbf3236720 2024-04-26T17:56:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.394+0000 7f3f8ada6740 10 client.15270 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T17:56:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.394+0000 7f3f8ada6740 10 client.15270 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:56:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.394+0000 7f3f8ada6740 20 client.15270 populate_metadata read hostname 'smithi031' 2024-04-26T17:56:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.394+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/3593539129 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55cbf32bbfc0 0x55cbf32be3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.394+0000 7f3f8ada6740 4 client.15270 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:56:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.394+0000 7f3f8ada6740 1 -- 172.21.15.31:0/3593539129 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cbf31a3ee0 con 0x55cbf32bbfc0 2024-04-26T17:56:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.394+0000 7f3f85e1f640 1 --2- 172.21.15.31:0/3593539129 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55cbf32bbfc0 0x55cbf32be3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.394+0000 7f3f697fa640 10 client.15270 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:56:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.394+0000 7f3f85e1f640 1 --2- 172.21.15.31:0/3593539129 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55cbf32bbfc0 0x55cbf32be3e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17543a87584c9b40 server_cookie=904b9e60e7e9ca98 in_seq=0 out_seq=0 2024-04-26T17:56:45.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f697fa640 1 -- 172.21.15.31:0/3593539129 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cbf31a3ee0 con 0x55cbf32bbfc0 2024-04-26T17:56:45.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f697fa640 10 client.15270 handle_command_reply: tid=0 2024-04-26T17:56:45.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f8ada6740 1 client.15270 shutdown 2024-04-26T17:56:45.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f8ada6740 1 -- 172.21.15.31:0/3593539129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 msgr2=0x55cbf3258a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/3593539129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3258a60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3f7009c120 tx=0x7f3f70009f30 comp rx=0 tx=0).stop 2024-04-26T17:56:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f8ada6740 1 -- 172.21.15.31:0/3593539129 shutdown_connections 2024-04-26T17:56:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/3593539129 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55cbf32bbfc0 0x55cbf32be3e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/3593539129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbf32454f0 0x55cbf3258fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/3593539129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cbf3236720 0x55cbf3258a60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f8ada6740 1 --2- 172.21.15.31:0/3593539129 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cbf3235e00 0x55cbf3258520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f697fa640 0 client.15270 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:56:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f8ada6740 1 -- 172.21.15.31:0/3593539129 >> 172.21.15.31:0/3593539129 conn(0x55cbf3232410 msgr2=0x55cbf32423c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f8ada6740 1 -- 172.21.15.31:0/3593539129 shutdown_connections 2024-04-26T17:56:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f8ada6740 1 -- 172.21.15.31:0/3593539129 wait complete. 2024-04-26T17:56:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.405+0000 7f3f8ada6740 20 client.15270 trim_cache size 0 max 16384 2024-04-26T17:56:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:56:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:56:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:56:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:56:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.406+0000 7f3f88083640 1 -- 172.21.15.31:0/2956673663 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3f6c0a3e60 msgr2=0x7f3f6c0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.406+0000 7f3f88083640 1 --2- 172.21.15.31:0/2956673663 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3f6c0a3e60 0x7f3f6c0a6320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c031440 tx=0x7f3f7c005b50 comp rx=0 tx=0).stop 2024-04-26T17:56:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.406+0000 7f3f88083640 1 -- 172.21.15.31:0/2956673663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f80104500 msgr2=0x7f3f801995f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.406+0000 7f3f88083640 1 --2- 172.21.15.31:0/2956673663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f80104500 0x7f3f801995f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3f7000d950 tx=0x7f3f7000de10 comp rx=0 tx=0).stop 2024-04-26T17:56:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.406+0000 7f3f88083640 1 -- 172.21.15.31:0/2956673663 shutdown_connections 2024-04-26T17:56:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.406+0000 7f3f88083640 1 --2- 172.21.15.31:0/2956673663 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3f8010cfb0 0x7f3f8019d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.406+0000 7f3f88083640 1 --2- 172.21.15.31:0/2956673663 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3f6c0a3e60 0x7f3f6c0a6320 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.406+0000 7f3f88083640 1 --2- 172.21.15.31:0/2956673663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f80104500 0x7f3f801995f0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.406+0000 7f3f88083640 1 --2- 172.21.15.31:0/2956673663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f80103be0 0x7f3f801990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.406+0000 7f3f88083640 1 -- 172.21.15.31:0/2956673663 >> 172.21.15.31:0/2956673663 conn(0x7f3f80077bb0 msgr2=0x7f3f800fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.406+0000 7f3f88083640 1 -- 172.21.15.31:0/2956673663 shutdown_connections 2024-04-26T17:56:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:45.406+0000 7f3f88083640 1 -- 172.21.15.31:0/2956673663 wait complete. 2024-04-26T17:56:45.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:45 smithi186 ceph-mon[32797]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-26T17:56:45.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.766+0000 7fe08fdeb700 1 -- 172.21.15.31:0/1777206355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08806bad0 msgr2=0x7fe0881102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:45.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.766+0000 7fe08fdeb700 1 --2- 172.21.15.31:0/1777206355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08806bad0 0x7fe0881102e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe08000b0a0 tx=0x7fe08000b3b0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.766+0000 7fe08fdeb700 1 -- 172.21.15.31:0/1777206355 shutdown_connections 2024-04-26T17:56:45.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.766+0000 7fe08fdeb700 1 --2- 172.21.15.31:0/1777206355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08806bad0 0x7fe0881102e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.766+0000 7fe08fdeb700 1 --2- 172.21.15.31:0/1777206355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0881094a0 0x7fe088109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.766+0000 7fe08fdeb700 1 --2- 172.21.15.31:0/1777206355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe08806b130 0x7fe08806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.766+0000 7fe08fdeb700 1 -- 172.21.15.31:0/1777206355 >> 172.21.15.31:0/1777206355 conn(0x7fe0880792d0 msgr2=0x7fe0880796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.766+0000 7fe08fdeb700 1 -- 172.21.15.31:0/1777206355 shutdown_connections 2024-04-26T17:56:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.766+0000 7fe08fdeb700 1 -- 172.21.15.31:0/1777206355 wait complete. 2024-04-26T17:56:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08fdeb700 1 Processor -- start 2024-04-26T17:56:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08fdeb700 1 -- start start 2024-04-26T17:56:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08fdeb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08806b130 0x7fe08819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08fdeb700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe08806bad0 0x7fe08819fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08fdeb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0881094a0 0x7fe088199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08fdeb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0881135a0 con 0x7fe08806b130 2024-04-26T17:56:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08fdeb700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe088113420 con 0x7fe08806bad0 2024-04-26T17:56:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08fdeb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe088113720 con 0x7fe0881094a0 2024-04-26T17:56:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08db87700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08806b130 0x7fe08819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08db87700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08806b130 0x7fe08819f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37976/0 (socket says 172.21.15.31:37976) 2024-04-26T17:56:45.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08db87700 1 -- 172.21.15.31:0/4221147465 learned_addr learned my addr 172.21.15.31:0/4221147465 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:45.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08d386700 1 --2- 172.21.15.31:0/4221147465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe08806bad0 0x7fe08819fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08e388700 1 --2- 172.21.15.31:0/4221147465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0881094a0 0x7fe088199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08db87700 1 -- 172.21.15.31:0/4221147465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0881094a0 msgr2=0x7fe088199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:45.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08db87700 1 --2- 172.21.15.31:0/4221147465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0881094a0 0x7fe088199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08db87700 1 -- 172.21.15.31:0/4221147465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe08806bad0 msgr2=0x7fe08819fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:45.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08db87700 1 --2- 172.21.15.31:0/4221147465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe08806bad0 0x7fe08819fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:45.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.767+0000 7fe08db87700 1 -- 172.21.15.31:0/4221147465 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe08819a080 con 0x7fe08806b130 2024-04-26T17:56:45.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.768+0000 7fe08db87700 1 --2- 172.21.15.31:0/4221147465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08806b130 0x7fe08819f6f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe08400d950 tx=0x7fe08400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:45.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.768+0000 7fe07effd700 1 -- 172.21.15.31:0/4221147465 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe084016070 con 0x7fe08806b130 2024-04-26T17:56:45.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.768+0000 7fe08fdeb700 1 -- 172.21.15.31:0/4221147465 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe08819a370 con 0x7fe08806b130 2024-04-26T17:56:45.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.768+0000 7fe07effd700 1 -- 172.21.15.31:0/4221147465 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe084010460 con 0x7fe08806b130 2024-04-26T17:56:45.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.768+0000 7fe07effd700 1 -- 172.21.15.31:0/4221147465 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08400f620 con 0x7fe08806b130 2024-04-26T17:56:45.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.768+0000 7fe08fdeb700 1 -- 172.21.15.31:0/4221147465 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0881a65c0 con 0x7fe08806b130 2024-04-26T17:56:45.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.769+0000 7fe08fdeb700 1 -- 172.21.15.31:0/4221147465 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe08806a110 con 0x7fe08806b130 2024-04-26T17:56:45.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.770+0000 7fe07effd700 1 -- 172.21.15.31:0/4221147465 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fe084019070 con 0x7fe08806b130 2024-04-26T17:56:45.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.771+0000 7fe07effd700 1 --2- 172.21.15.31:0/4221147465 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe0740a3e40 0x7fe0740a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:45.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.771+0000 7fe07effd700 1 -- 172.21.15.31:0/4221147465 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fe0840d04f0 con 0x7fe08806b130 2024-04-26T17:56:45.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.771+0000 7fe08d386700 1 --2- 172.21.15.31:0/4221147465 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe0740a3e40 0x7fe0740a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:45.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.771+0000 7fe08d386700 1 --2- 172.21.15.31:0/4221147465 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe0740a3e40 0x7fe0740a6300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe078005950 tx=0x7fe0780058e0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:45.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:45.775+0000 7fe07effd700 1 -- 172.21.15.31:0/4221147465 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe08409ca70 con 0x7fe08806b130 2024-04-26T17:56:46.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:46.029+0000 7fe08fdeb700 1 -- 172.21.15.31:0/4221147465 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe08802e490 con 0x7fe08806b130 2024-04-26T17:56:46.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:46.030+0000 7fe07effd700 1 -- 172.21.15.31:0/4221147465 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11871 (secure 0 0 0) 0x7fe08409c370 con 0x7fe08806b130 2024-04-26T17:56:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:46.029 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T17:56:28.628701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24446,24457,24473]},"id":1}]} 2024-04-26T17:56:46.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:46.032+0000 7fe08fdeb700 1 -- 172.21.15.31:0/4221147465 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe0740a3e40 msgr2=0x7fe0740a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:46.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:46.032+0000 7fe08fdeb700 1 --2- 172.21.15.31:0/4221147465 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe0740a3e40 0x7fe0740a6300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe078005950 tx=0x7fe0780058e0 comp rx=0 tx=0).stop 2024-04-26T17:56:46.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:46.032+0000 7fe08fdeb700 1 -- 172.21.15.31:0/4221147465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08806b130 msgr2=0x7fe08819f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:46.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:46.032+0000 7fe08fdeb700 1 --2- 172.21.15.31:0/4221147465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08806b130 0x7fe08819f6f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe08400d950 tx=0x7fe08400dd10 comp rx=0 tx=0).stop 2024-04-26T17:56:46.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:46.032+0000 7fe08fdeb700 1 -- 172.21.15.31:0/4221147465 shutdown_connections 2024-04-26T17:56:46.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:46.032+0000 7fe08fdeb700 1 --2- 172.21.15.31:0/4221147465 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe0740a3e40 0x7fe0740a6300 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:46.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:46.032+0000 7fe08fdeb700 1 --2- 172.21.15.31:0/4221147465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0881094a0 0x7fe088199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:46.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:46.032+0000 7fe08fdeb700 1 --2- 172.21.15.31:0/4221147465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08806b130 0x7fe08819f6f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:46.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:46.032+0000 7fe08fdeb700 1 --2- 172.21.15.31:0/4221147465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe08806bad0 0x7fe08819fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:46.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:46.032+0000 7fe08fdeb700 1 -- 172.21.15.31:0/4221147465 >> 172.21.15.31:0/4221147465 conn(0x7fe0880792d0 msgr2=0x7fe088100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:46.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:46.032+0000 7fe08fdeb700 1 -- 172.21.15.31:0/4221147465 shutdown_connections 2024-04-26T17:56:46.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:46.032+0000 7fe08fdeb700 1 -- 172.21.15.31:0/4221147465 wait complete. 2024-04-26T17:56:46.032 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-26T17:56:46.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:46 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/4221147465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:46.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:46 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/4221147465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:46.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:46 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/4221147465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:46.715 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T17:56:46.968 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:47.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:47 smithi031 ceph-mon[29164]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 78 KiB/s wr, 53 op/s 2024-04-26T17:56:47.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:47 smithi097 ceph-mon[32419]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 78 KiB/s wr, 53 op/s 2024-04-26T17:56:47.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:47 smithi186 ceph-mon[32797]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 78 KiB/s wr, 53 op/s 2024-04-26T17:56:48.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.696+0000 7effc2b8d700 1 -- 172.21.15.31:0/3292910035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effbc102320 msgr2=0x7effbc1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.696+0000 7effc2b8d700 1 --2- 172.21.15.31:0/3292910035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effbc102320 0x7effbc1027a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7effa8009990 tx=0x7effa8009ca0 comp rx=0 tx=0).stop 2024-04-26T17:56:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.696+0000 7effc2b8d700 1 -- 172.21.15.31:0/3292910035 shutdown_connections 2024-04-26T17:56:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.696+0000 7effc2b8d700 1 --2- 172.21.15.31:0/3292910035 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effbc102ce0 0x7effbc10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.696+0000 7effc2b8d700 1 --2- 172.21.15.31:0/3292910035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc108390 0x7effbc108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.696+0000 7effc2b8d700 1 --2- 172.21.15.31:0/3292910035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effbc102320 0x7effbc1027a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.696+0000 7effc2b8d700 1 -- 172.21.15.31:0/3292910035 >> 172.21.15.31:0/3292910035 conn(0x7effbc0792d0 msgr2=0x7effbc0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.696+0000 7effc2b8d700 1 -- 172.21.15.31:0/3292910035 shutdown_connections 2024-04-26T17:56:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.696+0000 7effc2b8d700 1 -- 172.21.15.31:0/3292910035 wait complete. 2024-04-26T17:56:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.697+0000 7effc2b8d700 1 Processor -- start 2024-04-26T17:56:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.697+0000 7effc2b8d700 1 -- start start 2024-04-26T17:56:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.697+0000 7effc2b8d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc102320 0x7effbc19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.697+0000 7effc2b8d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effbc102ce0 0x7effbc19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.697+0000 7effc2b8d700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effbc108390 0x7effbc198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.697+0000 7effc2b8d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1124d0 con 0x7effbc102ce0 2024-04-26T17:56:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.697+0000 7effc2b8d700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7effbc112350 con 0x7effbc108390 2024-04-26T17:56:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.697+0000 7effc2b8d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7effbc112650 con 0x7effbc102320 2024-04-26T17:56:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.697+0000 7effbbfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effbc102ce0 0x7effbc19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.697+0000 7effbbfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effbc102ce0 0x7effbc19ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38004/0 (socket says 172.21.15.31:38004) 2024-04-26T17:56:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.697+0000 7effbbfff700 1 -- 172.21.15.31:0/3626277783 learned_addr learned my addr 172.21.15.31:0/3626277783 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.697+0000 7effc112a700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effbc108390 0x7effbc198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.697+0000 7effc0929700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc102320 0x7effbc19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.698+0000 7effbbfff700 1 -- 172.21.15.31:0/3626277783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc102320 msgr2=0x7effbc19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.698+0000 7effbbfff700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc102320 0x7effbc19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.698+0000 7effbbfff700 1 -- 172.21.15.31:0/3626277783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effbc108390 msgr2=0x7effbc198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.698+0000 7effbbfff700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effbc108390 0x7effbc198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.698+0000 7effbbfff700 1 -- 172.21.15.31:0/3626277783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effbc199030 con 0x7effbc102ce0 2024-04-26T17:56:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.698+0000 7effc112a700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effbc108390 0x7effbc198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:56:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.698+0000 7effbbfff700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effbc102ce0 0x7effbc19ec10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7effa8000c00 tx=0x7effa80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.698+0000 7effb9ffb700 1 -- 172.21.15.31:0/3626277783 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa8027070 con 0x7effbc102ce0 2024-04-26T17:56:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.698+0000 7effc2b8d700 1 -- 172.21.15.31:0/3626277783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effbc1992c0 con 0x7effbc102ce0 2024-04-26T17:56:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.698+0000 7effb9ffb700 1 -- 172.21.15.31:0/3626277783 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effa8003f20 con 0x7effbc102ce0 2024-04-26T17:56:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.698+0000 7effb9ffb700 1 -- 172.21.15.31:0/3626277783 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa8022ca0 con 0x7effbc102ce0 2024-04-26T17:56:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.698+0000 7effc2b8d700 1 -- 172.21.15.31:0/3626277783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effbc1a5530 con 0x7effbc102ce0 2024-04-26T17:56:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.699+0000 7effc2b8d700 1 -- 172.21.15.31:0/3626277783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effbc103a40 con 0x7effbc102ce0 2024-04-26T17:56:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.701+0000 7effb9ffb700 1 -- 172.21.15.31:0/3626277783 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7effa802a070 con 0x7effbc102ce0 2024-04-26T17:56:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.701+0000 7effb9ffb700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7effac0a3e40 0x7effac0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.701+0000 7effc0929700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7effac0a3e40 0x7effac0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.701+0000 7effb9ffb700 1 -- 172.21.15.31:0/3626277783 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7effa80d1ee0 con 0x7effbc102ce0 2024-04-26T17:56:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.702+0000 7effc0929700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7effac0a3e40 0x7effac0a6300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7effb00045b0 tx=0x7effb000b3c0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:48.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.705+0000 7effb9ffb700 1 -- 172.21.15.31:0/3626277783 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7effa8031070 con 0x7effbc102ce0 2024-04-26T17:56:48.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.915+0000 7effc2b8d700 1 -- 172.21.15.31:0/3626277783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7effbc199d80 con 0x7effbc102ce0 2024-04-26T17:56:48.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.917+0000 7effb9ffb700 1 -- 172.21.15.31:0/3626277783 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22953 (secure 0 0 0) 0x7effa8013070 con 0x7effbc102ce0 2024-04-26T17:56:48.916 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:48.917 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":91,"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","created":"2024-04-26T17:44:48.618250+0000","modified":"2024-04-26T17:56:29.776619+0000","last_up_change":"2024-04-26T17:52:00.798886+0000","last_in_change":"2024-04-26T17:51:40.613593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T17:48:12.307250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T17:52:58.650338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T17:53:02.488144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f684ee5-eeee-454a-a00b-e5d6d2901572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6803","nonce":1957398328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6805","nonce":1957398328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6809","nonce":1957398328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6807","nonce":1957398328}]},"public_addr":"172.21.15.31:6803/1957398328","cluster_addr":"172.21.15.31:6805/1957398328","heartbeat_back_addr":"172.21.15.31:6809/1957398328","heartbeat_front_addr":"172.21.15.31:6807/1957398328","state":["exists","up"]},{"osd":1,"uuid":"08be6b26-cb60-41f7-be8b-01a0c93ebafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6801","nonce":4209866427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6803","nonce":4209866427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6807","nonce":4209866427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6805","nonce":4209866427}]},"public_addr":"172.21.15.97:6801/4209866427","cluster_addr":"172.21.15.97:6803/4209866427","heartbeat_back_addr":"172.21.15.97:6807/4209866427","heartbeat_front_addr":"172.21.15.97:6805/4209866427","state":["exists","up"]},{"osd":2,"uuid":"498216b1-a0ac-4923-ade8-ce50fb97d472","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6801","nonce":223535147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6803","nonce":223535147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6807","nonce":223535147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6805","nonce":223535147}]},"public_addr":"172.21.15.186:6801/223535147","cluster_addr":"172.21.15.186:6803/223535147","heartbeat_back_addr":"172.21.15.186:6807/223535147","heartbeat_front_addr":"172.21.15.186:6805/223535147","state":["exists","up"]},{"osd":3,"uuid":"568a352d-a430-4f0f-9a49-6f992e0ceae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6811","nonce":2640112341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6813","nonce":2640112341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6817","nonce":2640112341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6815","nonce":2640112341}]},"public_addr":"172.21.15.31:6811/2640112341","cluster_addr":"172.21.15.31:6813/2640112341","heartbeat_back_addr":"172.21.15.31:6817/2640112341","heartbeat_front_addr":"172.21.15.31:6815/2640112341","state":["exists","up"]},{"osd":4,"uuid":"112ccbf7-24e8-473c-9852-2e7167dd3108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6809","nonce":1359263143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6811","nonce":1359263143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6815","nonce":1359263143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6813","nonce":1359263143}]},"public_addr":"172.21.15.97:6809/1359263143","cluster_addr":"172.21.15.97:6811/1359263143","heartbeat_back_addr":"172.21.15.97:6815/1359263143","heartbeat_front_addr":"172.21.15.97:6813/1359263143","state":["exists","up"]},{"osd":5,"uuid":"eb17759e-1ad8-4bda-9f61-54d2a9781535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6809","nonce":2006134626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6811","nonce":2006134626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6815","nonce":2006134626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6813","nonce":2006134626}]},"public_addr":"172.21.15.186:6809/2006134626","cluster_addr":"172.21.15.186:6811/2006134626","heartbeat_back_addr":"172.21.15.186:6815/2006134626","heartbeat_front_addr":"172.21.15.186:6813/2006134626","state":["exists","up"]},{"osd":6,"uuid":"f1d11e38-40e9-4db4-868f-145f1ffc0d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6819","nonce":2955759395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6821","nonce":2955759395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6825","nonce":2955759395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6823","nonce":2955759395}]},"public_addr":"172.21.15.31:6819/2955759395","cluster_addr":"172.21.15.31:6821/2955759395","heartbeat_back_addr":"172.21.15.31:6825/2955759395","heartbeat_front_addr":"172.21.15.31:6823/2955759395","state":["exists","up"]},{"osd":7,"uuid":"4e95f728-23ff-4bca-9b94-afde2662a5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6817","nonce":3940083075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6819","nonce":3940083075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6823","nonce":3940083075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6821","nonce":3940083075}]},"public_addr":"172.21.15.97:6817/3940083075","cluster_addr":"172.21.15.97:6819/3940083075","heartbeat_back_addr":"172.21.15.97:6823/3940083075","heartbeat_front_addr":"172.21.15.97:6821/3940083075","state":["exists","up"]},{"osd":8,"uuid":"5f91c915-2568-4d20-b645-00dad2efe952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6817","nonce":3193558170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6819","nonce":3193558170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6823","nonce":3193558170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6821","nonce":3193558170}]},"public_addr":"172.21.15.186:6817/3193558170","cluster_addr":"172.21.15.186:6819/3193558170","heartbeat_back_addr":"172.21.15.186:6823/3193558170","heartbeat_front_addr":"172.21.15.186:6821/3193558170","state":["exists","up"]},{"osd":9,"uuid":"12fda87d-830a-461f-a6cb-4699c52dc5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6827","nonce":3660801905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6829","nonce":3660801905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6833","nonce":3660801905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6831","nonce":3660801905}]},"public_addr":"172.21.15.31:6827/3660801905","cluster_addr":"172.21.15.31:6829/3660801905","heartbeat_back_addr":"172.21.15.31:6833/3660801905","heartbeat_front_addr":"172.21.15.31:6831/3660801905","state":["exists","up"]},{"osd":10,"uuid":"92a639b3-918b-485c-bd30-258b4224993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6825","nonce":4097344903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6827","nonce":4097344903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6831","nonce":4097344903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6829","nonce":4097344903}]},"public_addr":"172.21.15.97:6825/4097344903","cluster_addr":"172.21.15.97:6827/4097344903","heartbeat_back_addr":"172.21.15.97:6831/4097344903","heartbeat_front_addr":"172.21.15.97:6829/4097344903","state":["exists","up"]},{"osd":11,"uuid":"842aa599-d21d-4e4a-b872-e9ac20d8c0ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6825","nonce":1063973352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6827","nonce":1063973352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6831","nonce":1063973352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6829","nonce":1063973352}]},"public_addr":"172.21.15.186:6825/1063973352","cluster_addr":"172.21.15.186:6827/1063973352","heartbeat_back_addr":"172.21.15.186:6831/1063973352","heartbeat_front_addr":"172.21.15.186:6829/1063973352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:21.305331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:44.877932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:08.784665+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:34.381762+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:58.999358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:23.620237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:49.961719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:15.363702+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:40.599703+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:07.391645+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:32.800851+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:59.384148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6801/2410008001":"2024-04-27T17:56:29.776588+0000","172.21.15.31:6800/2410008001":"2024-04-27T17:56:29.776588+0000","172.21.15.31:0/540352687":"2024-04-27T17:56:29.776588+0000","172.21.15.31:0/2095760002":"2024-04-27T17:56:29.776588+0000","172.21.15.31:0/3429421652":"2024-04-27T17:56:29.776588+0000","172.21.15.31:0/718966776":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/2035551671":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/614760140":"2024-04-27T17:56:29.776588+0000","172.21.15.31:6801/153341213":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/810461143":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/3148082955":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6801/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/288292084":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/3101596726":"2024-04-27T17:45:21.129062+0000","172.21.15.31:6800/153341213":"2024-04-27T17:45:48.218816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T17:56:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.919+0000 7effc2b8d700 1 -- 172.21.15.31:0/3626277783 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7effac0a3e40 msgr2=0x7effac0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.919+0000 7effc2b8d700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7effac0a3e40 0x7effac0a6300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7effb00045b0 tx=0x7effb000b3c0 comp rx=0 tx=0).stop 2024-04-26T17:56:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.919+0000 7effc2b8d700 1 -- 172.21.15.31:0/3626277783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effbc102ce0 msgr2=0x7effbc19ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.919+0000 7effc2b8d700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effbc102ce0 0x7effbc19ec10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7effa8000c00 tx=0x7effa80047b0 comp rx=0 tx=0).stop 2024-04-26T17:56:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.919+0000 7effc2b8d700 1 -- 172.21.15.31:0/3626277783 shutdown_connections 2024-04-26T17:56:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.919+0000 7effc2b8d700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7effac0a3e40 0x7effac0a6300 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.919+0000 7effc2b8d700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7effbc108390 0x7effbc198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.919+0000 7effc2b8d700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc102320 0x7effbc19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.919+0000 7effc2b8d700 1 --2- 172.21.15.31:0/3626277783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effbc102ce0 0x7effbc19ec10 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.919+0000 7effc2b8d700 1 -- 172.21.15.31:0/3626277783 >> 172.21.15.31:0/3626277783 conn(0x7effbc0792d0 msgr2=0x7effbc100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.919+0000 7effc2b8d700 1 -- 172.21.15.31:0/3626277783 shutdown_connections 2024-04-26T17:56:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:48.920+0000 7effc2b8d700 1 -- 172.21.15.31:0/3626277783 wait complete. 2024-04-26T17:56:49.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:49 smithi031 ceph-mon[29164]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-26T17:56:49.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:49 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3626277783' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:56:49.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:49 smithi097 ceph-mon[32419]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-26T17:56:49.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:49 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3626277783' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:56:49.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:49 smithi186 ceph-mon[32797]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-26T17:56:49.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:49 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3626277783' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T17:56:49.574 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-04-26T17:56:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:51.253 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:51 smithi031 ceph-mon[29164]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-26T17:56:51.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:51 smithi097 ceph-mon[32419]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-26T17:56:51.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:51 smithi186 ceph-mon[32797]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-26T17:56:52.190 INFO:teuthology.orchestra.run.smithi031.stdout:{"run_dir":"/var/run/ceph"} 2024-04-26T17:56:52.190 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-04-26T17:56:52.191 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 9496 2024-04-26T17:56:52.191 DEBUG:tasks.quiescer:Yielding 2024-04-26T17:56:52.191 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T17:56:52.202 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T17:56:52.202 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi031.front.sepia.ceph.com 2024-04-26T17:56:52.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2024-04-26T17:56:52.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:56:52.240 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:52.254 INFO:teuthology.orchestra.run.smithi031.stdout:Filesystem Size Used Avail Use% Mounted on 2024-04-26T17:56:52.255 INFO:teuthology.orchestra.run.smithi031.stdout:0@7e53e050-03f4-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/26ef2745-8372-4260-b242-0000b032cf66 340G 0 340G 0% /tmp/cephfs 2024-04-26T17:56:52.257 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2024-04-26T17:56:52.346 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2024-04-26T17:56:52.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:52 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:56:52.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:52 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:56:52.385 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2024-04-26T17:56:52.461 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:52.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:52 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:56:53.225 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:53 smithi031 ceph-mon[29164]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-04-26T17:56:53.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:53 smithi097 ceph-mon[32419]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-04-26T17:56:53.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:53 smithi186 ceph-mon[32797]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-04-26T17:56:54.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2024-04-26T17:56:54.265 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2024-04-26T17:56:54.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2024-04-26T17:56:54.347 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-26T17:56:54.423 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/: 2024-04-26T17:56:54.423 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:54.423 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 . 2024-04-26T17:56:54.423 INFO:teuthology.orchestra.run.smithi031.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 .. 2024-04-26T17:56:54.423 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 data 2024-04-26T17:56:54.423 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:54.423 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data: 2024-04-26T17:56:54.423 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:54.423 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:54.423 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 .. 2024-04-26T17:56:54.425 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2024-04-26T17:56:54.510 INFO:teuthology.orchestra.run.smithi031.stdout:[Service] 2024-04-26T17:56:54.511 INFO:teuthology.orchestra.run.smithi031.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2024-04-26T17:56:54.511 INFO:teuthology.orchestra.run.smithi031.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2024-04-26T17:56:54.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2024-04-26T17:56:54.648 INFO:teuthology.orchestra.run.smithi031.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2024-04-26T17:56:54.770 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:54 smithi031 ceph-mon[29164]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-26T17:56:54.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.877+0000 7fb8086f5700 1 -- 172.21.15.31:0/3463625244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb800102ce0 msgr2=0x7fb80010f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:54.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.877+0000 7fb8086f5700 1 --2- 172.21.15.31:0/3463625244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb800102ce0 0x7fb80010f1d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800b0a0 tx=0x7fb7f800b3b0 comp rx=0 tx=0).stop 2024-04-26T17:56:54.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.878+0000 7fb8086f5700 1 -- 172.21.15.31:0/3463625244 shutdown_connections 2024-04-26T17:56:54.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.878+0000 7fb8086f5700 1 --2- 172.21.15.31:0/3463625244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb800102ce0 0x7fb80010f1d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:54.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.878+0000 7fb8086f5700 1 --2- 172.21.15.31:0/3463625244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb800108390 0x7fb800108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:54.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.878+0000 7fb8086f5700 1 --2- 172.21.15.31:0/3463625244 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb800102320 0x7fb8001027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:54.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.878+0000 7fb8086f5700 1 -- 172.21.15.31:0/3463625244 >> 172.21.15.31:0/3463625244 conn(0x7fb8000792d0 msgr2=0x7fb8000796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:54.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.878+0000 7fb8086f5700 1 -- 172.21.15.31:0/3463625244 shutdown_connections 2024-04-26T17:56:54.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.878+0000 7fb8086f5700 1 -- 172.21.15.31:0/3463625244 wait complete. 2024-04-26T17:56:54.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.878+0000 7fb8086f5700 1 Processor -- start 2024-04-26T17:56:54.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.878+0000 7fb8086f5700 1 -- start start 2024-04-26T17:56:54.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb8086f5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb800102320 0x7fb80019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:54.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb8086f5700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb800102ce0 0x7fb80019ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:54.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb8086f5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb800108390 0x7fb800198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:54.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb8086f5700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb8001124b0 con 0x7fb800102320 2024-04-26T17:56:54.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb8086f5700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb800112330 con 0x7fb800102ce0 2024-04-26T17:56:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb8086f5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb800112630 con 0x7fb800108390 2024-04-26T17:56:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb806491700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb800102320 0x7fb80019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb806491700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb800102320 0x7fb80019e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39908/0 (socket says 172.21.15.31:39908) 2024-04-26T17:56:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb806491700 1 -- 172.21.15.31:0/4175266417 learned_addr learned my addr 172.21.15.31:0/4175266417 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb805c90700 1 --2- 172.21.15.31:0/4175266417 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb800102ce0 0x7fb80019ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb806491700 1 -- 172.21.15.31:0/4175266417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb800108390 msgr2=0x7fb800198870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T17:56:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb806491700 1 --2- 172.21.15.31:0/4175266417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb800108390 0x7fb800198870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb806491700 1 -- 172.21.15.31:0/4175266417 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb800102ce0 msgr2=0x7fb80019ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb806491700 1 --2- 172.21.15.31:0/4175266417 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb800102ce0 0x7fb80019ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb806491700 1 -- 172.21.15.31:0/4175266417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb800199010 con 0x7fb800102320 2024-04-26T17:56:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.879+0000 7fb806491700 1 --2- 172.21.15.31:0/4175266417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb800102320 0x7fb80019e6c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc007b50 tx=0x7fb7fc007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.880+0000 7fb7f77fe700 1 -- 172.21.15.31:0/4175266417 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc01a070 con 0x7fb800102320 2024-04-26T17:56:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.880+0000 7fb7f77fe700 1 -- 172.21.15.31:0/4175266417 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7fc00ea70 con 0x7fb800102320 2024-04-26T17:56:54.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.880+0000 7fb8086f5700 1 -- 172.21.15.31:0/4175266417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb800199300 con 0x7fb800102320 2024-04-26T17:56:54.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.880+0000 7fb7f77fe700 1 -- 172.21.15.31:0/4175266417 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc00aa30 con 0x7fb800102320 2024-04-26T17:56:54.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.880+0000 7fb8086f5700 1 -- 172.21.15.31:0/4175266417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb80006b2d0 con 0x7fb800102320 2024-04-26T17:56:54.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.881+0000 7fb8086f5700 1 -- 172.21.15.31:0/4175266417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb800103a40 con 0x7fb800102320 2024-04-26T17:56:54.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.882+0000 7fb7f77fe700 1 -- 172.21.15.31:0/4175266417 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fb7fc00abf0 con 0x7fb800102320 2024-04-26T17:56:54.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.883+0000 7fb7f77fe700 1 --2- 172.21.15.31:0/4175266417 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fb7ec0a3ec0 0x7fb7ec0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:54.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.883+0000 7fb7f77fe700 1 -- 172.21.15.31:0/4175266417 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fb7fc0d1aa0 con 0x7fb800102320 2024-04-26T17:56:54.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.883+0000 7fb805c90700 1 --2- 172.21.15.31:0/4175266417 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fb7ec0a3ec0 0x7fb7ec0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:54.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.883+0000 7fb805c90700 1 --2- 172.21.15.31:0/4175266417 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fb7ec0a3ec0 0x7fb7ec0a6380 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0005ea0 tx=0x7fb7f0005e30 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:54.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:54 smithi097 ceph-mon[32419]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-26T17:56:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:54.887+0000 7fb7f77fe700 1 -- 172.21.15.31:0/4175266417 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb7fc023020 con 0x7fb800102320 2024-04-26T17:56:55.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:54 smithi186 ceph-mon[32797]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-26T17:56:55.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:55.146+0000 7fb8086f5700 1 -- 172.21.15.31:0/4175266417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb80006a110 con 0x7fb800102320 2024-04-26T17:56:55.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:55.147+0000 7fb7f77fe700 1 -- 172.21.15.31:0/4175266417 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11872 (secure 0 0 0) 0x7fb7fc09e020 con 0x7fb800102320 2024-04-26T17:56:55.147 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:55.148 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T17:56:50.756579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24473,24446,24457]},"id":1}]} 2024-04-26T17:56:55.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:55.149+0000 7fb8086f5700 1 -- 172.21.15.31:0/4175266417 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fb7ec0a3ec0 msgr2=0x7fb7ec0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:55.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:55.149+0000 7fb8086f5700 1 --2- 172.21.15.31:0/4175266417 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fb7ec0a3ec0 0x7fb7ec0a6380 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0005ea0 tx=0x7fb7f0005e30 comp rx=0 tx=0).stop 2024-04-26T17:56:55.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:55.149+0000 7fb8086f5700 1 -- 172.21.15.31:0/4175266417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb800102320 msgr2=0x7fb80019e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:55.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:55.149+0000 7fb8086f5700 1 --2- 172.21.15.31:0/4175266417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb800102320 0x7fb80019e6c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc007b50 tx=0x7fb7fc007f10 comp rx=0 tx=0).stop 2024-04-26T17:56:55.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:55.149+0000 7fb8086f5700 1 -- 172.21.15.31:0/4175266417 shutdown_connections 2024-04-26T17:56:55.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:55.149+0000 7fb8086f5700 1 --2- 172.21.15.31:0/4175266417 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fb7ec0a3ec0 0x7fb7ec0a6380 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:55.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:55.149+0000 7fb8086f5700 1 --2- 172.21.15.31:0/4175266417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb800108390 0x7fb800198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:55.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:55.149+0000 7fb8086f5700 1 --2- 172.21.15.31:0/4175266417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb800102320 0x7fb80019e6c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:55.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:55.149+0000 7fb8086f5700 1 --2- 172.21.15.31:0/4175266417 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb800102ce0 0x7fb80019ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:55.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:55.149+0000 7fb8086f5700 1 -- 172.21.15.31:0/4175266417 >> 172.21.15.31:0/4175266417 conn(0x7fb8000792d0 msgr2=0x7fb800100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:55.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:55.150+0000 7fb8086f5700 1 -- 172.21.15.31:0/4175266417 shutdown_connections 2024-04-26T17:56:55.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:55.150+0000 7fb8086f5700 1 -- 172.21.15.31:0/4175266417 wait complete. 2024-04-26T17:56:55.151 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2024-04-26T17:56:55.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:55 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/4175266417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:55.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T17:53:05.425897+0000', 'modified': '2024-04-26T17:56:50.756579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24473, 'mds_1': 24457, 'mds_2': 24446}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24428': {'gid': 24428, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.97:6835/1721162755', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6834', 'nonce': 1721162755}, {'type': 'v1', 'addr': '172.21.15.97:6835', 'nonce': 1721162755}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24439': {'gid': 24439, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6835/580433615', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6834', 'nonce': 580433615}, {'type': 'v1', 'addr': '172.21.15.186:6835', 'nonce': 580433615}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6837/1510454985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6836', 'nonce': 1510454985}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 1510454985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24446': {'gid': 24446, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.97:6837/3334702590', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 3334702590}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 3334702590}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.186:6839/1587952500', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 1587952500}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 1587952500}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.97:6839/3765855684', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6838', 'nonce': 3765855684}, {'type': 'v1', 'addr': '172.21.15.97:6839', 'nonce': 3765855684}]}, 'join_fscid': -1, 'export_targets': [2], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24473, 'qdb_cluster': [24473, 24446, 24457]} 2024-04-26T17:56:55.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T17:56:55.846 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell mds.h status 2024-04-26T17:56:55.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:55 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/4175266417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:56.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:55 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/4175266417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:56:56.106 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:56.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:56 smithi097 ceph-mon[32419]: pgmap v16: 97 pgs: 97 active+clean; 2.0 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 70 KiB/s wr, 16 op/s 2024-04-26T17:56:57.016 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:56 smithi186 ceph-mon[32797]: pgmap v16: 97 pgs: 97 active+clean; 2.0 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 70 KiB/s wr, 16 op/s 2024-04-26T17:56:57.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:56 smithi031 ceph-mon[29164]: pgmap v16: 97 pgs: 97 active+clean; 2.0 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 70 KiB/s wr, 16 op/s 2024-04-26T17:56:57.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.750+0000 7f97c5ffd700 1 -- 172.21.15.31:0/2950895925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0075f70 msgr2=0x7f97c010f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:57.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.750+0000 7f97c5ffd700 1 --2- 172.21.15.31:0/2950895925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0075f70 0x7f97c010f910 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009990 tx=0x7f97a8009ca0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.750+0000 7f97c5ffd700 1 -- 172.21.15.31:0/2950895925 shutdown_connections 2024-04-26T17:56:57.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.750+0000 7f97c5ffd700 1 --2- 172.21.15.31:0/2950895925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97c0075650 0x7f97c0075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.750+0000 7f97c5ffd700 1 --2- 172.21.15.31:0/2950895925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97c010fe50 0x7f97c0112240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.750+0000 7f97c5ffd700 1 --2- 172.21.15.31:0/2950895925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0075f70 0x7f97c010f910 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.750+0000 7f97c5ffd700 1 -- 172.21.15.31:0/2950895925 >> 172.21.15.31:0/2950895925 conn(0x7f97c007a5d0 msgr2=0x7f97c007a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:57.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.750+0000 7f97c5ffd700 1 -- 172.21.15.31:0/2950895925 shutdown_connections 2024-04-26T17:56:57.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.750+0000 7f97c5ffd700 1 -- 172.21.15.31:0/2950895925 wait complete. 2024-04-26T17:56:57.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.750+0000 7f97c5ffd700 1 Processor -- start 2024-04-26T17:56:57.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97c5ffd700 1 -- start start 2024-04-26T17:56:57.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97c5ffd700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97c0075650 0x7f97c019bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97c5ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0075f70 0x7f97c019c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97c5ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97c010fe50 0x7f97c01a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97c5ffd700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f97c0115860 con 0x7f97c0075f70 2024-04-26T17:56:57.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97c5ffd700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f97c01156e0 con 0x7f97c0075650 2024-04-26T17:56:57.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97c5ffd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97c01159e0 con 0x7f97c010fe50 2024-04-26T17:56:57.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97beffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0075f70 0x7f97c019c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97beffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0075f70 0x7f97c019c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39918/0 (socket says 172.21.15.31:39918) 2024-04-26T17:56:57.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97beffd700 1 -- 172.21.15.31:0/1278098733 learned_addr learned my addr 172.21.15.31:0/1278098733 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:57.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97bffff700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97c010fe50 0x7f97c01a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97c0075650 0x7f97c019bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97beffd700 1 -- 172.21.15.31:0/1278098733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97c010fe50 msgr2=0x7f97c01a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:57.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97beffd700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97c010fe50 0x7f97c01a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97beffd700 1 -- 172.21.15.31:0/1278098733 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97c0075650 msgr2=0x7f97c019bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:57.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97beffd700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97c0075650 0x7f97c019bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97beffd700 1 -- 172.21.15.31:0/1278098733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c01a0e00 con 0x7f97c0075f70 2024-04-26T17:56:57.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.751+0000 7f97bffff700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97c010fe50 0x7f97c01a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:56:57.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.752+0000 7f97beffd700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0075f70 0x7f97c019c330 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f97a80057c0 tx=0x7f97a8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:57.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.752+0000 7f97bcff9700 1 -- 172.21.15.31:0/1278098733 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a8026070 con 0x7f97c0075f70 2024-04-26T17:56:57.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.752+0000 7f97c5ffd700 1 -- 172.21.15.31:0/1278098733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c01a1090 con 0x7f97c0075f70 2024-04-26T17:56:57.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.752+0000 7f97bcff9700 1 -- 172.21.15.31:0/1278098733 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97a8021410 con 0x7f97c0075f70 2024-04-26T17:56:57.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.752+0000 7f97bcff9700 1 -- 172.21.15.31:0/1278098733 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a8017650 con 0x7f97c0075f70 2024-04-26T17:56:57.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.752+0000 7f97c5ffd700 1 -- 172.21.15.31:0/1278098733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c01a8970 con 0x7f97c0075f70 2024-04-26T17:56:57.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.754+0000 7f97bcff9700 1 -- 172.21.15.31:0/1278098733 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f97a8029070 con 0x7f97c0075f70 2024-04-26T17:56:57.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.755+0000 7f97bcff9700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f97ac0a3e40 0x7f97ac0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.755+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f97ac0a3e40 0x7f97ac0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.755+0000 7f97bcff9700 1 -- 172.21.15.31:0/1278098733 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f97a80d1740 con 0x7f97c0075f70 2024-04-26T17:56:57.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.755+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f97ac0a3e40 0x7f97ac0a6300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f97b8009c60 tx=0x7f97b8009260 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:57.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97d088fb80 1 Processor -- start 2024-04-26T17:56:57.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97d088fb80 1 -- start start 2024-04-26T17:56:57.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97d088fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6f4a20 0x55c6da6f4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97d088fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da6ef1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97d088fb80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6ef6f0 0x55c6da6fb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97d088fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c6da6417e0 con 0x55c6da6eed50 2024-04-26T17:56:57.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97d088fb80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55c6da5a78e0 con 0x55c6da6ef6f0 2024-04-26T17:56:57.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97d088fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c6da63d7a0 con 0x55c6da6f4a20 2024-04-26T17:56:57.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97beffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da6ef1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97beffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da6ef1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39930/0 (socket says 172.21.15.31:39930) 2024-04-26T17:56:57.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97beffd700 1 -- 172.21.15.31:0/1468238870 learned_addr learned my addr 172.21.15.31:0/1468238870 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:57.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97bffff700 1 --2- 172.21.15.31:0/1468238870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6ef6f0 0x55c6da6fb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/1468238870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6f4a20 0x55c6da6f4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97bffff700 1 -- 172.21.15.31:0/1468238870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6f4a20 msgr2=0x55c6da6f4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:57.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97bffff700 1 --2- 172.21.15.31:0/1468238870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6f4a20 0x55c6da6f4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97bffff700 1 -- 172.21.15.31:0/1468238870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 msgr2=0x55c6da6ef1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:57.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97bffff700 1 --2- 172.21.15.31:0/1468238870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da6ef1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97bffff700 1 -- 172.21.15.31:0/1468238870 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6da58e880 con 0x55c6da6ef6f0 2024-04-26T17:56:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/1468238870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6f4a20 0x55c6da6f4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:56:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.756+0000 7f97beffd700 1 --2- 172.21.15.31:0/1468238870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da6ef1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:56:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.757+0000 7f97bffff700 1 --2- 172.21.15.31:0/1468238870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6ef6f0 0x55c6da6fb770 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f97b0007600 tx=0x7f97b0007910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=842f0969476bc67b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.757+0000 7f97b5ffb700 1 -- 172.21.15.31:0/1468238870 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0019020 con 0x55c6da6ef6f0 2024-04-26T17:56:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.757+0000 7f97b5ffb700 1 -- 172.21.15.31:0/1468238870 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97b0015dc0 con 0x55c6da6ef6f0 2024-04-26T17:56:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.757+0000 7f97b5ffb700 1 -- 172.21.15.31:0/1468238870 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0014d40 con 0x55c6da6ef6f0 2024-04-26T17:56:57.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.757+0000 7f97d088fb80 1 -- 172.21.15.31:0/1468238870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6ef6f0 msgr2=0x55c6da6fb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.757+0000 7f97d088fb80 1 --2- 172.21.15.31:0/1468238870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6ef6f0 0x55c6da6fb770 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f97b0007600 tx=0x7f97b0007910 comp rx=0 tx=0).stop 2024-04-26T17:56:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.757+0000 7f97d088fb80 1 -- 172.21.15.31:0/1468238870 shutdown_connections 2024-04-26T17:56:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.757+0000 7f97d088fb80 1 --2- 172.21.15.31:0/1468238870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6ef6f0 0x55c6da6fb770 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.757+0000 7f97d088fb80 1 --2- 172.21.15.31:0/1468238870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6f4a20 0x55c6da6f4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.757+0000 7f97d088fb80 1 --2- 172.21.15.31:0/1468238870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da6ef1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.757+0000 7f97d088fb80 1 -- 172.21.15.31:0/1468238870 >> 172.21.15.31:0/1468238870 conn(0x55c6da61d570 msgr2=0x55c6da6e9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.758+0000 7f97d088fb80 1 -- 172.21.15.31:0/1468238870 shutdown_connections 2024-04-26T17:56:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.758+0000 7f97d088fb80 1 -- 172.21.15.31:0/1468238870 wait complete. 2024-04-26T17:56:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.758+0000 7f97d088fb80 1 -- start start 2024-04-26T17:56:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.758+0000 7f97d088fb80 1 Processor -- start 2024-04-26T17:56:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.758+0000 7f97d088fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6eed50 0x55c6da6fb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.758+0000 7f97d088fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6ef6f0 0x55c6da6f8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.758+0000 7f97d088fb80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6f4a20 0x55c6da6f8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.758+0000 7f97d088fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c6da6fbcb0 con 0x55c6da6ef6f0 2024-04-26T17:56:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.758+0000 7f97d088fb80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55c6da5fba30 con 0x55c6da6f4a20 2024-04-26T17:56:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.758+0000 7f97d088fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c6da6fbe30 con 0x55c6da6eed50 2024-04-26T17:56:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f97d088fb80 10 client.0 _ll_register_callbacks cb 0x55c6da5d9900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:56:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f97beffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6ef6f0 0x55c6da6f8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f97bffff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6f4a20 0x55c6da6f8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f97beffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6ef6f0 0x55c6da6f8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39936/0 (socket says 172.21.15.31:39936) 2024-04-26T17:56:57.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f97beffd700 1 -- 172.21.15.31:0/292882956 learned_addr learned my addr 172.21.15.31:0/292882956 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:57.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/292882956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6eed50 0x55c6da6fb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f97bffff700 1 -- 172.21.15.31:0/292882956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6eed50 msgr2=0x55c6da6fb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:57.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f97bffff700 1 --2- 172.21.15.31:0/292882956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6eed50 0x55c6da6fb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f97bffff700 1 -- 172.21.15.31:0/292882956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6ef6f0 msgr2=0x55c6da6f8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:57.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f97bffff700 1 --2- 172.21.15.31:0/292882956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6ef6f0 0x55c6da6f8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f97bffff700 1 -- 172.21.15.31:0/292882956 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6da5ff430 con 0x55c6da6f4a20 2024-04-26T17:56:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/292882956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6eed50 0x55c6da6fb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:56:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f97bffff700 1 --2- 172.21.15.31:0/292882956 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6f4a20 0x55c6da6f8cb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f97b0007600 tx=0x7f97b0014c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86421ff2a578a8db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f979effd700 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T17:56:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f979effd700 1 -- 172.21.15.31:0/292882956 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0019070 con 0x55c6da6f4a20 2024-04-26T17:56:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f979effd700 1 -- 172.21.15.31:0/292882956 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f97b0003a00 con 0x55c6da6f4a20 2024-04-26T17:56:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.759+0000 7f97d088fb80 1 -- 172.21.15.31:0/292882956 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97b0015dc0 con 0x55c6da6f4a20 2024-04-26T17:56:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.760+0000 7f979effd700 1 -- 172.21.15.31:0/292882956 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f97b002a020 con 0x55c6da6f4a20 2024-04-26T17:56:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.760+0000 7f97d088fb80 10 client.25025 fetch_fsmap learned FSMap version 27 2024-04-26T17:56:57.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.760+0000 7f97d088fb80 1 -- 172.21.15.31:0/292882956 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c6da706510 con 0x55c6da6f4a20 2024-04-26T17:56:57.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.760+0000 7f979effd700 1 -- 172.21.15.31:0/292882956 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97b0015dc0 con 0x55c6da6f4a20 2024-04-26T17:56:57.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.760+0000 7f979effd700 1 -- 172.21.15.31:0/292882956 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7f97b0004030 con 0x55c6da6f4a20 2024-04-26T17:56:57.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.760+0000 7f97d088fb80 10 client.25025 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T17:56:57.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.760+0000 7f97d088fb80 10 client.25025 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T17:56:57.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.760+0000 7f97d088fb80 20 client.25025 populate_metadata read hostname 'smithi031' 2024-04-26T17:56:57.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.760+0000 7f97d088fb80 1 --2- 172.21.15.31:0/292882956 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55c6da708790 0x55c6da709c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.760+0000 7f97d088fb80 4 client.25025 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:56:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.760+0000 7f97d088fb80 1 -- 172.21.15.31:0/292882956 --> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6da64f1c0 con 0x55c6da708790 2024-04-26T17:56:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.760+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/292882956 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55c6da708790 0x55c6da709c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.761+0000 7f979effd700 10 client.25025 ms_handle_connect on v2:172.21.15.97:6836/3334702590 2024-04-26T17:56:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.761+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/292882956 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55c6da708790 0x55c6da709c00 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=520342095fbd4df8 server_cookie=38ee6a21753a8fe1 in_seq=0 out_seq=0 2024-04-26T17:56:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.762+0000 7f979effd700 1 -- 172.21.15.31:0/292882956 <== mds.2 v2:172.21.15.97:6836/3334702590 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c6da64f1c0 con 0x55c6da708790 2024-04-26T17:56:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.762+0000 7f979effd700 10 client.25025 handle_command_reply: tid=0 2024-04-26T17:56:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.762+0000 7f97d088fb80 1 client.25025 shutdown 2024-04-26T17:56:57.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.762+0000 7f97d088fb80 1 -- 172.21.15.31:0/292882956 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6f4a20 msgr2=0x55c6da6f8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:57.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.762+0000 7f97d088fb80 1 --2- 172.21.15.31:0/292882956 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6f4a20 0x55c6da6f8cb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f97b0007600 tx=0x7f97b0014c10 comp rx=0 tx=0).stop 2024-04-26T17:56:57.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.763+0000 7f97d088fb80 1 -- 172.21.15.31:0/292882956 shutdown_connections 2024-04-26T17:56:57.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.763+0000 7f97d088fb80 1 --2- 172.21.15.31:0/292882956 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55c6da708790 0x55c6da709c00 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.763+0000 7f97d088fb80 1 --2- 172.21.15.31:0/292882956 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6f4a20 0x55c6da6f8cb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.763+0000 7f97d088fb80 1 --2- 172.21.15.31:0/292882956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6eed50 0x55c6da6fb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.763+0000 7f97d088fb80 1 --2- 172.21.15.31:0/292882956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6ef6f0 0x55c6da6f8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.763+0000 7f97d088fb80 1 -- 172.21.15.31:0/292882956 >> 172.21.15.31:0/292882956 conn(0x55c6da61d570 msgr2=0x55c6da6edd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:57.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.763+0000 7f979effd700 0 client.25025 ms_handle_reset on v2:172.21.15.97:6836/3334702590 2024-04-26T17:56:57.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.763+0000 7f97d088fb80 1 -- 172.21.15.31:0/292882956 shutdown_connections 2024-04-26T17:56:57.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.763+0000 7f97d088fb80 1 -- 172.21.15.31:0/292882956 wait complete. 2024-04-26T17:56:57.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.763+0000 7f97d088fb80 20 client.25025 trim_cache size 0 max 16384 2024-04-26T17:56:57.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.783+0000 7f97d088fb80 1 Processor -- start 2024-04-26T17:56:57.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.783+0000 7f97d088fb80 1 -- start start 2024-04-26T17:56:57.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.783+0000 7f97d088fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da6ef130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.783+0000 7f97d088fb80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6ef6f0 0x55c6da6e42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.783+0000 7f97d088fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6f4a20 0x55c6da6edd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.783+0000 7f97d088fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c6da707f40 con 0x55c6da6eed50 2024-04-26T17:56:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.783+0000 7f97d088fb80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55c6da708610 con 0x55c6da6ef6f0 2024-04-26T17:56:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.783+0000 7f97d088fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c6da708ce0 con 0x55c6da6f4a20 2024-04-26T17:56:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97bf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da6ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97beffd700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6ef6f0 0x55c6da6e42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97beffd700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6ef6f0 0x55c6da6e42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:57104/0 (socket says 172.21.15.31:57104) 2024-04-26T17:56:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97beffd700 1 -- 172.21.15.31:0/1670228008 learned_addr learned my addr 172.21.15.31:0/1670228008 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:57.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97bffff700 1 --2- 172.21.15.31:0/1670228008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6f4a20 0x55c6da6edd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97beffd700 1 -- 172.21.15.31:0/1670228008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6f4a20 msgr2=0x55c6da6edd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:57.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97beffd700 1 --2- 172.21.15.31:0/1670228008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6f4a20 0x55c6da6edd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97beffd700 1 -- 172.21.15.31:0/1670228008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 msgr2=0x55c6da6ef130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:57.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97beffd700 1 --2- 172.21.15.31:0/1670228008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da6ef130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97beffd700 1 -- 172.21.15.31:0/1670228008 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6da6ee960 con 0x55c6da6ef6f0 2024-04-26T17:56:57.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/1670228008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da6ef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:56:57.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97beffd700 1 --2- 172.21.15.31:0/1670228008 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6ef6f0 0x55c6da6e42b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f97a80cef00 tx=0x7f97a8088040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c04e112c895e121c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:57.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97b57fa700 1 -- 172.21.15.31:0/1670228008 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a809d420 con 0x55c6da6ef6f0 2024-04-26T17:56:57.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97b57fa700 1 -- 172.21.15.31:0/1670228008 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97a809dd50 con 0x55c6da6ef6f0 2024-04-26T17:56:57.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.784+0000 7f97b57fa700 1 -- 172.21.15.31:0/1670228008 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a809cfb0 con 0x55c6da6ef6f0 2024-04-26T17:56:57.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.785+0000 7f97d088fb80 1 -- 172.21.15.31:0/1670228008 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6ef6f0 msgr2=0x55c6da6e42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:57.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.785+0000 7f97d088fb80 1 --2- 172.21.15.31:0/1670228008 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6ef6f0 0x55c6da6e42b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f97a80cef00 tx=0x7f97a8088040 comp rx=0 tx=0).stop 2024-04-26T17:56:57.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.785+0000 7f97d088fb80 1 -- 172.21.15.31:0/1670228008 shutdown_connections 2024-04-26T17:56:57.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.785+0000 7f97d088fb80 1 --2- 172.21.15.31:0/1670228008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6f4a20 0x55c6da6edd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.785+0000 7f97d088fb80 1 --2- 172.21.15.31:0/1670228008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da6ef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.785+0000 7f97d088fb80 1 --2- 172.21.15.31:0/1670228008 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6ef6f0 0x55c6da6e42b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.785+0000 7f97d088fb80 1 -- 172.21.15.31:0/1670228008 >> 172.21.15.31:0/1670228008 conn(0x55c6da61d570 msgr2=0x55c6da6e97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:57.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.785+0000 7f97d088fb80 1 -- 172.21.15.31:0/1670228008 shutdown_connections 2024-04-26T17:56:57.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.785+0000 7f97d088fb80 1 -- 172.21.15.31:0/1670228008 wait complete. 2024-04-26T17:56:57.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.785+0000 7f97d088fb80 1 -- start start 2024-04-26T17:56:57.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97d088fb80 1 Processor -- start 2024-04-26T17:56:57.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97d088fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da70f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97d088fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6ef6f0 0x55c6da70f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97d088fb80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6f4a20 0x55c6da76e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97d088fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c6da709a80 con 0x55c6da6eed50 2024-04-26T17:56:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97d088fb80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55c6da7093b0 con 0x55c6da6f4a20 2024-04-26T17:56:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97d088fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c6da70a150 con 0x55c6da6ef6f0 2024-04-26T17:56:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97d088fb80 10 client.0 _ll_register_callbacks cb 0x55c6da5d9900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:56:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97bf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da70f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97bf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da70f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39940/0 (socket says 172.21.15.31:39940) 2024-04-26T17:56:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97bf7fe700 1 -- 172.21.15.31:0/1171525397 learned_addr learned my addr 172.21.15.31:0/1171525397 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97bffff700 1 --2- 172.21.15.31:0/1171525397 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6f4a20 0x55c6da76e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97beffd700 1 --2- 172.21.15.31:0/1171525397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6ef6f0 0x55c6da70f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97bf7fe700 1 -- 172.21.15.31:0/1171525397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6ef6f0 msgr2=0x55c6da70f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/1171525397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6ef6f0 0x55c6da70f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97bf7fe700 1 -- 172.21.15.31:0/1171525397 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6f4a20 msgr2=0x55c6da76e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:57.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/1171525397 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6f4a20 0x55c6da76e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.786+0000 7f97bf7fe700 1 -- 172.21.15.31:0/1171525397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6da76ee60 con 0x55c6da6eed50 2024-04-26T17:56:57.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.787+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/1171525397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da70f250 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x55c6da70a7d0 tx=0x7f97b80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959ae65582a6ba37 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:57.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.787+0000 7f979effd700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:56:57.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.787+0000 7f979effd700 1 -- 172.21.15.31:0/1171525397 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b801e370 con 0x55c6da6eed50 2024-04-26T17:56:57.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.787+0000 7f979effd700 1 -- 172.21.15.31:0/1171525397 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f97b8029590 con 0x55c6da6eed50 2024-04-26T17:56:57.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.787+0000 7f97d088fb80 1 -- 172.21.15.31:0/1171525397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97a809dd50 con 0x55c6da6eed50 2024-04-26T17:56:57.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.787+0000 7f979effd700 1 -- 172.21.15.31:0/1171525397 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f97b8035020 con 0x55c6da6eed50 2024-04-26T17:56:57.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.787+0000 7f97d088fb80 10 client.15306 fetch_fsmap learned FSMap version 27 2024-04-26T17:56:57.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.787+0000 7f97d088fb80 1 -- 172.21.15.31:0/1171525397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c6da7701c0 con 0x55c6da6eed50 2024-04-26T17:56:57.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.787+0000 7f979effd700 1 -- 172.21.15.31:0/1171525397 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97a809dd50 con 0x55c6da6eed50 2024-04-26T17:56:57.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.787+0000 7f979effd700 1 -- 172.21.15.31:0/1171525397 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7f97b8003680 con 0x55c6da6eed50 2024-04-26T17:56:57.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.787+0000 7f97d088fb80 10 client.15306 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T17:56:57.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.787+0000 7f97d088fb80 10 client.15306 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T17:56:57.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.787+0000 7f97d088fb80 20 client.15306 populate_metadata read hostname 'smithi031' 2024-04-26T17:56:57.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.788+0000 7f97d088fb80 1 --2- 172.21.15.31:0/1171525397 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55c6da774e80 0x55c6da773b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:57.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.788+0000 7f97d088fb80 4 client.15306 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T17:56:57.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.788+0000 7f97d088fb80 1 -- 172.21.15.31:0/1171525397 --> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] -- command(tid 0: {"prefix": "status"}) -- 0x55c6da649260 con 0x55c6da774e80 2024-04-26T17:56:57.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.788+0000 7f97beffd700 1 --2- 172.21.15.31:0/1171525397 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55c6da774e80 0x55c6da773b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:57.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.788+0000 7f979effd700 10 client.15306 ms_handle_connect on v2:172.21.15.97:6836/3334702590 2024-04-26T17:56:57.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.788+0000 7f97beffd700 1 --2- 172.21.15.31:0/1171525397 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55c6da774e80 0x55c6da773b60 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=72e5f5c5e33f1b56 server_cookie=2b15269446db9245 in_seq=0 out_seq=0 2024-04-26T17:56:57.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.789+0000 7f979effd700 1 -- 172.21.15.31:0/1171525397 <== mds.2 v2:172.21.15.97:6836/3334702590 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55c6da649260 con 0x55c6da774e80 2024-04-26T17:56:57.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.789+0000 7f979effd700 10 client.15306 handle_command_reply: tid=0 2024-04-26T17:56:57.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.789+0000 7f97d088fb80 1 client.15306 shutdown 2024-04-26T17:56:57.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.789+0000 7f97d088fb80 1 -- 172.21.15.31:0/1171525397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 msgr2=0x55c6da70f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:57.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.789+0000 7f97d088fb80 1 --2- 172.21.15.31:0/1171525397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da70f250 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x55c6da70a7d0 tx=0x7f97b80047c0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.789+0000 7f97d088fb80 1 -- 172.21.15.31:0/1171525397 shutdown_connections 2024-04-26T17:56:57.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.789+0000 7f97d088fb80 1 --2- 172.21.15.31:0/1171525397 >> [v2:172.21.15.97:6836/3334702590,v1:172.21.15.97:6837/3334702590] conn(0x55c6da774e80 0x55c6da773b60 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.789+0000 7f97d088fb80 1 --2- 172.21.15.31:0/1171525397 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c6da6f4a20 0x55c6da76e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.789+0000 7f97d088fb80 1 --2- 172.21.15.31:0/1171525397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6da6eed50 0x55c6da70f250 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.789+0000 7f979effd700 0 client.15306 ms_handle_reset on v2:172.21.15.97:6836/3334702590 2024-04-26T17:56:57.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.789+0000 7f97d088fb80 1 --2- 172.21.15.31:0/1171525397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6da6ef6f0 0x55c6da70f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.789+0000 7f97d088fb80 1 -- 172.21.15.31:0/1171525397 >> 172.21.15.31:0/1171525397 conn(0x55c6da61d570 msgr2=0x55c6da6edd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:57.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.789+0000 7f97d088fb80 1 -- 172.21.15.31:0/1171525397 shutdown_connections 2024-04-26T17:56:57.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.789+0000 7f97d088fb80 1 -- 172.21.15.31:0/1171525397 wait complete. 2024-04-26T17:56:57.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.790+0000 7f97d088fb80 20 client.15306 trim_cache size 0 max 16384 2024-04-26T17:56:57.805 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T17:56:57.805 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "7e53e050-03f4-11ef-bc93-c7b262605968", 2024-04-26T17:56:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 2, 2024-04-26T17:56:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24446, 2024-04-26T17:56:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-26T17:56:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-26T17:56:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-26T17:56:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 218.11223463300001, 2024-04-26T17:56:57.807 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 27, 2024-04-26T17:56:57.807 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 91, 2024-04-26T17:56:57.807 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T17:56:57.807 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 260.95509546300002 2024-04-26T17:56:57.807 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T17:56:57.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.790+0000 7f97c5ffd700 1 -- 172.21.15.31:0/1278098733 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f97ac0a3e40 msgr2=0x7f97ac0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:57.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.790+0000 7f97c5ffd700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f97ac0a3e40 0x7f97ac0a6300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f97b8009c60 tx=0x7f97b8009260 comp rx=0 tx=0).stop 2024-04-26T17:56:57.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.790+0000 7f97c5ffd700 1 -- 172.21.15.31:0/1278098733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0075f70 msgr2=0x7f97c019c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:57.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.790+0000 7f97c5ffd700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0075f70 0x7f97c019c330 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f97a80057c0 tx=0x7f97a8009580 comp rx=0 tx=0).stop 2024-04-26T17:56:57.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.790+0000 7f97c5ffd700 1 -- 172.21.15.31:0/1278098733 shutdown_connections 2024-04-26T17:56:57.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.790+0000 7f97c5ffd700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f97ac0a3e40 0x7f97ac0a6300 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.790+0000 7f97c5ffd700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97c0075650 0x7f97c019bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.790+0000 7f97c5ffd700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97c010fe50 0x7f97c01a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.790+0000 7f97c5ffd700 1 --2- 172.21.15.31:0/1278098733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0075f70 0x7f97c019c330 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:57.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.790+0000 7f97c5ffd700 1 -- 172.21.15.31:0/1278098733 >> 172.21.15.31:0/1278098733 conn(0x7f97c007a5d0 msgr2=0x7f97c01019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:57.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.790+0000 7f97c5ffd700 1 -- 172.21.15.31:0/1278098733 shutdown_connections 2024-04-26T17:56:57.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:56:57.790+0000 7f97c5ffd700 1 -- 172.21.15.31:0/1278098733 wait complete. 2024-04-26T17:56:58.468 INFO:teuthology.orchestra.run.smithi031.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2024-04-26T17:56:58.472 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-26T17:56:58.486 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T17:56:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.499+0000 7f3366966640 1 -- 172.21.15.31:0/4092307046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f336006bb20 msgr2=0x7f336010f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.499+0000 7f3366966640 1 --2- 172.21.15.31:0/4092307046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f336006bb20 0x7f336010f7d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3358009ca0 tx=0x7f335802a950 comp rx=0 tx=0).stop 2024-04-26T17:56:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.499+0000 7f3366966640 1 -- 172.21.15.31:0/4092307046 shutdown_connections 2024-04-26T17:56:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.499+0000 7f3366966640 1 --2- 172.21.15.31:0/4092307046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f336006bb20 0x7f336010f7d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.499+0000 7f3366966640 1 --2- 172.21.15.31:0/4092307046 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f336006b160 0x7f336006b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.499+0000 7f3366966640 1 --2- 172.21.15.31:0/4092307046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3360108b30 0x7f3360108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.499+0000 7f3366966640 1 -- 172.21.15.31:0/4092307046 >> 172.21.15.31:0/4092307046 conn(0x7f3360077630 msgr2=0x7f3360077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.499+0000 7f3366966640 1 -- 172.21.15.31:0/4092307046 shutdown_connections 2024-04-26T17:56:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.499+0000 7f3366966640 1 -- 172.21.15.31:0/4092307046 wait complete. 2024-04-26T17:56:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.499+0000 7f3366966640 1 Processor -- start 2024-04-26T17:56:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f3366966640 1 -- start start 2024-04-26T17:56:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f3366966640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f336006b160 0x7f3360198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f3366966640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f336006bb20 0x7f33601994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f3366966640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3360108b30 0x7f336019d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f3366966640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3360112c00 con 0x7f336006b160 2024-04-26T17:56:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f3366966640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3360112a80 con 0x7f336006bb20 2024-04-26T17:56:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f3366966640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3360112d80 con 0x7f3360108b30 2024-04-26T17:56:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f335ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f336006b160 0x7f3360198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f335ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f336006b160 0x7f3360198f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39956/0 (socket says 172.21.15.31:39956) 2024-04-26T17:56:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f335ffff640 1 -- 172.21.15.31:0/3751694091 learned_addr learned my addr 172.21.15.31:0/3751694091 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f335f7fe640 1 --2- 172.21.15.31:0/3751694091 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f336006bb20 0x7f33601994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f3364f03640 1 --2- 172.21.15.31:0/3751694091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3360108b30 0x7f336019d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f335ffff640 1 -- 172.21.15.31:0/3751694091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3360108b30 msgr2=0x7f336019d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f335ffff640 1 --2- 172.21.15.31:0/3751694091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3360108b30 0x7f336019d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f335ffff640 1 -- 172.21.15.31:0/3751694091 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f336006bb20 msgr2=0x7f33601994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f335ffff640 1 --2- 172.21.15.31:0/3751694091 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f336006bb20 0x7f33601994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.500+0000 7f335ffff640 1 -- 172.21.15.31:0/3751694091 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f336019df70 con 0x7f336006b160 2024-04-26T17:56:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.501+0000 7f335ffff640 1 --2- 172.21.15.31:0/3751694091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f336006b160 0x7f3360198f80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f335000d950 tx=0x7f335000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.501+0000 7f335d7fa640 1 -- 172.21.15.31:0/3751694091 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3350017030 con 0x7f336006b160 2024-04-26T17:56:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.501+0000 7f335d7fa640 1 -- 172.21.15.31:0/3751694091 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3350004020 con 0x7f336006b160 2024-04-26T17:56:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.501+0000 7f335d7fa640 1 -- 172.21.15.31:0/3751694091 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33500047a0 con 0x7f336006b160 2024-04-26T17:56:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.501+0000 7f3366966640 1 -- 172.21.15.31:0/3751694091 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f336019e260 con 0x7f336006b160 2024-04-26T17:56:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.501+0000 7f3366966640 1 -- 172.21.15.31:0/3751694091 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33601a5af0 con 0x7f336006b160 2024-04-26T17:56:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.503+0000 7f335d7fa640 1 -- 172.21.15.31:0/3751694091 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f3350009a30 con 0x7f336006b160 2024-04-26T17:56:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.504+0000 7f335d7fa640 1 --2- 172.21.15.31:0/3751694091 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f334c0a3ee0 0x7f334c0a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.504+0000 7f335d7fa640 1 -- 172.21.15.31:0/3751694091 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f33500d0340 con 0x7f336006b160 2024-04-26T17:56:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.504+0000 7f335f7fe640 1 --2- 172.21.15.31:0/3751694091 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f334c0a3ee0 0x7f334c0a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.504+0000 7f336967f740 1 Processor -- start 2024-04-26T17:56:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f336967f740 1 -- start start 2024-04-26T17:56:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f336967f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add1e00 0x55ec0add21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f335f7fe640 1 --2- 172.21.15.31:0/3751694091 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f334c0a3ee0 0x7f334c0a63a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3360107f50 tx=0x7f334800b390 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f336967f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 0x55ec0ade0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f336967f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0ade14f0 0x55ec0ade38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f336967f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ec0ad38460 con 0x55ec0ade14f0 2024-04-26T17:56:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f336967f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ec0acf2590 con 0x55ec0add1e00 2024-04-26T17:56:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f336967f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ec0ad34420 con 0x55ec0add2720 2024-04-26T17:56:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f335ffff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add1e00 0x55ec0add21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f335ffff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add1e00 0x55ec0add21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:57148/0 (socket says 172.21.15.31:57148) 2024-04-26T17:56:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f335ffff640 1 -- 172.21.15.31:0/1417243955 learned_addr learned my addr 172.21.15.31:0/1417243955 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f3364f03640 1 --2- 172.21.15.31:0/1417243955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0ade14f0 0x55ec0ade38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f335f7fe640 1 --2- 172.21.15.31:0/1417243955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 0x55ec0ade0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f335ffff640 1 -- 172.21.15.31:0/1417243955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 msgr2=0x55ec0ade0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f335ffff640 1 --2- 172.21.15.31:0/1417243955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 0x55ec0ade0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f335ffff640 1 -- 172.21.15.31:0/1417243955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0ade14f0 msgr2=0x55ec0ade38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f335ffff640 1 --2- 172.21.15.31:0/1417243955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0ade14f0 0x55ec0ade38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.505+0000 7f335ffff640 1 -- 172.21.15.31:0/1417243955 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec0acff0d0 con 0x55ec0add1e00 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.506+0000 7f335ffff640 1 --2- 172.21.15.31:0/1417243955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add1e00 0x55ec0add21e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f335009bec0 tx=0x7f33500056a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a81226b75f118f7c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.506+0000 7f33567fc640 1 -- 172.21.15.31:0/1417243955 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33500b5070 con 0x55ec0add1e00 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.506+0000 7f33567fc640 1 -- 172.21.15.31:0/1417243955 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f335009c1c0 con 0x55ec0add1e00 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.506+0000 7f33567fc640 1 -- 172.21.15.31:0/1417243955 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3350014070 con 0x55ec0add1e00 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.507+0000 7f336967f740 1 -- 172.21.15.31:0/1417243955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add1e00 msgr2=0x55ec0add21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.507+0000 7f336967f740 1 --2- 172.21.15.31:0/1417243955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add1e00 0x55ec0add21e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f335009bec0 tx=0x7f33500056a0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.507+0000 7f336967f740 1 -- 172.21.15.31:0/1417243955 shutdown_connections 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.507+0000 7f336967f740 1 --2- 172.21.15.31:0/1417243955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0ade14f0 0x55ec0ade38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.507+0000 7f336967f740 1 --2- 172.21.15.31:0/1417243955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 0x55ec0ade0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.507+0000 7f336967f740 1 --2- 172.21.15.31:0/1417243955 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add1e00 0x55ec0add21e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.507+0000 7f336967f740 1 -- 172.21.15.31:0/1417243955 >> 172.21.15.31:0/1417243955 conn(0x55ec0adce410 msgr2=0x55ec0adce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.507+0000 7f336967f740 1 -- 172.21.15.31:0/1417243955 shutdown_connections 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.507+0000 7f336967f740 1 -- 172.21.15.31:0/1417243955 wait complete. 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.507+0000 7f336967f740 1 -- start start 2024-04-26T17:56:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.507+0000 7f336967f740 1 Processor -- start 2024-04-26T17:56:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f336967f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0add1e00 0x55ec0adde460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f336967f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 0x55ec0adde9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f336967f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0ade14f0 0x55ec0addeee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f336967f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ec0ade4120 con 0x55ec0add1e00 2024-04-26T17:56:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f336967f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ec0acbfb60 con 0x55ec0ade14f0 2024-04-26T17:56:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f336967f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ec0ade42a0 con 0x55ec0add2720 2024-04-26T17:56:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f335ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0add1e00 0x55ec0adde460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f335ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0add1e00 0x55ec0adde460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39964/0 (socket says 172.21.15.31:39964) 2024-04-26T17:56:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f335ffff640 1 -- 172.21.15.31:0/2763061607 learned_addr learned my addr 172.21.15.31:0/2763061607 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f336967f740 10 client.0 _ll_register_callbacks cb 0x55ec0ace42b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:56:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f335f7fe640 1 --2- 172.21.15.31:0/2763061607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 0x55ec0adde9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f3364f03640 1 --2- 172.21.15.31:0/2763061607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0ade14f0 0x55ec0addeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f335ffff640 1 -- 172.21.15.31:0/2763061607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 msgr2=0x55ec0adde9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f335ffff640 1 --2- 172.21.15.31:0/2763061607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 0x55ec0adde9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f335ffff640 1 -- 172.21.15.31:0/2763061607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0ade14f0 msgr2=0x55ec0addeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f335ffff640 1 --2- 172.21.15.31:0/2763061607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0ade14f0 0x55ec0addeee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f335ffff640 1 -- 172.21.15.31:0/2763061607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec0aa4a210 con 0x55ec0add1e00 2024-04-26T17:56:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f335ffff640 1 --2- 172.21.15.31:0/2763061607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0add1e00 0x55ec0adde460 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f33500ce0e0 tx=0x7f33500cda20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff4cf2622c60e11 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.508+0000 7f333f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:56:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.509+0000 7f333f7fe640 1 -- 172.21.15.31:0/2763061607 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33500b9070 con 0x55ec0add1e00 2024-04-26T17:56:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.509+0000 7f333f7fe640 1 -- 172.21.15.31:0/2763061607 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f3350098830 con 0x55ec0add1e00 2024-04-26T17:56:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.509+0000 7f336967f740 1 -- 172.21.15.31:0/2763061607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f335009c1c0 con 0x55ec0add1e00 2024-04-26T17:56:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.509+0000 7f333f7fe640 1 -- 172.21.15.31:0/2763061607 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3350014040 con 0x55ec0add1e00 2024-04-26T17:56:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.509+0000 7f336967f740 10 client.15318 fetch_fsmap learned FSMap version 27 2024-04-26T17:56:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.509+0000 7f336967f740 1 -- 172.21.15.31:0/2763061607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ec0ab16620 con 0x55ec0add1e00 2024-04-26T17:56:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.509+0000 7f333f7fe640 1 -- 172.21.15.31:0/2763061607 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f335009c1c0 con 0x55ec0add1e00 2024-04-26T17:56:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.509+0000 7f333f7fe640 1 -- 172.21.15.31:0/2763061607 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7f3350097f10 con 0x55ec0add1e00 2024-04-26T17:56:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.509+0000 7f336967f740 10 client.15318 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T17:56:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.509+0000 7f336967f740 10 client.15318 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:56:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.509+0000 7f336967f740 20 client.15318 populate_metadata read hostname 'smithi031' 2024-04-26T17:56:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.509+0000 7f336967f740 1 --2- 172.21.15.31:0/2763061607 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec0adf1560 0x55ec0adf3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.509+0000 7f336967f740 4 client.15318 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:56:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.509+0000 7f336967f740 1 -- 172.21.15.31:0/2763061607 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec0ad46070 con 0x55ec0adf1560 2024-04-26T17:56:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.510+0000 7f335f7fe640 1 --2- 172.21.15.31:0/2763061607 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec0adf1560 0x55ec0adf3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.510+0000 7f333f7fe640 10 client.15318 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:56:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.510+0000 7f335f7fe640 1 --2- 172.21.15.31:0/2763061607 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec0adf1560 0x55ec0adf3950 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b792fc50479cea43 server_cookie=e4495b0886cd6642 in_seq=0 out_seq=0 2024-04-26T17:56:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f333f7fe640 1 -- 172.21.15.31:0/2763061607 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ec0ad46070 con 0x55ec0adf1560 2024-04-26T17:56:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f333f7fe640 10 client.15318 handle_command_reply: tid=0 2024-04-26T17:56:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f336967f740 1 client.15318 shutdown 2024-04-26T17:56:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f336967f740 1 -- 172.21.15.31:0/2763061607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0add1e00 msgr2=0x55ec0adde460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f336967f740 1 --2- 172.21.15.31:0/2763061607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0add1e00 0x55ec0adde460 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f33500ce0e0 tx=0x7f33500cda20 comp rx=0 tx=0).stop 2024-04-26T17:56:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f336967f740 1 -- 172.21.15.31:0/2763061607 shutdown_connections 2024-04-26T17:56:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f336967f740 1 --2- 172.21.15.31:0/2763061607 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec0adf1560 0x55ec0adf3950 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f336967f740 1 --2- 172.21.15.31:0/2763061607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0ade14f0 0x55ec0addeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f336967f740 1 --2- 172.21.15.31:0/2763061607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 0x55ec0adde9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f333f7fe640 0 client.15318 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:56:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f336967f740 1 --2- 172.21.15.31:0/2763061607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0add1e00 0x55ec0adde460 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f336967f740 1 -- 172.21.15.31:0/2763061607 >> 172.21.15.31:0/2763061607 conn(0x55ec0adce410 msgr2=0x55ec0add4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f336967f740 1 -- 172.21.15.31:0/2763061607 shutdown_connections 2024-04-26T17:56:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f336967f740 1 -- 172.21.15.31:0/2763061607 wait complete. 2024-04-26T17:56:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.511+0000 7f336967f740 20 client.15318 trim_cache size 0 max 16384 2024-04-26T17:56:58.518 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/: 2024-04-26T17:56:58.518 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.518 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 . 2024-04-26T17:56:58.518 INFO:teuthology.orchestra.run.smithi031.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 .. 2024-04-26T17:56:58.518 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 data 2024-04-26T17:56:58.519 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.519 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data: 2024-04-26T17:56:58.519 INFO:teuthology.orchestra.run.smithi031.stdout:total 35 2024-04-26T17:56:58.519 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 . 2024-04-26T17:56:58.519 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 .. 2024-04-26T17:56:58.519 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 base 2024-04-26T17:56:58.519 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 26 17:56 global 2024-04-26T17:56:58.519 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 log 2024-04-26T17:56:58.519 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_commit_ts 2024-04-26T17:56:58.519 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_dynshmem 2024-04-26T17:56:58.519 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 26 17:56 pg_hba.conf 2024-04-26T17:56:58.519 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 26 17:56 pg_ident.conf 2024-04-26T17:56:58.519 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 pg_logical 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 17:56 pg_multixact 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_notify 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_replslot 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_serial 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_snapshots 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_stat 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_stat_tmp 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 pg_subtrans 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_tblspc 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_twophase 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 PG_VERSION 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 17:56 pg_wal 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 pg_xact 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 26 17:56 postgresql.auto.conf 2024-04-26T17:56:58.520 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 26 17:56 postgresql.conf 2024-04-26T17:56:58.521 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.521 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/base: 2024-04-26T17:56:58.521 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.521 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 . 2024-04-26T17:56:58.521 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.521 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 17:56 1 2024-04-26T17:56:58.521 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 17:56 13493 2024-04-26T17:56:58.521 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 17:56 13494 2024-04-26T17:56:58.521 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.521 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-26T17:56:58.521 INFO:teuthology.orchestra.run.smithi031.stdout:total 7777 2024-04-26T17:56:58.522 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 17:56 . 2024-04-26T17:56:58.522 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 .. 2024-04-26T17:56:58.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 112 2024-04-26T17:56:58.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 113 2024-04-26T17:56:58.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 17:56 1247 2024-04-26T17:56:58.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1247_fsm 2024-04-26T17:56:58.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1247_vm 2024-04-26T17:56:58.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 17:56 1249 2024-04-26T17:56:58.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1249_fsm 2024-04-26T17:56:58.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1249_vm 2024-04-26T17:56:58.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 17:56 1255 2024-04-26T17:56:58.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1255_fsm 2024-04-26T17:56:58.522 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1255_vm 2024-04-26T17:56:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 17:56 1259 2024-04-26T17:56:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1259_fsm 2024-04-26T17:56:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1259_vm 2024-04-26T17:56:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 17:56 13344 2024-04-26T17:56:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13344_fsm 2024-04-26T17:56:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13344_vm 2024-04-26T17:56:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13346 2024-04-26T17:56:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13348 2024-04-26T17:56:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13349 2024-04-26T17:56:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13349_fsm 2024-04-26T17:56:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13349_vm 2024-04-26T17:56:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13351 2024-04-26T17:56:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13353 2024-04-26T17:56:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13354 2024-04-26T17:56:58.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13354_fsm 2024-04-26T17:56:58.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13354_vm 2024-04-26T17:56:58.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13356 2024-04-26T17:56:58.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13358 2024-04-26T17:56:58.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13359 2024-04-26T17:56:58.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13359_fsm 2024-04-26T17:56:58.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13359_vm 2024-04-26T17:56:58.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13361 2024-04-26T17:56:58.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13363 2024-04-26T17:56:58.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 1417 2024-04-26T17:56:58.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 1418 2024-04-26T17:56:58.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 174 2024-04-26T17:56:58.524 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 175 2024-04-26T17:56:58.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2187 2024-04-26T17:56:58.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2224 2024-04-26T17:56:58.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2328 2024-04-26T17:56:58.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2336 2024-04-26T17:56:58.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2337 2024-04-26T17:56:58.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2579 2024-04-26T17:56:58.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2600 2024-04-26T17:56:58.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2600_fsm 2024-04-26T17:56:58.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2600_vm 2024-04-26T17:56:58.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2601 2024-04-26T17:56:58.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2601_fsm 2024-04-26T17:56:58.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2601_vm 2024-04-26T17:56:58.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2602 2024-04-26T17:56:58.525 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2602_fsm 2024-04-26T17:56:58.526 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2602_vm 2024-04-26T17:56:58.526 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2603 2024-04-26T17:56:58.526 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2603_fsm 2024-04-26T17:56:58.526 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2603_vm 2024-04-26T17:56:58.526 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2604 2024-04-26T17:56:58.526 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2605 2024-04-26T17:56:58.526 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2605_fsm 2024-04-26T17:56:58.526 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2605_vm 2024-04-26T17:56:58.527 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2606 2024-04-26T17:56:58.527 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2606_fsm 2024-04-26T17:56:58.527 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2606_vm 2024-04-26T17:56:58.527 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2607 2024-04-26T17:56:58.527 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2607_fsm 2024-04-26T17:56:58.527 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2607_vm 2024-04-26T17:56:58.527 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 17:56 2608 2024-04-26T17:56:58.528 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2608_fsm 2024-04-26T17:56:58.528 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2608_vm 2024-04-26T17:56:58.528 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 17:56 2609 2024-04-26T17:56:58.528 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2609_fsm 2024-04-26T17:56:58.528 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2609_vm 2024-04-26T17:56:58.528 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2610 2024-04-26T17:56:58.528 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2610_fsm 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2610_vm 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2611 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2612 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2612_fsm 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2612_vm 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2613 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2615 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2615_fsm 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2615_vm 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2616 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2616_fsm 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2616_vm 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 17:56 2617 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2617_fsm 2024-04-26T17:56:58.529 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2617_vm 2024-04-26T17:56:58.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 17:56 2618 2024-04-26T17:56:58.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2618_fsm 2024-04-26T17:56:58.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2618_vm 2024-04-26T17:56:58.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 26 17:56 2619 2024-04-26T17:56:58.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2619_fsm 2024-04-26T17:56:58.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2619_vm 2024-04-26T17:56:58.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2620 2024-04-26T17:56:58.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2650 2024-04-26T17:56:58.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2651 2024-04-26T17:56:58.530 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2652 2024-04-26T17:56:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2653 2024-04-26T17:56:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2654 2024-04-26T17:56:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2655 2024-04-26T17:56:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2656 2024-04-26T17:56:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2657 2024-04-26T17:56:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 17:56 2658 2024-04-26T17:56:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 17:56 2659 2024-04-26T17:56:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2660 2024-04-26T17:56:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2661 2024-04-26T17:56:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2662 2024-04-26T17:56:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2663 2024-04-26T17:56:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2664 2024-04-26T17:56:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2665 2024-04-26T17:56:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2666 2024-04-26T17:56:58.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2667 2024-04-26T17:56:58.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2668 2024-04-26T17:56:58.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2669 2024-04-26T17:56:58.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2670 2024-04-26T17:56:58.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 17:56 2673 2024-04-26T17:56:58.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 17:56 2674 2024-04-26T17:56:58.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 17:56 2675 2024-04-26T17:56:58.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2678 2024-04-26T17:56:58.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2679 2024-04-26T17:56:58.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2680 2024-04-26T17:56:58.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2681 2024-04-26T17:56:58.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2682 2024-04-26T17:56:58.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2683 2024-04-26T17:56:58.532 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2684 2024-04-26T17:56:58.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2685 2024-04-26T17:56:58.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2686 2024-04-26T17:56:58.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2687 2024-04-26T17:56:58.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2688 2024-04-26T17:56:58.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2689 2024-04-26T17:56:58.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 17:56 2690 2024-04-26T17:56:58.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 17:56 2691 2024-04-26T17:56:58.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2692 2024-04-26T17:56:58.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2693 2024-04-26T17:56:58.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2696 2024-04-26T17:56:58.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2699 2024-04-26T17:56:58.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2701 2024-04-26T17:56:58.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2702 2024-04-26T17:56:58.533 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2703 2024-04-26T17:56:58.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2704 2024-04-26T17:56:58.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2753 2024-04-26T17:56:58.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2753_fsm 2024-04-26T17:56:58.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2753_vm 2024-04-26T17:56:58.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2754 2024-04-26T17:56:58.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2755 2024-04-26T17:56:58.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2756 2024-04-26T17:56:58.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2757 2024-04-26T17:56:58.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2830 2024-04-26T17:56:58.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2831 2024-04-26T17:56:58.534 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2832 2024-04-26T17:56:58.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2833 2024-04-26T17:56:58.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2834 2024-04-26T17:56:58.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2835 2024-04-26T17:56:58.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2836 2024-04-26T17:56:58.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2837 2024-04-26T17:56:58.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 17:56 2838 2024-04-26T17:56:58.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2838_fsm 2024-04-26T17:56:58.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2838_vm 2024-04-26T17:56:58.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2839 2024-04-26T17:56:58.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2840 2024-04-26T17:56:58.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2840_fsm 2024-04-26T17:56:58.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2840_vm 2024-04-26T17:56:58.535 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2841 2024-04-26T17:56:58.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2995 2024-04-26T17:56:58.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2996 2024-04-26T17:56:58.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3079 2024-04-26T17:56:58.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3079_fsm 2024-04-26T17:56:58.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3079_vm 2024-04-26T17:56:58.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3080 2024-04-26T17:56:58.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3081 2024-04-26T17:56:58.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 3085 2024-04-26T17:56:58.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3118 2024-04-26T17:56:58.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3119 2024-04-26T17:56:58.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 17:56 3164 2024-04-26T17:56:58.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3256 2024-04-26T17:56:58.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3257 2024-04-26T17:56:58.536 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3258 2024-04-26T17:56:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3350 2024-04-26T17:56:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3351 2024-04-26T17:56:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3379 2024-04-26T17:56:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3380 2024-04-26T17:56:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3381 2024-04-26T17:56:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3394 2024-04-26T17:56:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3394_fsm 2024-04-26T17:56:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3394_vm 2024-04-26T17:56:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3395 2024-04-26T17:56:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3429 2024-04-26T17:56:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3430 2024-04-26T17:56:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3431 2024-04-26T17:56:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3433 2024-04-26T17:56:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3439 2024-04-26T17:56:58.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3440 2024-04-26T17:56:58.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 3455 2024-04-26T17:56:58.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 17:56 3456 2024-04-26T17:56:58.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3456_fsm 2024-04-26T17:56:58.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3456_vm 2024-04-26T17:56:58.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3466 2024-04-26T17:56:58.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3467 2024-04-26T17:56:58.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3468 2024-04-26T17:56:58.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3501 2024-04-26T17:56:58.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3502 2024-04-26T17:56:58.538 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3503 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3534 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3541 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3541_fsm 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3541_vm 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3542 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3574 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3575 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3576 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3596 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3597 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3598 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3599 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3600 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3600_fsm 2024-04-26T17:56:58.539 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3600_vm 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3601 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3601_fsm 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3601_vm 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3602 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3602_fsm 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3602_vm 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3603 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3603_fsm 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3603_vm 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3604 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3605 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3606 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3607 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3608 2024-04-26T17:56:58.540 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 3609 2024-04-26T17:56:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3712 2024-04-26T17:56:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3764 2024-04-26T17:56:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3764_fsm 2024-04-26T17:56:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3764_vm 2024-04-26T17:56:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3766 2024-04-26T17:56:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3767 2024-04-26T17:56:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3997 2024-04-26T17:56:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4143 2024-04-26T17:56:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4144 2024-04-26T17:56:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4145 2024-04-26T17:56:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4146 2024-04-26T17:56:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4147 2024-04-26T17:56:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4148 2024-04-26T17:56:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4149 2024-04-26T17:56:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4150 2024-04-26T17:56:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4151 2024-04-26T17:56:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4152 2024-04-26T17:56:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4153 2024-04-26T17:56:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4154 2024-04-26T17:56:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4155 2024-04-26T17:56:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4156 2024-04-26T17:56:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4157 2024-04-26T17:56:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4158 2024-04-26T17:56:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4159 2024-04-26T17:56:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4160 2024-04-26T17:56:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4161 2024-04-26T17:56:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4162 2024-04-26T17:56:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4163 2024-04-26T17:56:58.543 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4164 2024-04-26T17:56:58.543 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4165 2024-04-26T17:56:58.545 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4166 2024-04-26T17:56:58.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4167 2024-04-26T17:56:58.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4168 2024-04-26T17:56:58.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4169 2024-04-26T17:56:58.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4170 2024-04-26T17:56:58.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4171 2024-04-26T17:56:58.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4172 2024-04-26T17:56:58.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4173 2024-04-26T17:56:58.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4174 2024-04-26T17:56:58.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 5002 2024-04-26T17:56:58.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 548 2024-04-26T17:56:58.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 549 2024-04-26T17:56:58.546 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6102 2024-04-26T17:56:58.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6104 2024-04-26T17:56:58.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6106 2024-04-26T17:56:58.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6110 2024-04-26T17:56:58.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6111 2024-04-26T17:56:58.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6112 2024-04-26T17:56:58.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6113 2024-04-26T17:56:58.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6117 2024-04-26T17:56:58.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 826 2024-04-26T17:56:58.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 827 2024-04-26T17:56:58.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 828 2024-04-26T17:56:58.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 17:56 pg_filenode.map 2024-04-26T17:56:58.547 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 PG_VERSION 2024-04-26T17:56:58.548 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.548 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-26T17:56:58.548 INFO:teuthology.orchestra.run.smithi031.stdout:total 7777 2024-04-26T17:56:58.548 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 17:56 . 2024-04-26T17:56:58.548 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 .. 2024-04-26T17:56:58.548 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 112 2024-04-26T17:56:58.548 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 113 2024-04-26T17:56:58.548 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 17:56 1247 2024-04-26T17:56:58.548 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1247_fsm 2024-04-26T17:56:58.548 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1247_vm 2024-04-26T17:56:58.548 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 17:56 1249 2024-04-26T17:56:58.548 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1249_fsm 2024-04-26T17:56:58.548 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1249_vm 2024-04-26T17:56:58.548 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 17:56 1255 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1255_fsm 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1255_vm 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 17:56 1259 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1259_fsm 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1259_vm 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 17:56 13344 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13344_fsm 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13344_vm 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13346 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13348 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13349 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13349_fsm 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13349_vm 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13351 2024-04-26T17:56:58.549 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13353 2024-04-26T17:56:58.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13354 2024-04-26T17:56:58.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13354_fsm 2024-04-26T17:56:58.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13354_vm 2024-04-26T17:56:58.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13356 2024-04-26T17:56:58.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13358 2024-04-26T17:56:58.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13359 2024-04-26T17:56:58.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13359_fsm 2024-04-26T17:56:58.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13359_vm 2024-04-26T17:56:58.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13361 2024-04-26T17:56:58.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13363 2024-04-26T17:56:58.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 1417 2024-04-26T17:56:58.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 1418 2024-04-26T17:56:58.550 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 174 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 175 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2187 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2224 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2328 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2336 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2337 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2579 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2600 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2600_fsm 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2600_vm 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2601 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2601_fsm 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2601_vm 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2602 2024-04-26T17:56:58.551 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2602_fsm 2024-04-26T17:56:58.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2602_vm 2024-04-26T17:56:58.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2603 2024-04-26T17:56:58.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2603_fsm 2024-04-26T17:56:58.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2603_vm 2024-04-26T17:56:58.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2604 2024-04-26T17:56:58.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2605 2024-04-26T17:56:58.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2605_fsm 2024-04-26T17:56:58.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2605_vm 2024-04-26T17:56:58.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2606 2024-04-26T17:56:58.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2606_fsm 2024-04-26T17:56:58.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2606_vm 2024-04-26T17:56:58.552 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2607 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2607_fsm 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2607_vm 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 17:56 2608 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2608_fsm 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2608_vm 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 17:56 2609 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2609_fsm 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2609_vm 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2610 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2610_fsm 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2610_vm 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2611 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2612 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2612_fsm 2024-04-26T17:56:58.553 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2612_vm 2024-04-26T17:56:58.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2613 2024-04-26T17:56:58.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2615 2024-04-26T17:56:58.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2615_fsm 2024-04-26T17:56:58.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2615_vm 2024-04-26T17:56:58.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2616 2024-04-26T17:56:58.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2616_fsm 2024-04-26T17:56:58.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2616_vm 2024-04-26T17:56:58.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 17:56 2617 2024-04-26T17:56:58.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2617_fsm 2024-04-26T17:56:58.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2617_vm 2024-04-26T17:56:58.554 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 17:56 2618 2024-04-26T17:56:58.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2618_fsm 2024-04-26T17:56:58.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2618_vm 2024-04-26T17:56:58.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 26 17:56 2619 2024-04-26T17:56:58.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2619_fsm 2024-04-26T17:56:58.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2619_vm 2024-04-26T17:56:58.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2620 2024-04-26T17:56:58.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2650 2024-04-26T17:56:58.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2651 2024-04-26T17:56:58.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2652 2024-04-26T17:56:58.555 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2653 2024-04-26T17:56:58.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2654 2024-04-26T17:56:58.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2655 2024-04-26T17:56:58.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2656 2024-04-26T17:56:58.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2657 2024-04-26T17:56:58.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 17:56 2658 2024-04-26T17:56:58.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 17:56 2659 2024-04-26T17:56:58.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2660 2024-04-26T17:56:58.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2661 2024-04-26T17:56:58.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2662 2024-04-26T17:56:58.556 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2663 2024-04-26T17:56:58.557 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2664 2024-04-26T17:56:58.557 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2665 2024-04-26T17:56:58.557 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2666 2024-04-26T17:56:58.557 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2667 2024-04-26T17:56:58.557 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2668 2024-04-26T17:56:58.559 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2669 2024-04-26T17:56:58.559 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2670 2024-04-26T17:56:58.559 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 17:56 2673 2024-04-26T17:56:58.559 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 17:56 2674 2024-04-26T17:56:58.559 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 17:56 2675 2024-04-26T17:56:58.559 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2678 2024-04-26T17:56:58.559 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2679 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2680 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2681 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2682 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2683 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2684 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2685 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2686 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2687 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2688 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2689 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 17:56 2690 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 17:56 2691 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2692 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2693 2024-04-26T17:56:58.560 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2696 2024-04-26T17:56:58.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2699 2024-04-26T17:56:58.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2701 2024-04-26T17:56:58.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2702 2024-04-26T17:56:58.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2703 2024-04-26T17:56:58.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2704 2024-04-26T17:56:58.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2753 2024-04-26T17:56:58.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2753_fsm 2024-04-26T17:56:58.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2753_vm 2024-04-26T17:56:58.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2754 2024-04-26T17:56:58.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2755 2024-04-26T17:56:58.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2756 2024-04-26T17:56:58.561 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2757 2024-04-26T17:56:58.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2830 2024-04-26T17:56:58.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2831 2024-04-26T17:56:58.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2832 2024-04-26T17:56:58.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2833 2024-04-26T17:56:58.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2834 2024-04-26T17:56:58.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2835 2024-04-26T17:56:58.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2836 2024-04-26T17:56:58.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2837 2024-04-26T17:56:58.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 17:56 2838 2024-04-26T17:56:58.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2838_fsm 2024-04-26T17:56:58.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2838_vm 2024-04-26T17:56:58.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2839 2024-04-26T17:56:58.562 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2840 2024-04-26T17:56:58.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2840_fsm 2024-04-26T17:56:58.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2840_vm 2024-04-26T17:56:58.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2841 2024-04-26T17:56:58.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2995 2024-04-26T17:56:58.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2996 2024-04-26T17:56:58.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3079 2024-04-26T17:56:58.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3079_fsm 2024-04-26T17:56:58.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3079_vm 2024-04-26T17:56:58.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3080 2024-04-26T17:56:58.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3081 2024-04-26T17:56:58.563 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 3085 2024-04-26T17:56:58.564 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3118 2024-04-26T17:56:58.564 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3119 2024-04-26T17:56:58.564 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 17:56 3164 2024-04-26T17:56:58.564 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3256 2024-04-26T17:56:58.564 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3257 2024-04-26T17:56:58.564 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3258 2024-04-26T17:56:58.564 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3350 2024-04-26T17:56:58.564 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3351 2024-04-26T17:56:58.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3379 2024-04-26T17:56:58.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3380 2024-04-26T17:56:58.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3381 2024-04-26T17:56:58.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3394 2024-04-26T17:56:58.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3394_fsm 2024-04-26T17:56:58.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3394_vm 2024-04-26T17:56:58.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3395 2024-04-26T17:56:58.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3429 2024-04-26T17:56:58.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3430 2024-04-26T17:56:58.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3431 2024-04-26T17:56:58.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3433 2024-04-26T17:56:58.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3439 2024-04-26T17:56:58.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3440 2024-04-26T17:56:58.565 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 3455 2024-04-26T17:56:58.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 17:56 3456 2024-04-26T17:56:58.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3456_fsm 2024-04-26T17:56:58.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3456_vm 2024-04-26T17:56:58.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3466 2024-04-26T17:56:58.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3467 2024-04-26T17:56:58.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3468 2024-04-26T17:56:58.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3501 2024-04-26T17:56:58.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3502 2024-04-26T17:56:58.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3503 2024-04-26T17:56:58.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3534 2024-04-26T17:56:58.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3541 2024-04-26T17:56:58.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3541_fsm 2024-04-26T17:56:58.566 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3541_vm 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3542 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3574 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3575 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3576 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3596 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3597 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3598 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3599 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3600 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3600_fsm 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3600_vm 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3601 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3601_fsm 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3601_vm 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3602 2024-04-26T17:56:58.567 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3602_fsm 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3602_vm 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3603 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3603_fsm 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3603_vm 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3604 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3605 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3606 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3607 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3608 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 3609 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3712 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3764 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3764_fsm 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3764_vm 2024-04-26T17:56:58.568 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3766 2024-04-26T17:56:58.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3767 2024-04-26T17:56:58.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3997 2024-04-26T17:56:58.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4143 2024-04-26T17:56:58.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4144 2024-04-26T17:56:58.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4145 2024-04-26T17:56:58.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4146 2024-04-26T17:56:58.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4147 2024-04-26T17:56:58.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4148 2024-04-26T17:56:58.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4149 2024-04-26T17:56:58.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4150 2024-04-26T17:56:58.569 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4151 2024-04-26T17:56:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4152 2024-04-26T17:56:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4153 2024-04-26T17:56:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4154 2024-04-26T17:56:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4155 2024-04-26T17:56:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4156 2024-04-26T17:56:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4157 2024-04-26T17:56:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4158 2024-04-26T17:56:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4159 2024-04-26T17:56:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4160 2024-04-26T17:56:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4161 2024-04-26T17:56:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4162 2024-04-26T17:56:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4163 2024-04-26T17:56:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4164 2024-04-26T17:56:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4165 2024-04-26T17:56:58.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4166 2024-04-26T17:56:58.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4167 2024-04-26T17:56:58.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4168 2024-04-26T17:56:58.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4169 2024-04-26T17:56:58.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4170 2024-04-26T17:56:58.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4171 2024-04-26T17:56:58.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4172 2024-04-26T17:56:58.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4173 2024-04-26T17:56:58.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4174 2024-04-26T17:56:58.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 5002 2024-04-26T17:56:58.571 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 548 2024-04-26T17:56:58.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 549 2024-04-26T17:56:58.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6102 2024-04-26T17:56:58.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6104 2024-04-26T17:56:58.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6106 2024-04-26T17:56:58.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6110 2024-04-26T17:56:58.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6111 2024-04-26T17:56:58.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6112 2024-04-26T17:56:58.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6113 2024-04-26T17:56:58.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6117 2024-04-26T17:56:58.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 826 2024-04-26T17:56:58.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 827 2024-04-26T17:56:58.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 828 2024-04-26T17:56:58.572 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 17:56 pg_filenode.map 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 PG_VERSION 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout:total 7777 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 17:56 . 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 .. 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 112 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 113 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 17:56 1247 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1247_fsm 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1247_vm 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 17:56 1249 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1249_fsm 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1249_vm 2024-04-26T17:56:58.573 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 17:56 1255 2024-04-26T17:56:58.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1255_fsm 2024-04-26T17:56:58.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1255_vm 2024-04-26T17:56:58.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 17:56 1259 2024-04-26T17:56:58.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1259_fsm 2024-04-26T17:56:58.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1259_vm 2024-04-26T17:56:58.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 17:56 13344 2024-04-26T17:56:58.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13344_fsm 2024-04-26T17:56:58.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13344_vm 2024-04-26T17:56:58.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13346 2024-04-26T17:56:58.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13348 2024-04-26T17:56:58.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13349 2024-04-26T17:56:58.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13349_fsm 2024-04-26T17:56:58.574 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13349_vm 2024-04-26T17:56:58.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13351 2024-04-26T17:56:58.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13353 2024-04-26T17:56:58.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13354 2024-04-26T17:56:58.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13354_fsm 2024-04-26T17:56:58.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13354_vm 2024-04-26T17:56:58.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13356 2024-04-26T17:56:58.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13358 2024-04-26T17:56:58.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13359 2024-04-26T17:56:58.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13359_fsm 2024-04-26T17:56:58.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13359_vm 2024-04-26T17:56:58.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13361 2024-04-26T17:56:58.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13363 2024-04-26T17:56:58.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 1417 2024-04-26T17:56:58.575 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 1418 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 174 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 175 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2187 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2224 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2328 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2336 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2337 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2579 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2600 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2600_fsm 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2600_vm 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2601 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2601_fsm 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2601_vm 2024-04-26T17:56:58.576 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2602 2024-04-26T17:56:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2602_fsm 2024-04-26T17:56:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2602_vm 2024-04-26T17:56:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2603 2024-04-26T17:56:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2603_fsm 2024-04-26T17:56:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2603_vm 2024-04-26T17:56:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2604 2024-04-26T17:56:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2605 2024-04-26T17:56:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2605_fsm 2024-04-26T17:56:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2605_vm 2024-04-26T17:56:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2606 2024-04-26T17:56:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2606_fsm 2024-04-26T17:56:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2606_vm 2024-04-26T17:56:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2607 2024-04-26T17:56:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2607_fsm 2024-04-26T17:56:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2607_vm 2024-04-26T17:56:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 17:56 2608 2024-04-26T17:56:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2608_fsm 2024-04-26T17:56:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2608_vm 2024-04-26T17:56:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 17:56 2609 2024-04-26T17:56:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2609_fsm 2024-04-26T17:56:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2609_vm 2024-04-26T17:56:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2610 2024-04-26T17:56:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2610_fsm 2024-04-26T17:56:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2610_vm 2024-04-26T17:56:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2611 2024-04-26T17:56:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2612 2024-04-26T17:56:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2612_fsm 2024-04-26T17:56:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2612_vm 2024-04-26T17:56:58.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2613 2024-04-26T17:56:58.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2615 2024-04-26T17:56:58.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2615_fsm 2024-04-26T17:56:58.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2615_vm 2024-04-26T17:56:58.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2616 2024-04-26T17:56:58.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2616_fsm 2024-04-26T17:56:58.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2616_vm 2024-04-26T17:56:58.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 17:56 2617 2024-04-26T17:56:58.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2617_fsm 2024-04-26T17:56:58.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2617_vm 2024-04-26T17:56:58.579 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 17:56 2618 2024-04-26T17:56:58.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2618_fsm 2024-04-26T17:56:58.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2618_vm 2024-04-26T17:56:58.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 26 17:56 2619 2024-04-26T17:56:58.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2619_fsm 2024-04-26T17:56:58.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2619_vm 2024-04-26T17:56:58.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2620 2024-04-26T17:56:58.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2650 2024-04-26T17:56:58.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2651 2024-04-26T17:56:58.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2652 2024-04-26T17:56:58.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2653 2024-04-26T17:56:58.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2654 2024-04-26T17:56:58.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2655 2024-04-26T17:56:58.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2656 2024-04-26T17:56:58.580 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2657 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 17:56 2658 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 17:56 2659 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2660 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2661 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2662 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2663 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2664 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2665 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2666 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2667 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2668 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2669 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2670 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 17:56 2673 2024-04-26T17:56:58.581 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 17:56 2674 2024-04-26T17:56:58.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 17:56 2675 2024-04-26T17:56:58.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2678 2024-04-26T17:56:58.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2679 2024-04-26T17:56:58.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2680 2024-04-26T17:56:58.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2681 2024-04-26T17:56:58.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2682 2024-04-26T17:56:58.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2683 2024-04-26T17:56:58.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2684 2024-04-26T17:56:58.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2685 2024-04-26T17:56:58.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2686 2024-04-26T17:56:58.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2687 2024-04-26T17:56:58.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2688 2024-04-26T17:56:58.582 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2689 2024-04-26T17:56:58.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 17:56 2690 2024-04-26T17:56:58.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 17:56 2691 2024-04-26T17:56:58.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2692 2024-04-26T17:56:58.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2693 2024-04-26T17:56:58.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2696 2024-04-26T17:56:58.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2699 2024-04-26T17:56:58.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2701 2024-04-26T17:56:58.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2702 2024-04-26T17:56:58.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2703 2024-04-26T17:56:58.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2704 2024-04-26T17:56:58.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2753 2024-04-26T17:56:58.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2753_fsm 2024-04-26T17:56:58.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2753_vm 2024-04-26T17:56:58.583 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2754 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2755 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2756 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2757 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2830 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2831 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2832 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2833 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2834 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2835 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2836 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2837 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 17:56 2838 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2838_fsm 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2838_vm 2024-04-26T17:56:58.584 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2839 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2840 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2840_fsm 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2840_vm 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2841 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2995 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2996 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3079 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3079_fsm 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3079_vm 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3080 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3081 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 3085 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3118 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3119 2024-04-26T17:56:58.585 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 17:56 3164 2024-04-26T17:56:58.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3256 2024-04-26T17:56:58.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3257 2024-04-26T17:56:58.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3258 2024-04-26T17:56:58.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3350 2024-04-26T17:56:58.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3351 2024-04-26T17:56:58.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3379 2024-04-26T17:56:58.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3380 2024-04-26T17:56:58.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3381 2024-04-26T17:56:58.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3394 2024-04-26T17:56:58.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3394_fsm 2024-04-26T17:56:58.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3394_vm 2024-04-26T17:56:58.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3395 2024-04-26T17:56:58.586 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3429 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3430 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3431 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3433 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3439 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3440 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 3455 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 17:56 3456 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3456_fsm 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3456_vm 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3466 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3467 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3468 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3501 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3502 2024-04-26T17:56:58.587 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3503 2024-04-26T17:56:58.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3534 2024-04-26T17:56:58.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3541 2024-04-26T17:56:58.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3541_fsm 2024-04-26T17:56:58.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3541_vm 2024-04-26T17:56:58.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3542 2024-04-26T17:56:58.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3574 2024-04-26T17:56:58.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3575 2024-04-26T17:56:58.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3576 2024-04-26T17:56:58.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3596 2024-04-26T17:56:58.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3597 2024-04-26T17:56:58.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3598 2024-04-26T17:56:58.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3599 2024-04-26T17:56:58.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3600 2024-04-26T17:56:58.588 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3600_fsm 2024-04-26T17:56:58.589 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3600_vm 2024-04-26T17:56:58.589 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3601 2024-04-26T17:56:58.589 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3601_fsm 2024-04-26T17:56:58.589 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3601_vm 2024-04-26T17:56:58.589 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3602 2024-04-26T17:56:58.589 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3602_fsm 2024-04-26T17:56:58.589 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3602_vm 2024-04-26T17:56:58.589 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3603 2024-04-26T17:56:58.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3603_fsm 2024-04-26T17:56:58.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3603_vm 2024-04-26T17:56:58.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3604 2024-04-26T17:56:58.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3605 2024-04-26T17:56:58.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3606 2024-04-26T17:56:58.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3607 2024-04-26T17:56:58.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3608 2024-04-26T17:56:58.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 3609 2024-04-26T17:56:58.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3712 2024-04-26T17:56:58.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3764 2024-04-26T17:56:58.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3764_fsm 2024-04-26T17:56:58.590 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3764_vm 2024-04-26T17:56:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3766 2024-04-26T17:56:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3767 2024-04-26T17:56:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3997 2024-04-26T17:56:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4143 2024-04-26T17:56:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4144 2024-04-26T17:56:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4145 2024-04-26T17:56:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4146 2024-04-26T17:56:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4147 2024-04-26T17:56:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4148 2024-04-26T17:56:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4149 2024-04-26T17:56:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4150 2024-04-26T17:56:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4151 2024-04-26T17:56:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4152 2024-04-26T17:56:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4153 2024-04-26T17:56:58.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4154 2024-04-26T17:56:58.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4155 2024-04-26T17:56:58.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4156 2024-04-26T17:56:58.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4157 2024-04-26T17:56:58.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4158 2024-04-26T17:56:58.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4159 2024-04-26T17:56:58.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4160 2024-04-26T17:56:58.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4161 2024-04-26T17:56:58.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4162 2024-04-26T17:56:58.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4163 2024-04-26T17:56:58.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4164 2024-04-26T17:56:58.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4165 2024-04-26T17:56:58.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4166 2024-04-26T17:56:58.592 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4167 2024-04-26T17:56:58.593 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4168 2024-04-26T17:56:58.593 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4169 2024-04-26T17:56:58.593 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4170 2024-04-26T17:56:58.593 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4171 2024-04-26T17:56:58.593 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4172 2024-04-26T17:56:58.593 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4173 2024-04-26T17:56:58.593 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4174 2024-04-26T17:56:58.593 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 5002 2024-04-26T17:56:58.593 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 548 2024-04-26T17:56:58.593 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 549 2024-04-26T17:56:58.593 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6102 2024-04-26T17:56:58.593 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6104 2024-04-26T17:56:58.593 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6106 2024-04-26T17:56:58.593 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6110 2024-04-26T17:56:58.594 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6111 2024-04-26T17:56:58.594 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6112 2024-04-26T17:56:58.594 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6113 2024-04-26T17:56:58.594 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6117 2024-04-26T17:56:58.594 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 826 2024-04-26T17:56:58.594 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 827 2024-04-26T17:56:58.594 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 828 2024-04-26T17:56:58.594 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 17:56 pg_filenode.map 2024-04-26T17:56:58.594 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 PG_VERSION 2024-04-26T17:56:58.594 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.594 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/global: 2024-04-26T17:56:58.594 INFO:teuthology.orchestra.run.smithi031.stdout:total 537 2024-04-26T17:56:58.594 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 26 17:56 . 2024-04-26T17:56:58.595 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.595 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1213 2024-04-26T17:56:58.595 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1213_fsm 2024-04-26T17:56:58.595 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1213_vm 2024-04-26T17:56:58.595 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1214 2024-04-26T17:56:58.595 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1214_fsm 2024-04-26T17:56:58.595 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1214_vm 2024-04-26T17:56:58.595 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 1232 2024-04-26T17:56:58.595 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 1233 2024-04-26T17:56:58.595 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1260 2024-04-26T17:56:58.595 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1260_fsm 2024-04-26T17:56:58.595 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1260_vm 2024-04-26T17:56:58.595 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1261 2024-04-26T17:56:58.595 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1261_fsm 2024-04-26T17:56:58.596 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1261_vm 2024-04-26T17:56:58.596 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1262 2024-04-26T17:56:58.596 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1262_fsm 2024-04-26T17:56:58.596 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1262_vm 2024-04-26T17:56:58.596 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2396 2024-04-26T17:56:58.596 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2396_fsm 2024-04-26T17:56:58.596 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2396_vm 2024-04-26T17:56:58.596 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2397 2024-04-26T17:56:58.596 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2671 2024-04-26T17:56:58.596 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2672 2024-04-26T17:56:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2676 2024-04-26T17:56:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2677 2024-04-26T17:56:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2694 2024-04-26T17:56:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2695 2024-04-26T17:56:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2697 2024-04-26T17:56:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2698 2024-04-26T17:56:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2846 2024-04-26T17:56:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2847 2024-04-26T17:56:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2964 2024-04-26T17:56:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2965 2024-04-26T17:56:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2966 2024-04-26T17:56:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2967 2024-04-26T17:56:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3592 2024-04-26T17:56:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3593 2024-04-26T17:56:58.598 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4060 2024-04-26T17:56:58.598 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4061 2024-04-26T17:56:58.598 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4175 2024-04-26T17:56:58.598 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4176 2024-04-26T17:56:58.598 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4177 2024-04-26T17:56:58.598 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4178 2024-04-26T17:56:58.598 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4181 2024-04-26T17:56:58.598 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4182 2024-04-26T17:56:58.598 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4183 2024-04-26T17:56:58.598 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4184 2024-04-26T17:56:58.598 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4185 2024-04-26T17:56:58.598 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4186 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6000 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6001 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6002 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6100 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6114 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6115 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 pg_control 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 17:56 pg_filenode.map 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/log: 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.599 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-26T17:56:58.600 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.600 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:58.600 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.600 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.600 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-26T17:56:58.600 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.600 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:58.600 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.600 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.600 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-26T17:56:58.600 INFO:teuthology.orchestra.run.smithi031.stdout:total 1 2024-04-26T17:56:58.600 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 . 2024-04-26T17:56:58.600 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.600 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 mappings 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 26 17:56 replorigin_checkpoint 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 snapshots 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 .. 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 .. 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.601 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 17:56 . 2024-04-26T17:56:58.602 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.602 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 members 2024-04-26T17:56:58.602 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 offsets 2024-04-26T17:56:58.602 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.602 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-26T17:56:58.602 INFO:teuthology.orchestra.run.smithi031.stdout:total 8 2024-04-26T17:56:58.602 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 . 2024-04-26T17:56:58.602 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 17:56 .. 2024-04-26T17:56:58.602 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 0000 2024-04-26T17:56:58.602 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.602 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-26T17:56:58.602 INFO:teuthology.orchestra.run.smithi031.stdout:total 8 2024-04-26T17:56:58.602 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 . 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 17:56 .. 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 0000 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.603 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:58.605 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.605 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.605 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-26T17:56:58.605 INFO:teuthology.orchestra.run.smithi031.stdout:total 8 2024-04-26T17:56:58.605 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 . 2024-04-26T17:56:58.605 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.605 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 0000 2024-04-26T17:56:58.605 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.605 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-26T17:56:58.605 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.605 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:58.605 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.605 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.605 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-26T17:56:58.606 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.606 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:58.606 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.606 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.606 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-26T17:56:58.606 INFO:teuthology.orchestra.run.smithi031.stdout:total 16384 2024-04-26T17:56:58.606 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 17:56 . 2024-04-26T17:56:58.606 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.606 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 17:56 000000010000000000000001 2024-04-26T17:56:58.606 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 archive_status 2024-04-26T17:56:58.606 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.606 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-26T17:56:58.606 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T17:56:58.607 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T17:56:58.607 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 17:56 .. 2024-04-26T17:56:58.607 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:56:58.607 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-26T17:56:58.607 INFO:teuthology.orchestra.run.smithi031.stdout:total 8 2024-04-26T17:56:58.607 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 . 2024-04-26T17:56:58.607 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 26 17:56 .. 2024-04-26T17:56:58.607 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 0000 2024-04-26T17:56:58.608 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2024-04-26T17:56:58.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.528+0000 7f336967f740 1 Processor -- start 2024-04-26T17:56:58.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.528+0000 7f336967f740 1 -- start start 2024-04-26T17:56:58.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.528+0000 7f336967f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add1e00 0x55ec0adcd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.528+0000 7f336967f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 0x55ec0adf6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.528+0000 7f336967f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0ade14f0 0x55ec0adf6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.528+0000 7f336967f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f33601aeae0 con 0x55ec0ade14f0 2024-04-26T17:56:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.528+0000 7f336967f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f33601ac3b0 con 0x55ec0add1e00 2024-04-26T17:56:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.528+0000 7f336967f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f33601aa220 con 0x55ec0add2720 2024-04-26T17:56:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.529+0000 7f3364f03640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0ade14f0 0x55ec0adf6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.529+0000 7f335ffff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add1e00 0x55ec0adcd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.529+0000 7f3364f03640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0ade14f0 0x55ec0adf6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39980/0 (socket says 172.21.15.31:39980) 2024-04-26T17:56:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.529+0000 7f3364f03640 1 -- 172.21.15.31:0/1004767763 learned_addr learned my addr 172.21.15.31:0/1004767763 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.529+0000 7f335f7fe640 1 --2- 172.21.15.31:0/1004767763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 0x55ec0adf6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.529+0000 7f335ffff640 1 -- 172.21.15.31:0/1004767763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 msgr2=0x55ec0adf6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.529+0000 7f335ffff640 1 --2- 172.21.15.31:0/1004767763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 0x55ec0adf6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.529+0000 7f335ffff640 1 -- 172.21.15.31:0/1004767763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0ade14f0 msgr2=0x55ec0adf6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.529+0000 7f335ffff640 1 --2- 172.21.15.31:0/1004767763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0ade14f0 0x55ec0adf6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.529+0000 7f335ffff640 1 -- 172.21.15.31:0/1004767763 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec0add2550 con 0x55ec0add1e00 2024-04-26T17:56:58.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.529+0000 7f335ffff640 1 --2- 172.21.15.31:0/1004767763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add1e00 0x55ec0adcd430 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f335009a470 tx=0x7f33500041d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26694695798d926a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.529+0000 7f3355ffb640 1 -- 172.21.15.31:0/1004767763 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33500b9040 con 0x55ec0add1e00 2024-04-26T17:56:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.529+0000 7f3355ffb640 1 -- 172.21.15.31:0/1004767763 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33500cdcd0 con 0x55ec0add1e00 2024-04-26T17:56:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.529+0000 7f3355ffb640 1 -- 172.21.15.31:0/1004767763 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33500b5040 con 0x55ec0add1e00 2024-04-26T17:56:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.530+0000 7f336967f740 1 -- 172.21.15.31:0/1004767763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add1e00 msgr2=0x55ec0adcd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.530+0000 7f336967f740 1 --2- 172.21.15.31:0/1004767763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add1e00 0x55ec0adcd430 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f335009a470 tx=0x7f33500041d0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.530+0000 7f336967f740 1 -- 172.21.15.31:0/1004767763 shutdown_connections 2024-04-26T17:56:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.530+0000 7f336967f740 1 --2- 172.21.15.31:0/1004767763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0ade14f0 0x55ec0adf6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.530+0000 7f336967f740 1 --2- 172.21.15.31:0/1004767763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0add2720 0x55ec0adf6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.530+0000 7f336967f740 1 --2- 172.21.15.31:0/1004767763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add1e00 0x55ec0adcd430 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.530+0000 7f336967f740 1 -- 172.21.15.31:0/1004767763 >> 172.21.15.31:0/1004767763 conn(0x55ec0adce410 msgr2=0x55ec0adccd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.530+0000 7f336967f740 1 -- 172.21.15.31:0/1004767763 shutdown_connections 2024-04-26T17:56:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.530+0000 7f336967f740 1 -- 172.21.15.31:0/1004767763 wait complete. 2024-04-26T17:56:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.530+0000 7f336967f740 1 -- start start 2024-04-26T17:56:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f336967f740 1 Processor -- start 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f336967f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0add1e00 0x55ec0adf6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f336967f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add2720 0x55ec0adf4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f336967f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0ade14f0 0x55ec0adf49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f336967f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f33601aa870 con 0x55ec0add1e00 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f336967f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f33601abce0 con 0x55ec0add2720 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f336967f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f33601aaf40 con 0x55ec0ade14f0 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f336967f740 10 client.0 _ll_register_callbacks cb 0x7f33500b3070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f335ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0add1e00 0x55ec0adf6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f335ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0add1e00 0x55ec0adf6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39994/0 (socket says 172.21.15.31:39994) 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f335ffff640 1 -- 172.21.15.31:0/4206314788 learned_addr learned my addr 172.21.15.31:0/4206314788 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f3364f03640 1 --2- 172.21.15.31:0/4206314788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0ade14f0 0x55ec0adf49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f335f7fe640 1 --2- 172.21.15.31:0/4206314788 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add2720 0x55ec0adf4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f335ffff640 1 -- 172.21.15.31:0/4206314788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0ade14f0 msgr2=0x55ec0adf49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f335ffff640 1 --2- 172.21.15.31:0/4206314788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0ade14f0 0x55ec0adf49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f335ffff640 1 -- 172.21.15.31:0/4206314788 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add2720 msgr2=0x55ec0adf4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:56:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f335ffff640 1 --2- 172.21.15.31:0/4206314788 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add2720 0x55ec0adf4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f335ffff640 1 -- 172.21.15.31:0/4206314788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec0ae56740 con 0x55ec0add1e00 2024-04-26T17:56:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f3364f03640 1 --2- 172.21.15.31:0/4206314788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0ade14f0 0x55ec0adf49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:56:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f335ffff640 1 --2- 172.21.15.31:0/4206314788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0add1e00 0x55ec0adf6e80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f33500a0730 tx=0x7f3350098a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4b64993646b622 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:56:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.531+0000 7f333f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:56:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.532+0000 7f333f7fe640 1 -- 172.21.15.31:0/4206314788 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33500b9070 con 0x55ec0add1e00 2024-04-26T17:56:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.532+0000 7f333f7fe640 1 -- 172.21.15.31:0/4206314788 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f335009a4c0 con 0x55ec0add1e00 2024-04-26T17:56:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.532+0000 7f336967f740 1 -- 172.21.15.31:0/4206314788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33500cdcd0 con 0x55ec0add1e00 2024-04-26T17:56:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.532+0000 7f333f7fe640 1 -- 172.21.15.31:0/4206314788 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f33500b5040 con 0x55ec0add1e00 2024-04-26T17:56:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.532+0000 7f336967f740 10 client.15324 fetch_fsmap learned FSMap version 27 2024-04-26T17:56:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.532+0000 7f336967f740 1 -- 172.21.15.31:0/4206314788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ec0ae56740 con 0x55ec0add1e00 2024-04-26T17:56:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.532+0000 7f333f7fe640 1 -- 172.21.15.31:0/4206314788 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33500cdcd0 con 0x55ec0add1e00 2024-04-26T17:56:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.532+0000 7f333f7fe640 1 -- 172.21.15.31:0/4206314788 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7f335009b210 con 0x55ec0add1e00 2024-04-26T17:56:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.532+0000 7f336967f740 10 client.15324 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T17:56:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.532+0000 7f336967f740 10 client.15324 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:56:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.532+0000 7f336967f740 20 client.15324 populate_metadata read hostname 'smithi031' 2024-04-26T17:56:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.532+0000 7f336967f740 1 --2- 172.21.15.31:0/4206314788 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec0ae57b20 0x55ec0ae59f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:56:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.532+0000 7f336967f740 4 client.15324 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:56:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.532+0000 7f336967f740 1 -- 172.21.15.31:0/4206314788 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ec0ad3fee0 con 0x55ec0ae57b20 2024-04-26T17:56:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.533+0000 7f335f7fe640 1 --2- 172.21.15.31:0/4206314788 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec0ae57b20 0x55ec0ae59f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:56:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.533+0000 7f333f7fe640 10 client.15324 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:56:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.533+0000 7f335f7fe640 1 --2- 172.21.15.31:0/4206314788 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec0ae57b20 0x55ec0ae59f10 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f83dd2e2e4ccef63 server_cookie=8edc304af85eb4c3 in_seq=0 out_seq=0 2024-04-26T17:56:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.537+0000 7f333f7fe640 1 -- 172.21.15.31:0/4206314788 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ec0ad3fee0 con 0x55ec0ae57b20 2024-04-26T17:56:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.537+0000 7f333f7fe640 10 client.15324 handle_command_reply: tid=0 2024-04-26T17:56:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.537+0000 7f336967f740 1 client.15324 shutdown 2024-04-26T17:56:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.537+0000 7f336967f740 1 -- 172.21.15.31:0/4206314788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0add1e00 msgr2=0x55ec0adf6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.537+0000 7f336967f740 1 --2- 172.21.15.31:0/4206314788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0add1e00 0x55ec0adf6e80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f33500a0730 tx=0x7f3350098a60 comp rx=0 tx=0).stop 2024-04-26T17:56:58.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.537+0000 7f336967f740 1 -- 172.21.15.31:0/4206314788 shutdown_connections 2024-04-26T17:56:58.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.537+0000 7f336967f740 1 --2- 172.21.15.31:0/4206314788 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec0ae57b20 0x55ec0ae59f10 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.537+0000 7f336967f740 1 --2- 172.21.15.31:0/4206314788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec0ade14f0 0x55ec0adf49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.537+0000 7f333f7fe640 0 client.15324 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:56:58.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.537+0000 7f336967f740 1 --2- 172.21.15.31:0/4206314788 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec0add2720 0x55ec0adf4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.537+0000 7f336967f740 1 --2- 172.21.15.31:0/4206314788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec0add1e00 0x55ec0adf6e80 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.537+0000 7f336967f740 1 -- 172.21.15.31:0/4206314788 >> 172.21.15.31:0/4206314788 conn(0x55ec0adce410 msgr2=0x55ec0adde400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:58.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f336967f740 1 -- 172.21.15.31:0/4206314788 shutdown_connections 2024-04-26T17:56:58.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f336967f740 1 -- 172.21.15.31:0/4206314788 wait complete. 2024-04-26T17:56:58.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f336967f740 20 client.15324 trim_cache size 0 max 16384 2024-04-26T17:56:58.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:56:58.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:56:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:56:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:56:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f3366966640 1 -- 172.21.15.31:0/3751694091 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f334c0a3ee0 msgr2=0x7f334c0a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f3366966640 1 --2- 172.21.15.31:0/3751694091 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f334c0a3ee0 0x7f334c0a63a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3360107f50 tx=0x7f334800b390 comp rx=0 tx=0).stop 2024-04-26T17:56:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f3366966640 1 -- 172.21.15.31:0/3751694091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f336006b160 msgr2=0x7f3360198f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:56:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f3366966640 1 --2- 172.21.15.31:0/3751694091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f336006b160 0x7f3360198f80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f335000d950 tx=0x7f335000de10 comp rx=0 tx=0).stop 2024-04-26T17:56:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f3366966640 1 -- 172.21.15.31:0/3751694091 shutdown_connections 2024-04-26T17:56:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f3366966640 1 --2- 172.21.15.31:0/3751694091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3360108b30 0x7f336019d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f3366966640 1 --2- 172.21.15.31:0/3751694091 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f336006bb20 0x7f33601994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f3366966640 1 --2- 172.21.15.31:0/3751694091 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f334c0a3ee0 0x7f334c0a63a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f3366966640 1 --2- 172.21.15.31:0/3751694091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f336006b160 0x7f3360198f80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:56:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f3366966640 1 -- 172.21.15.31:0/3751694091 >> 172.21.15.31:0/3751694091 conn(0x7f3360077630 msgr2=0x7f3360100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:56:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f3366966640 1 -- 172.21.15.31:0/3751694091 shutdown_connections 2024-04-26T17:56:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:56:58.538+0000 7f3366966640 1 -- 172.21.15.31:0/3751694091 wait complete. 2024-04-26T17:56:58.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 32 -i' 2024-04-26T17:56:58.757 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:56:58.789 INFO:teuthology.orchestra.run.smithi031.stderr:dropping old tables... 2024-04-26T17:56:58.789 INFO:teuthology.orchestra.run.smithi031.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2024-04-26T17:56:58.789 INFO:teuthology.orchestra.run.smithi031.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2024-04-26T17:56:58.789 INFO:teuthology.orchestra.run.smithi031.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2024-04-26T17:56:58.789 INFO:teuthology.orchestra.run.smithi031.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2024-04-26T17:56:58.789 INFO:teuthology.orchestra.run.smithi031.stderr:creating tables... 2024-04-26T17:56:58.799 INFO:teuthology.orchestra.run.smithi031.stderr:generating data (client-side)... 2024-04-26T17:56:58.872 INFO:teuthology.orchestra.run.smithi031.stderr:100000 of 3200000 tuples (3%) done (elapsed 0.05 s, remaining 1.70 s) 2024-04-26T17:56:59.035 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:56:58 smithi031 ceph-mon[29164]: pgmap v17: 97 pgs: 97 active+clean; 2.0 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 69 KiB/s wr, 9 op/s 2024-04-26T17:56:59.054 INFO:teuthology.orchestra.run.smithi031.stderr:200000 of 3200000 tuples (6%) done (elapsed 0.24 s, remaining 3.56 s) 2024-04-26T17:56:59.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:56:58 smithi097 ceph-mon[32419]: pgmap v17: 97 pgs: 97 active+clean; 2.0 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 69 KiB/s wr, 9 op/s 2024-04-26T17:56:59.169 INFO:teuthology.orchestra.run.smithi031.stderr:300000 of 3200000 tuples (9%) done (elapsed 0.35 s, remaining 3.41 s) 2024-04-26T17:56:59.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:56:58 smithi186 ceph-mon[32797]: pgmap v17: 97 pgs: 97 active+clean; 2.0 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 69 KiB/s wr, 9 op/s 2024-04-26T17:56:59.316 INFO:teuthology.orchestra.run.smithi031.stderr:400000 of 3200000 tuples (12%) done (elapsed 0.50 s, remaining 3.49 s) 2024-04-26T17:56:59.475 INFO:teuthology.orchestra.run.smithi031.stderr:500000 of 3200000 tuples (15%) done (elapsed 0.66 s, remaining 3.55 s) 2024-04-26T17:56:59.696 INFO:teuthology.orchestra.run.smithi031.stderr:600000 of 3200000 tuples (18%) done (elapsed 0.88 s, remaining 3.81 s) 2024-04-26T17:56:59.769 INFO:teuthology.orchestra.run.smithi031.stderr:700000 of 3200000 tuples (21%) done (elapsed 0.95 s, remaining 3.40 s) 2024-04-26T17:56:59.952 INFO:teuthology.orchestra.run.smithi031.stderr:800000 of 3200000 tuples (25%) done (elapsed 1.14 s, remaining 3.41 s) 2024-04-26T17:57:00.122 INFO:teuthology.orchestra.run.smithi031.stderr:900000 of 3200000 tuples (28%) done (elapsed 1.31 s, remaining 3.34 s) 2024-04-26T17:57:00.186 INFO:teuthology.orchestra.run.smithi031.stderr:1000000 of 3200000 tuples (31%) done (elapsed 1.37 s, remaining 3.01 s) 2024-04-26T17:57:00.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.481+0000 7fbf3249f700 1 -- 172.21.15.31:0/1393695149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c102310 msgr2=0x7fbf2c102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:00.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.481+0000 7fbf3249f700 1 --2- 172.21.15.31:0/1393695149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c102310 0x7fbf2c102790 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fbf20009990 tx=0x7fbf20009ca0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.481+0000 7fbf3249f700 1 -- 172.21.15.31:0/1393695149 shutdown_connections 2024-04-26T17:57:00.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.481+0000 7fbf3249f700 1 --2- 172.21.15.31:0/1393695149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf2c102cd0 0x7fbf2c10f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.481+0000 7fbf3249f700 1 --2- 172.21.15.31:0/1393695149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf2c108380 0x7fbf2c108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.481+0000 7fbf3249f700 1 --2- 172.21.15.31:0/1393695149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c102310 0x7fbf2c102790 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.481+0000 7fbf3249f700 1 -- 172.21.15.31:0/1393695149 >> 172.21.15.31:0/1393695149 conn(0x7fbf2c079290 msgr2=0x7fbf2c0796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:00.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.481+0000 7fbf3249f700 1 -- 172.21.15.31:0/1393695149 shutdown_connections 2024-04-26T17:57:00.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.481+0000 7fbf3249f700 1 -- 172.21.15.31:0/1393695149 wait complete. 2024-04-26T17:57:00.483 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:57:00 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:57:00.339+0000 7f50dc78b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:57:00.483 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:57:00 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:57:00.339+0000 7f50dc78b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:57:00.483 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:57:00 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:57:00.339+0000 7f50dc78b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:57:00.483 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:57:00 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:57:00.339+0000 7f50dc78b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:57:00.483 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 17:57:00 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T17:57:00.339+0000 7f50dc78b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T17:57:00.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.482+0000 7fbf3249f700 1 Processor -- start 2024-04-26T17:57:00.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.482+0000 7fbf3249f700 1 -- start start 2024-04-26T17:57:00.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.482+0000 7fbf3249f700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf2c102310 0x7fbf2c19f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.482+0000 7fbf3249f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf2c102cd0 0x7fbf2c19fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.482+0000 7fbf3249f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c108380 0x7fbf2c199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.482+0000 7fbf3249f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2c112510 con 0x7fbf2c108380 2024-04-26T17:57:00.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.482+0000 7fbf3249f700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2c112390 con 0x7fbf2c102310 2024-04-26T17:57:00.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.482+0000 7fbf3249f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2c112690 con 0x7fbf2c102cd0 2024-04-26T17:57:00.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.483+0000 7fbf2b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf2c102cd0 0x7fbf2c19fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.483+0000 7fbf2b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf2c102cd0 0x7fbf2c19fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:41154/0 (socket says 172.21.15.31:41154) 2024-04-26T17:57:00.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.483+0000 7fbf2b7fe700 1 -- 172.21.15.31:0/3150427125 learned_addr learned my addr 172.21.15.31:0/3150427125 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:00.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.483+0000 7fbf30a3c700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c108380 0x7fbf2c199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.483+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf2c102310 0x7fbf2c19f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.483+0000 7fbf2b7fe700 1 -- 172.21.15.31:0/3150427125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf2c102310 msgr2=0x7fbf2c19f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:00.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.483+0000 7fbf2b7fe700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf2c102310 0x7fbf2c19f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.483+0000 7fbf2b7fe700 1 -- 172.21.15.31:0/3150427125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c108380 msgr2=0x7fbf2c199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:00.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.483+0000 7fbf2b7fe700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c108380 0x7fbf2c199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.483+0000 7fbf2b7fe700 1 -- 172.21.15.31:0/3150427125 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf2c19a220 con 0x7fbf2c102cd0 2024-04-26T17:57:00.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.483+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf2c102310 0x7fbf2c19f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:00.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.483+0000 7fbf2b7fe700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf2c102cd0 0x7fbf2c19fdf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbf20005910 tx=0x7fbf2000b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:00.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.483+0000 7fbf297fa700 1 -- 172.21.15.31:0/3150427125 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf20027070 con 0x7fbf2c102cd0 2024-04-26T17:57:00.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.484+0000 7fbf3249f700 1 -- 172.21.15.31:0/3150427125 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf2c19a4b0 con 0x7fbf2c102cd0 2024-04-26T17:57:00.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.484+0000 7fbf297fa700 1 -- 172.21.15.31:0/3150427125 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf20022470 con 0x7fbf2c102cd0 2024-04-26T17:57:00.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.484+0000 7fbf297fa700 1 -- 172.21.15.31:0/3150427125 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf20017650 con 0x7fbf2c102cd0 2024-04-26T17:57:00.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.484+0000 7fbf3249f700 1 -- 172.21.15.31:0/3150427125 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf2c104220 con 0x7fbf2c102cd0 2024-04-26T17:57:00.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.486+0000 7fbf297fa700 1 -- 172.21.15.31:0/3150427125 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fbf2002a070 con 0x7fbf2c102cd0 2024-04-26T17:57:00.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.487+0000 7fbf297fa700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fbf140a3ec0 0x7fbf140a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.488+0000 7fbf297fa700 1 -- 172.21.15.31:0/3150427125 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fbf200d16b0 con 0x7fbf2c102cd0 2024-04-26T17:57:00.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.488+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fbf140a3ec0 0x7fbf140a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.488+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fbf140a3ec0 0x7fbf140a6380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c1008f0 tx=0x7fbf1c009260 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:00.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.488+0000 7fbf3cd31b80 1 Processor -- start 2024-04-26T17:57:00.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.488+0000 7fbf3cd31b80 1 -- start start 2024-04-26T17:57:00.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf3cd31b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf3cd31b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf3cd31b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5dbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf3cd31b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56391e5196f0 con 0x56391e5cfde0 2024-04-26T17:57:00.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf3cd31b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56391e4b8590 con 0x56391e5cf4c0 2024-04-26T17:57:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf3cd31b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56391e5156b0 con 0x56391e5d9b90 2024-04-26T17:57:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf2bfff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf2bfff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5cf8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:57224/0 (socket says 172.21.15.31:57224) 2024-04-26T17:57:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf2bfff700 1 -- 172.21.15.31:0/712090802 learned_addr learned my addr 172.21.15.31:0/712090802 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf2b7fe700 1 --2- 172.21.15.31:0/712090802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf30a3c700 1 --2- 172.21.15.31:0/712090802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5dbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf2bfff700 1 -- 172.21.15.31:0/712090802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 msgr2=0x56391e5dbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/712090802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5dbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf2bfff700 1 -- 172.21.15.31:0/712090802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 msgr2=0x56391e5d9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/712090802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf2bfff700 1 -- 172.21.15.31:0/712090802 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56391e466880 con 0x56391e5cf4c0 2024-04-26T17:57:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf2b7fe700 1 --2- 172.21.15.31:0/712090802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.489+0000 7fbf30a3c700 1 --2- 172.21.15.31:0/712090802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5dbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.490+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/712090802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5cf8a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c0043c0 tx=0x7fbf1c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c3b1b1067efa2e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.490+0000 7fbf127fc700 1 -- 172.21.15.31:0/712090802 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c003d10 con 0x56391e5cf4c0 2024-04-26T17:57:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.490+0000 7fbf127fc700 1 -- 172.21.15.31:0/712090802 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf1c01e8a0 con 0x56391e5cf4c0 2024-04-26T17:57:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.490+0000 7fbf127fc700 1 -- 172.21.15.31:0/712090802 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c00e7a0 con 0x56391e5cf4c0 2024-04-26T17:57:00.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.491+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/712090802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 msgr2=0x56391e5cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:00.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.491+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/712090802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5cf8a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c0043c0 tx=0x7fbf1c014000 comp rx=0 tx=0).stop 2024-04-26T17:57:00.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.491+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/712090802 shutdown_connections 2024-04-26T17:57:00.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.491+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/712090802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5cf8a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.491+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/712090802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5dbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.491+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/712090802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.491+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/712090802 >> 172.21.15.31:0/712090802 conn(0x56391e4f3cf0 msgr2=0x56391e5c2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.491+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/712090802 shutdown_connections 2024-04-26T17:57:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.491+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/712090802 wait complete. 2024-04-26T17:57:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.492+0000 7fbf3cd31b80 1 -- start start 2024-04-26T17:57:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.492+0000 7fbf3cd31b80 1 Processor -- start 2024-04-26T17:57:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf3cd31b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5d93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf3cd31b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5cfde0 0x56391e5d66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf3cd31b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5d9b90 0x56391e5d6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf3cd31b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56391e5dc760 con 0x56391e5d9b90 2024-04-26T17:57:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf3cd31b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56391e47f900 con 0x56391e5cf4c0 2024-04-26T17:57:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf3cd31b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56391e5dc8e0 con 0x56391e5cfde0 2024-04-26T17:57:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf3cd31b80 10 client.0 _ll_register_callbacks cb 0x56391e4ad530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:57:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf2bfff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5d93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf2b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5cfde0 0x56391e5d66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf2bfff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5d93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:57230/0 (socket says 172.21.15.31:57230) 2024-04-26T17:57:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf2bfff700 1 -- 172.21.15.31:0/3410402371 learned_addr learned my addr 172.21.15.31:0/3410402371 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf30a3c700 1 --2- 172.21.15.31:0/3410402371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5d9b90 0x56391e5d6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf2bfff700 1 -- 172.21.15.31:0/3410402371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5cfde0 msgr2=0x56391e5d66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/3410402371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5cfde0 0x56391e5d66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf2bfff700 1 -- 172.21.15.31:0/3410402371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5d9b90 msgr2=0x56391e5d6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/3410402371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5d9b90 0x56391e5d6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf2bfff700 1 -- 172.21.15.31:0/3410402371 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56391e5d7340 con 0x56391e5cf4c0 2024-04-26T17:57:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.493+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/3410402371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5d93b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c004390 tx=0x7fbf1c003dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67ce103673de45fa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.494+0000 7fbf0b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T17:57:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.494+0000 7fbf0b7fe700 1 -- 172.21.15.31:0/3410402371 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c01ece0 con 0x56391e5cf4c0 2024-04-26T17:57:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.494+0000 7fbf0b7fe700 1 -- 172.21.15.31:0/3410402371 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fbf1c02d070 con 0x56391e5cf4c0 2024-04-26T17:57:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.494+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/3410402371 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf1c01e8a0 con 0x56391e5cf4c0 2024-04-26T17:57:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.494+0000 7fbf0b7fe700 1 -- 172.21.15.31:0/3410402371 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbf1c038050 con 0x56391e5cf4c0 2024-04-26T17:57:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.494+0000 7fbf3cd31b80 10 client.25067 fetch_fsmap learned FSMap version 27 2024-04-26T17:57:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.494+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/3410402371 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56391e5e7280 con 0x56391e5cf4c0 2024-04-26T17:57:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.494+0000 7fbf0b7fe700 1 -- 172.21.15.31:0/3410402371 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf1c005500 con 0x56391e5cf4c0 2024-04-26T17:57:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.494+0000 7fbf0b7fe700 1 -- 172.21.15.31:0/3410402371 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7fbf1c026400 con 0x56391e5cf4c0 2024-04-26T17:57:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.494+0000 7fbf3cd31b80 10 client.25067 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T17:57:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.494+0000 7fbf3cd31b80 10 client.25067 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T17:57:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.494+0000 7fbf3cd31b80 20 client.25067 populate_metadata read hostname 'smithi031' 2024-04-26T17:57:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.494+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/3410402371 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x56391e5e9c40 0x56391e5eb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.495+0000 7fbf3cd31b80 4 client.25067 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:57:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.495+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/3410402371 --> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbf1c01e8a0 con 0x56391e5e9c40 2024-04-26T17:57:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.495+0000 7fbf2b7fe700 1 --2- 172.21.15.31:0/3410402371 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x56391e5e9c40 0x56391e5eb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.495+0000 7fbf0b7fe700 10 client.25067 ms_handle_connect on v2:172.21.15.186:6838/1587952500 2024-04-26T17:57:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.495+0000 7fbf2b7fe700 1 --2- 172.21.15.31:0/3410402371 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x56391e5e9c40 0x56391e5eb0b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2ed7a31562c9d810 server_cookie=be3cd2bf1d6fba2f in_seq=0 out_seq=0 2024-04-26T17:57:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.496+0000 7fbf0b7fe700 1 -- 172.21.15.31:0/3410402371 <== mds.1 v2:172.21.15.186:6838/1587952500 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fbf1c01e8a0 con 0x56391e5e9c40 2024-04-26T17:57:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.496+0000 7fbf0b7fe700 10 client.25067 handle_command_reply: tid=0 2024-04-26T17:57:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.496+0000 7fbf3cd31b80 1 client.25067 shutdown 2024-04-26T17:57:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.496+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/3410402371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 msgr2=0x56391e5d93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.496+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/3410402371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5d93b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c004390 tx=0x7fbf1c003dd0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.496+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/3410402371 shutdown_connections 2024-04-26T17:57:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.496+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/3410402371 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x56391e5e9c40 0x56391e5eb0b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.496+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/3410402371 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5d93b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.496+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/3410402371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5d9b90 0x56391e5d6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.496+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/3410402371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5cfde0 0x56391e5d66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.496+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/3410402371 >> 172.21.15.31:0/3410402371 conn(0x56391e4f3cf0 msgr2=0x56391e5c1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.496+0000 7fbf0b7fe700 0 client.25067 ms_handle_reset on v2:172.21.15.186:6838/1587952500 2024-04-26T17:57:00.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.497+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/3410402371 shutdown_connections 2024-04-26T17:57:00.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.497+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/3410402371 wait complete. 2024-04-26T17:57:00.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.497+0000 7fbf3cd31b80 20 client.25067 trim_cache size 0 max 16384 2024-04-26T17:57:00.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.531+0000 7fbf3cd31b80 1 Processor -- start 2024-04-26T17:57:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.531+0000 7fbf3cd31b80 1 -- start start 2024-04-26T17:57:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.531+0000 7fbf3cd31b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5e9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.531+0000 7fbf3cd31b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.531+0000 7fbf3cd31b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5f4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.531+0000 7fbf3cd31b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56391e5e93f0 con 0x56391e5cfde0 2024-04-26T17:57:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.531+0000 7fbf3cd31b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56391e5e9ac0 con 0x56391e5cf4c0 2024-04-26T17:57:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.531+0000 7fbf3cd31b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56391e5ea190 con 0x56391e5d9b90 2024-04-26T17:57:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.531+0000 7fbf2b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.531+0000 7fbf30a3c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5f4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.531+0000 7fbf2b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40058/0 (socket says 172.21.15.31:40058) 2024-04-26T17:57:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.531+0000 7fbf2b7fe700 1 -- 172.21.15.31:0/2902015387 learned_addr learned my addr 172.21.15.31:0/2902015387 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.531+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/2902015387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5e9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.532+0000 7fbf30a3c700 1 -- 172.21.15.31:0/2902015387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 msgr2=0x56391e5e9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.532+0000 7fbf30a3c700 1 --2- 172.21.15.31:0/2902015387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5e9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.532+0000 7fbf30a3c700 1 -- 172.21.15.31:0/2902015387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 msgr2=0x56391e5d6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.532+0000 7fbf30a3c700 1 --2- 172.21.15.31:0/2902015387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.532+0000 7fbf30a3c700 1 -- 172.21.15.31:0/2902015387 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56391e5e98e0 con 0x56391e5d9b90 2024-04-26T17:57:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.532+0000 7fbf2b7fe700 1 --2- 172.21.15.31:0/2902015387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.532+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/2902015387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5e9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:57:00.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.532+0000 7fbf30a3c700 1 --2- 172.21.15.31:0/2902015387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5f4540 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009d70 tx=0x7fbf24009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8ed80e29d9e8790 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:00.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.532+0000 7fbf11ffb700 1 -- 172.21.15.31:0/2902015387 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf24004db0 con 0x56391e5d9b90 2024-04-26T17:57:00.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.532+0000 7fbf11ffb700 1 -- 172.21.15.31:0/2902015387 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf2401b070 con 0x56391e5d9b90 2024-04-26T17:57:00.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.532+0000 7fbf11ffb700 1 -- 172.21.15.31:0/2902015387 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf24017410 con 0x56391e5d9b90 2024-04-26T17:57:00.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.533+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/2902015387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 msgr2=0x56391e5f4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:00.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.533+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/2902015387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5f4540 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009d70 tx=0x7fbf24009da0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.533+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/2902015387 shutdown_connections 2024-04-26T17:57:00.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.533+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/2902015387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5e9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.533+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/2902015387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5f4540 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.533+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/2902015387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.533+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/2902015387 >> 172.21.15.31:0/2902015387 conn(0x56391e4f3cf0 msgr2=0x56391e5c1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:00.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.533+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/2902015387 shutdown_connections 2024-04-26T17:57:00.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.533+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/2902015387 wait complete. 2024-04-26T17:57:00.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.533+0000 7fbf3cd31b80 1 -- start start 2024-04-26T17:57:00.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.534+0000 7fbf3cd31b80 1 Processor -- start 2024-04-26T17:57:00.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.534+0000 7fbf3cd31b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5d4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.534+0000 7fbf3cd31b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.534+0000 7fbf3cd31b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5d2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.534+0000 7fbf3cd31b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56391e5eaf30 con 0x56391e5cfde0 2024-04-26T17:57:00.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.534+0000 7fbf3cd31b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56391e5ea860 con 0x56391e5cf4c0 2024-04-26T17:57:00.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.534+0000 7fbf3cd31b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56391e5eb600 con 0x56391e5d9b90 2024-04-26T17:57:00.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.534+0000 7fbf3cd31b80 10 client.0 _ll_register_callbacks cb 0x56391e4ad530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:57:00.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.534+0000 7fbf2b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.534+0000 7fbf2b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40074/0 (socket says 172.21.15.31:40074) 2024-04-26T17:57:00.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.534+0000 7fbf30a3c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5d2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.534+0000 7fbf2b7fe700 1 -- 172.21.15.31:0/2578818232 learned_addr learned my addr 172.21.15.31:0/2578818232 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:00.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.534+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/2578818232 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5d4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.535+0000 7fbf30a3c700 1 -- 172.21.15.31:0/2578818232 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 msgr2=0x56391e5d4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:00.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.535+0000 7fbf30a3c700 1 --2- 172.21.15.31:0/2578818232 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5d4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.535+0000 7fbf30a3c700 1 -- 172.21.15.31:0/2578818232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 msgr2=0x56391e5d2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:00.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.535+0000 7fbf30a3c700 1 --2- 172.21.15.31:0/2578818232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.535+0000 7fbf30a3c700 1 -- 172.21.15.31:0/2578818232 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56391e5f3550 con 0x56391e5d9b90 2024-04-26T17:57:00.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.535+0000 7fbf30a3c700 1 --2- 172.21.15.31:0/2578818232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5d2f70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbf240060b0 tx=0x7fbf24005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcbd8e998ec6d54f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:00.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.535+0000 7fbf0b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T17:57:00.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.535+0000 7fbf0b7fe700 1 -- 172.21.15.31:0/2578818232 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf24004ee0 con 0x56391e5d9b90 2024-04-26T17:57:00.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.535+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/2578818232 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf2401b070 con 0x56391e5d9b90 2024-04-26T17:57:00.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.535+0000 7fbf0b7fe700 1 -- 172.21.15.31:0/2578818232 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fbf24005080 con 0x56391e5d9b90 2024-04-26T17:57:00.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.536+0000 7fbf0b7fe700 1 -- 172.21.15.31:0/2578818232 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbf24005280 con 0x56391e5d9b90 2024-04-26T17:57:00.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.536+0000 7fbf3cd31b80 10 client.25018 fetch_fsmap learned FSMap version 27 2024-04-26T17:57:00.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.536+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/2578818232 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56391e5f4830 con 0x56391e5d9b90 2024-04-26T17:57:00.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.536+0000 7fbf0b7fe700 1 -- 172.21.15.31:0/2578818232 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf2401b070 con 0x56391e5d9b90 2024-04-26T17:57:00.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.536+0000 7fbf0b7fe700 1 -- 172.21.15.31:0/2578818232 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7fbf24026020 con 0x56391e5d9b90 2024-04-26T17:57:00.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.536+0000 7fbf3cd31b80 10 client.25018 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T17:57:00.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.536+0000 7fbf3cd31b80 10 client.25018 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T17:57:00.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.536+0000 7fbf3cd31b80 20 client.25018 populate_metadata read hostname 'smithi031' 2024-04-26T17:57:00.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.536+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/2578818232 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x56391e654d70 0x56391e653a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:00.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.536+0000 7fbf3cd31b80 4 client.25018 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T17:57:00.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.536+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/2578818232 --> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] -- command(tid 0: {"prefix": "status"}) -- 0x56391e5270d0 con 0x56391e654d70 2024-04-26T17:57:00.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.536+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/2578818232 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x56391e654d70 0x56391e653a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:00.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.537+0000 7fbf0b7fe700 10 client.25018 ms_handle_connect on v2:172.21.15.186:6838/1587952500 2024-04-26T17:57:00.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.537+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/2578818232 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x56391e654d70 0x56391e653a50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6c7b4b34f32193d7 server_cookie=a8a91b9fca0754cc in_seq=0 out_seq=0 2024-04-26T17:57:00.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.537+0000 7fbf0b7fe700 1 -- 172.21.15.31:0/2578818232 <== mds.1 v2:172.21.15.186:6838/1587952500 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56391e5270d0 con 0x56391e654d70 2024-04-26T17:57:00.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.537+0000 7fbf0b7fe700 10 client.25018 handle_command_reply: tid=0 2024-04-26T17:57:00.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.537+0000 7fbf3cd31b80 1 client.25018 shutdown 2024-04-26T17:57:00.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.538+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/2578818232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 msgr2=0x56391e5d2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:00.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.538+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/2578818232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5d2f70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbf240060b0 tx=0x7fbf24005970 comp rx=0 tx=0).stop 2024-04-26T17:57:00.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.538+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/2578818232 shutdown_connections 2024-04-26T17:57:00.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.538+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/2578818232 >> [v2:172.21.15.186:6838/1587952500,v1:172.21.15.186:6839/1587952500] conn(0x56391e654d70 0x56391e653a50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.538+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/2578818232 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56391e5cf4c0 0x56391e5d4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.538+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/2578818232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56391e5d9b90 0x56391e5d2f70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.538+0000 7fbf3cd31b80 1 --2- 172.21.15.31:0/2578818232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56391e5cfde0 0x56391e5d2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.538+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/2578818232 >> 172.21.15.31:0/2578818232 conn(0x56391e4f3cf0 msgr2=0x56391e5d4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:00.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.538+0000 7fbf0b7fe700 0 client.25018 ms_handle_reset on v2:172.21.15.186:6838/1587952500 2024-04-26T17:57:00.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.538+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/2578818232 shutdown_connections 2024-04-26T17:57:00.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.538+0000 7fbf3cd31b80 1 -- 172.21.15.31:0/2578818232 wait complete. 2024-04-26T17:57:00.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.538+0000 7fbf3cd31b80 20 client.25018 trim_cache size 0 max 16384 2024-04-26T17:57:00.543 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T17:57:00.543 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "7e53e050-03f4-11ef-bc93-c7b262605968", 2024-04-26T17:57:00.543 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 1, 2024-04-26T17:57:00.543 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24457, 2024-04-26T17:57:00.543 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-26T17:57:00.543 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-26T17:57:00.543 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-26T17:57:00.543 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 221.86875669400001, 2024-04-26T17:57:00.543 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 27, 2024-04-26T17:57:00.543 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 91, 2024-04-26T17:57:00.543 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T17:57:00.543 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 260.63237448699999 2024-04-26T17:57:00.543 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T17:57:00.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.539+0000 7fbf3249f700 1 -- 172.21.15.31:0/3150427125 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fbf140a3ec0 msgr2=0x7fbf140a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.539+0000 7fbf3249f700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fbf140a3ec0 0x7fbf140a6380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c1008f0 tx=0x7fbf1c009260 comp rx=0 tx=0).stop 2024-04-26T17:57:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.539+0000 7fbf3249f700 1 -- 172.21.15.31:0/3150427125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf2c102cd0 msgr2=0x7fbf2c19fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.539+0000 7fbf3249f700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf2c102cd0 0x7fbf2c19fdf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbf20005910 tx=0x7fbf2000b670 comp rx=0 tx=0).stop 2024-04-26T17:57:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.539+0000 7fbf3249f700 1 -- 172.21.15.31:0/3150427125 shutdown_connections 2024-04-26T17:57:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.539+0000 7fbf3249f700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fbf140a3ec0 0x7fbf140a6380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.539+0000 7fbf3249f700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf2c102310 0x7fbf2c19f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.539+0000 7fbf3249f700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c108380 0x7fbf2c199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.539+0000 7fbf3249f700 1 --2- 172.21.15.31:0/3150427125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf2c102cd0 0x7fbf2c19fdf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.539+0000 7fbf3249f700 1 -- 172.21.15.31:0/3150427125 >> 172.21.15.31:0/3150427125 conn(0x7fbf2c079290 msgr2=0x7fbf2c1066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.539+0000 7fbf3249f700 1 -- 172.21.15.31:0/3150427125 shutdown_connections 2024-04-26T17:57:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:00.539+0000 7fbf3249f700 1 -- 172.21.15.31:0/3150427125 wait complete. 2024-04-26T17:57:00.939 INFO:teuthology.orchestra.run.smithi031.stderr:1100000 of 3200000 tuples (34%) done (elapsed 2.12 s, remaining 4.05 s) 2024-04-26T17:57:01.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:00 smithi097 ceph-mon[32419]: pgmap v18: 97 pgs: 97 active+clean; 2.0 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 69 KiB/s wr, 9 op/s 2024-04-26T17:57:01.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:00 smithi186 ceph-mon[32797]: pgmap v18: 97 pgs: 97 active+clean; 2.0 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 69 KiB/s wr, 9 op/s 2024-04-26T17:57:01.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:00 smithi031 ceph-mon[29164]: pgmap v18: 97 pgs: 97 active+clean; 2.0 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 69 KiB/s wr, 9 op/s 2024-04-26T17:57:01.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T17:57:01.651 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:57:03.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:02 smithi031 ceph-mon[29164]: pgmap v19: 97 pgs: 97 active+clean; 161 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 22 MiB/s wr, 150 op/s 2024-04-26T17:57:03.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:02 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:57:03.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:02 smithi031 ceph-mon[29164]: mgrmap e25: x(active, since 32s), standbys: z, y 2024-04-26T17:57:03.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:02 smithi097 ceph-mon[32419]: pgmap v19: 97 pgs: 97 active+clean; 161 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 22 MiB/s wr, 150 op/s 2024-04-26T17:57:03.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:02 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:57:03.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:02 smithi097 ceph-mon[32419]: mgrmap e25: x(active, since 32s), standbys: z, y 2024-04-26T17:57:03.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:02 smithi186 ceph-mon[32797]: pgmap v19: 97 pgs: 97 active+clean; 161 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 22 MiB/s wr, 150 op/s 2024-04-26T17:57:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:02 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:57:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:02 smithi186 ceph-mon[32797]: mgrmap e25: x(active, since 32s), standbys: z, y 2024-04-26T17:57:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.422+0000 7fee98312700 1 -- 172.21.15.31:0/3360954011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee901094c0 msgr2=0x7fee901098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.422+0000 7fee98312700 1 --2- 172.21.15.31:0/3360954011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee901094c0 0x7fee901098a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fee8c009990 tx=0x7fee8c009ca0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.422+0000 7fee98312700 1 -- 172.21.15.31:0/3360954011 shutdown_connections 2024-04-26T17:57:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.422+0000 7fee98312700 1 --2- 172.21.15.31:0/3360954011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee90103e30 0x7fee90110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.422+0000 7fee98312700 1 --2- 172.21.15.31:0/3360954011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee901094c0 0x7fee901098a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.422+0000 7fee98312700 1 --2- 172.21.15.31:0/3360954011 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee90103470 0x7fee901038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.422+0000 7fee98312700 1 -- 172.21.15.31:0/3360954011 >> 172.21.15.31:0/3360954011 conn(0x7fee9007a580 msgr2=0x7fee9007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.422+0000 7fee98312700 1 -- 172.21.15.31:0/3360954011 shutdown_connections 2024-04-26T17:57:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.422+0000 7fee98312700 1 -- 172.21.15.31:0/3360954011 wait complete. 2024-04-26T17:57:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.423+0000 7fee98312700 1 Processor -- start 2024-04-26T17:57:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.423+0000 7fee98312700 1 -- start start 2024-04-26T17:57:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.423+0000 7fee98312700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee90103470 0x7fee90075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.423+0000 7fee98312700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee90103e30 0x7fee90075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.423+0000 7fee98312700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee901094c0 0x7fee900761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.423+0000 7fee98312700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fee90113640 con 0x7fee90103e30 2024-04-26T17:57:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.423+0000 7fee98312700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fee901134c0 con 0x7fee90103470 2024-04-26T17:57:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.423+0000 7fee98312700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fee901137c0 con 0x7fee901094c0 2024-04-26T17:57:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.423+0000 7fee958ad700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee90103e30 0x7fee90075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.423+0000 7fee960ae700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee90103470 0x7fee90075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.423+0000 7fee960ae700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee90103470 0x7fee90075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:57272/0 (socket says 172.21.15.31:57272) 2024-04-26T17:57:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.423+0000 7fee960ae700 1 -- 172.21.15.31:0/589825927 learned_addr learned my addr 172.21.15.31:0/589825927 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.423+0000 7fee968af700 1 --2- 172.21.15.31:0/589825927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee901094c0 0x7fee900761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.424+0000 7fee958ad700 1 -- 172.21.15.31:0/589825927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee901094c0 msgr2=0x7fee900761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.424+0000 7fee958ad700 1 --2- 172.21.15.31:0/589825927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee901094c0 0x7fee900761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.424+0000 7fee958ad700 1 -- 172.21.15.31:0/589825927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee90103470 msgr2=0x7fee90075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.424+0000 7fee958ad700 1 --2- 172.21.15.31:0/589825927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee90103470 0x7fee90075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.424+0000 7fee958ad700 1 -- 172.21.15.31:0/589825927 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee901a8080 con 0x7fee90103e30 2024-04-26T17:57:03.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.424+0000 7fee960ae700 1 --2- 172.21.15.31:0/589825927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee90103470 0x7fee90075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:03.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.424+0000 7fee958ad700 1 --2- 172.21.15.31:0/589825927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee90103e30 0x7fee90075c60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fee8000e9f0 tx=0x7fee8000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:03.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.424+0000 7fee877fe700 1 -- 172.21.15.31:0/589825927 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8000cbd0 con 0x7fee90103e30 2024-04-26T17:57:03.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.424+0000 7fee877fe700 1 -- 172.21.15.31:0/589825927 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee8000cd70 con 0x7fee90103e30 2024-04-26T17:57:03.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.424+0000 7fee98312700 1 -- 172.21.15.31:0/589825927 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee901a8370 con 0x7fee90103e30 2024-04-26T17:57:03.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.424+0000 7fee877fe700 1 -- 172.21.15.31:0/589825927 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee80018680 con 0x7fee90103e30 2024-04-26T17:57:03.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.424+0000 7fee98312700 1 -- 172.21.15.31:0/589825927 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee901a8860 con 0x7fee90103e30 2024-04-26T17:57:03.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.427+0000 7fee877fe700 1 -- 172.21.15.31:0/589825927 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fee80010430 con 0x7fee90103e30 2024-04-26T17:57:03.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.427+0000 7fee877fe700 1 --2- 172.21.15.31:0/589825927 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fee7c0a3ed0 0x7fee7c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.427+0000 7fee960ae700 1 --2- 172.21.15.31:0/589825927 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fee7c0a3ed0 0x7fee7c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.427+0000 7fee877fe700 1 -- 172.21.15.31:0/589825927 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fee800d0c60 con 0x7fee90103e30 2024-04-26T17:57:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.428+0000 7feea2ba4b80 1 Processor -- start 2024-04-26T17:57:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.428+0000 7fee960ae700 1 --2- 172.21.15.31:0/589825927 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fee7c0a3ed0 0x7fee7c0a6390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee8c006010 tx=0x7fee8c01a040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.428+0000 7feea2ba4b80 1 -- start start 2024-04-26T17:57:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.428+0000 7feea2ba4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903e2ad0 0x5557903e2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.428+0000 7feea2ba4b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557903e33f0 0x5557903dccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.428+0000 7feea2ba4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903dd2d0 0x5557903dd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.428+0000 7feea2ba4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55579032f6f0 con 0x5557903dd2d0 2024-04-26T17:57:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.428+0000 7feea2ba4b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5557902ce590 con 0x5557903e33f0 2024-04-26T17:57:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.428+0000 7feea2ba4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55579032b6b0 con 0x5557903e2ad0 2024-04-26T17:57:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee968af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903dd2d0 0x5557903dd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee958ad700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557903e33f0 0x5557903dccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee960ae700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903e2ad0 0x5557903e2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee968af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903dd2d0 0x5557903dd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40102/0 (socket says 172.21.15.31:40102) 2024-04-26T17:57:03.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee968af700 1 -- 172.21.15.31:0/4142175901 learned_addr learned my addr 172.21.15.31:0/4142175901 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee968af700 1 -- 172.21.15.31:0/4142175901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903e2ad0 msgr2=0x5557903e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee968af700 1 --2- 172.21.15.31:0/4142175901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903e2ad0 0x5557903e2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee968af700 1 -- 172.21.15.31:0/4142175901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557903e33f0 msgr2=0x5557903dccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee968af700 1 --2- 172.21.15.31:0/4142175901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557903e33f0 0x5557903dccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee968af700 1 -- 172.21.15.31:0/4142175901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55579027c880 con 0x5557903dd2d0 2024-04-26T17:57:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee960ae700 1 --2- 172.21.15.31:0/4142175901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903e2ad0 0x5557903e2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee958ad700 1 --2- 172.21.15.31:0/4142175901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557903e33f0 0x5557903dccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee968af700 1 --2- 172.21.15.31:0/4142175901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903dd2d0 0x5557903dd710 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fee88009440 tx=0x7fee88009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da798c735a25b5e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee84d70700 1 -- 172.21.15.31:0/4142175901 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee88013810 con 0x5557903dd2d0 2024-04-26T17:57:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee84d70700 1 -- 172.21.15.31:0/4142175901 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee88015440 con 0x5557903dd2d0 2024-04-26T17:57:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.429+0000 7fee84d70700 1 -- 172.21.15.31:0/4142175901 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee88013e60 con 0x5557903dd2d0 2024-04-26T17:57:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.430+0000 7feea2ba4b80 1 -- 172.21.15.31:0/4142175901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903dd2d0 msgr2=0x5557903dd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.430+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/4142175901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903dd2d0 0x5557903dd710 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fee88009440 tx=0x7fee88009920 comp rx=0 tx=0).stop 2024-04-26T17:57:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.430+0000 7feea2ba4b80 1 -- 172.21.15.31:0/4142175901 shutdown_connections 2024-04-26T17:57:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.430+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/4142175901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903dd2d0 0x5557903dd710 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.430+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/4142175901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903e2ad0 0x5557903e2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.430+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/4142175901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557903e33f0 0x5557903dccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.430+0000 7feea2ba4b80 1 -- 172.21.15.31:0/4142175901 >> 172.21.15.31:0/4142175901 conn(0x555790309cf0 msgr2=0x5557903d81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.430+0000 7feea2ba4b80 1 -- 172.21.15.31:0/4142175901 shutdown_connections 2024-04-26T17:57:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.430+0000 7feea2ba4b80 1 -- 172.21.15.31:0/4142175901 wait complete. 2024-04-26T17:57:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7feea2ba4b80 1 -- start start 2024-04-26T17:57:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7feea2ba4b80 1 Processor -- start 2024-04-26T17:57:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7feea2ba4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 0x5557903e90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7feea2ba4b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557903e2ad0 0x5557903e63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7feea2ba4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e33f0 0x5557903e6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7feea2ba4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5557903e9f10 con 0x5557903e33f0 2024-04-26T17:57:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7feea2ba4b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x555790295900 con 0x5557903e2ad0 2024-04-26T17:57:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7feea2ba4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5557903ea090 con 0x5557903dd2d0 2024-04-26T17:57:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7fee968af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e33f0 0x5557903e6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7feea2ba4b80 10 client.0 _ll_register_callbacks cb 0x5557902c3530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:57:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7fee960ae700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 0x5557903e90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7fee960ae700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 0x5557903e90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:41208/0 (socket says 172.21.15.31:41208) 2024-04-26T17:57:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7fee968af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e33f0 0x5557903e6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40106/0 (socket says 172.21.15.31:40106) 2024-04-26T17:57:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7fee960ae700 1 -- 172.21.15.31:0/384574281 learned_addr learned my addr 172.21.15.31:0/384574281 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.431+0000 7fee958ad700 1 --2- 172.21.15.31:0/384574281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557903e2ad0 0x5557903e63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.432+0000 7fee968af700 1 -- 172.21.15.31:0/384574281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 msgr2=0x5557903e90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.432+0000 7fee968af700 1 --2- 172.21.15.31:0/384574281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 0x5557903e90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.432+0000 7fee968af700 1 -- 172.21.15.31:0/384574281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557903e2ad0 msgr2=0x5557903e63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.432+0000 7fee968af700 1 --2- 172.21.15.31:0/384574281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557903e2ad0 0x5557903e63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.432+0000 7fee968af700 1 -- 172.21.15.31:0/384574281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557903e6f70 con 0x5557903e33f0 2024-04-26T17:57:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.432+0000 7fee968af700 1 --2- 172.21.15.31:0/384574281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e33f0 0x5557903e6930 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fee88015db0 tx=0x7fee88013840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ccf0e97085344ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.432+0000 7fee79ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:57:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.432+0000 7fee79ffb700 1 -- 172.21.15.31:0/384574281 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee88015b60 con 0x5557903e33f0 2024-04-26T17:57:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.432+0000 7fee79ffb700 1 -- 172.21.15.31:0/384574281 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fee88003a00 con 0x5557903e33f0 2024-04-26T17:57:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.432+0000 7feea2ba4b80 1 -- 172.21.15.31:0/384574281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee88015440 con 0x5557903e33f0 2024-04-26T17:57:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.432+0000 7fee79ffb700 1 -- 172.21.15.31:0/384574281 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fee8802b020 con 0x5557903e33f0 2024-04-26T17:57:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.432+0000 7feea2ba4b80 10 client.15360 fetch_fsmap learned FSMap version 28 2024-04-26T17:57:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.432+0000 7feea2ba4b80 1 -- 172.21.15.31:0/384574281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5557903f4910 con 0x5557903e33f0 2024-04-26T17:57:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.432+0000 7fee79ffb700 1 -- 172.21.15.31:0/384574281 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee88015440 con 0x5557903e33f0 2024-04-26T17:57:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.433+0000 7fee79ffb700 1 -- 172.21.15.31:0/384574281 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fee880053d0 con 0x5557903e33f0 2024-04-26T17:57:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.433+0000 7feea2ba4b80 10 client.15360 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T17:57:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.433+0000 7feea2ba4b80 10 client.15360 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T17:57:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.433+0000 7feea2ba4b80 20 client.15360 populate_metadata read hostname 'smithi031' 2024-04-26T17:57:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.433+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/384574281 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5557903f6b00 0x5557903f78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.433+0000 7feea2ba4b80 4 client.15360 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:57:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.433+0000 7feea2ba4b80 1 -- 172.21.15.31:0/384574281 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55579033d0d0 con 0x5557903f6b00 2024-04-26T17:57:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.433+0000 7fee960ae700 1 --2- 172.21.15.31:0/384574281 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5557903f6b00 0x5557903f78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.433+0000 7fee960ae700 1 --2- 172.21.15.31:0/384574281 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5557903f6b00 0x5557903f78a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8949beec6c2d88d server_cookie=9f07f0c70886af6 in_seq=0 out_seq=0 2024-04-26T17:57:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.433+0000 7fee79ffb700 10 client.15360 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7fee79ffb700 1 -- 172.21.15.31:0/384574281 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55579033d0d0 con 0x5557903f6b00 2024-04-26T17:57:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7fee79ffb700 10 client.15360 handle_command_reply: tid=0 2024-04-26T17:57:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7feea2ba4b80 1 client.15360 shutdown 2024-04-26T17:57:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7feea2ba4b80 1 -- 172.21.15.31:0/384574281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e33f0 msgr2=0x5557903e6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/384574281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e33f0 0x5557903e6930 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fee88015db0 tx=0x7fee88013840 comp rx=0 tx=0).stop 2024-04-26T17:57:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7feea2ba4b80 1 -- 172.21.15.31:0/384574281 shutdown_connections 2024-04-26T17:57:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/384574281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e33f0 0x5557903e6930 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/384574281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 0x5557903e90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/384574281 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5557903f6b00 0x5557903f78a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/384574281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557903e2ad0 0x5557903e63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7feea2ba4b80 1 -- 172.21.15.31:0/384574281 >> 172.21.15.31:0/384574281 conn(0x555790309cf0 msgr2=0x5557903d7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7fee79ffb700 0 client.15360 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7feea2ba4b80 1 -- 172.21.15.31:0/384574281 shutdown_connections 2024-04-26T17:57:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7feea2ba4b80 1 -- 172.21.15.31:0/384574281 wait complete. 2024-04-26T17:57:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.434+0000 7feea2ba4b80 20 client.15360 trim_cache size 0 max 16384 2024-04-26T17:57:03.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.455+0000 7feea2ba4b80 1 Processor -- start 2024-04-26T17:57:03.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.455+0000 7feea2ba4b80 1 -- start start 2024-04-26T17:57:03.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.455+0000 7feea2ba4b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555790309cf0 0x5557903dccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.455+0000 7feea2ba4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 0x5557903e3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.455+0000 7feea2ba4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e2ad0 0x5557903d7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.455+0000 7feea2ba4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5557903f62b0 con 0x5557903e2ad0 2024-04-26T17:57:03.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.455+0000 7feea2ba4b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5557903f6980 con 0x555790309cf0 2024-04-26T17:57:03.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.455+0000 7feea2ba4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5557903f7050 con 0x5557903dd2d0 2024-04-26T17:57:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.455+0000 7fee968af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e2ad0 0x5557903d7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.455+0000 7fee960ae700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555790309cf0 0x5557903dccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.455+0000 7fee968af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e2ad0 0x5557903d7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40114/0 (socket says 172.21.15.31:40114) 2024-04-26T17:57:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.455+0000 7fee968af700 1 -- 172.21.15.31:0/1738173169 learned_addr learned my addr 172.21.15.31:0/1738173169 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.456+0000 7fee958ad700 1 --2- 172.21.15.31:0/1738173169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 0x5557903e3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.456+0000 7fee968af700 1 -- 172.21.15.31:0/1738173169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 msgr2=0x5557903e3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.456+0000 7fee968af700 1 --2- 172.21.15.31:0/1738173169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 0x5557903e3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.456+0000 7fee968af700 1 -- 172.21.15.31:0/1738173169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555790309cf0 msgr2=0x5557903dccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.456+0000 7fee968af700 1 --2- 172.21.15.31:0/1738173169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555790309cf0 0x5557903dccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.456+0000 7fee968af700 1 -- 172.21.15.31:0/1738173169 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557903e9a80 con 0x5557903e2ad0 2024-04-26T17:57:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.456+0000 7fee960ae700 1 --2- 172.21.15.31:0/1738173169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555790309cf0 0x5557903dccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.456+0000 7fee968af700 1 --2- 172.21.15.31:0/1738173169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e2ad0 0x5557903d7330 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fee88007b40 tx=0x7fee88009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fdd3c896c02c28 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.456+0000 7fee7bfff700 1 -- 172.21.15.31:0/1738173169 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee88005a10 con 0x5557903e2ad0 2024-04-26T17:57:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.456+0000 7fee7bfff700 1 -- 172.21.15.31:0/1738173169 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee88015a20 con 0x5557903e2ad0 2024-04-26T17:57:03.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.456+0000 7fee7bfff700 1 -- 172.21.15.31:0/1738173169 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee88013c40 con 0x5557903e2ad0 2024-04-26T17:57:03.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.457+0000 7feea2ba4b80 1 -- 172.21.15.31:0/1738173169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e2ad0 msgr2=0x5557903d7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:03.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.457+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/1738173169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e2ad0 0x5557903d7330 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fee88007b40 tx=0x7fee88009ce0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.457+0000 7feea2ba4b80 1 -- 172.21.15.31:0/1738173169 shutdown_connections 2024-04-26T17:57:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.457+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/1738173169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555790309cf0 0x5557903dccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.457+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/1738173169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e2ad0 0x5557903d7330 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.457+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/1738173169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 0x5557903e3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.457+0000 7feea2ba4b80 1 -- 172.21.15.31:0/1738173169 >> 172.21.15.31:0/1738173169 conn(0x5557903dda20 msgr2=0x5557903e37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.457+0000 7feea2ba4b80 1 -- 172.21.15.31:0/1738173169 shutdown_connections 2024-04-26T17:57:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.457+0000 7feea2ba4b80 1 -- 172.21.15.31:0/1738173169 wait complete. 2024-04-26T17:57:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.457+0000 7feea2ba4b80 1 -- start start 2024-04-26T17:57:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.457+0000 7feea2ba4b80 1 Processor -- start 2024-04-26T17:57:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7feea2ba4b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555790309cf0 0x555790401960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7feea2ba4b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 0x5557903fc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7feea2ba4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e2ad0 0x5557903fcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7feea2ba4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5557903f7df0 con 0x5557903e2ad0 2024-04-26T17:57:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7feea2ba4b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5557903f7720 con 0x555790309cf0 2024-04-26T17:57:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7feea2ba4b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5557903f84f0 con 0x5557903dd2d0 2024-04-26T17:57:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7feea2ba4b80 10 client.0 _ll_register_callbacks cb 0x5557902c3530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:57:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7fee960ae700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555790309cf0 0x555790401960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7fee960ae700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555790309cf0 0x555790401960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:57316/0 (socket says 172.21.15.31:57316) 2024-04-26T17:57:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7fee960ae700 1 -- 172.21.15.31:0/1893643140 learned_addr learned my addr 172.21.15.31:0/1893643140 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7fee958ad700 1 --2- 172.21.15.31:0/1893643140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 0x5557903fc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7fee968af700 1 --2- 172.21.15.31:0/1893643140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e2ad0 0x5557903fcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7fee960ae700 1 -- 172.21.15.31:0/1893643140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 msgr2=0x5557903fc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7fee960ae700 1 --2- 172.21.15.31:0/1893643140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 0x5557903fc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7fee960ae700 1 -- 172.21.15.31:0/1893643140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e2ad0 msgr2=0x5557903fcb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7fee960ae700 1 --2- 172.21.15.31:0/1893643140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e2ad0 0x5557903fcb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7fee960ae700 1 -- 172.21.15.31:0/1893643140 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55579045dc90 con 0x555790309cf0 2024-04-26T17:57:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7fee960ae700 1 --2- 172.21.15.31:0/1893643140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555790309cf0 0x555790401960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fee8c000c00 tx=0x7fee8c009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26d23a31cc8cd1ed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.458+0000 7fee79ffb700 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T17:57:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7fee79ffb700 1 -- 172.21.15.31:0/1893643140 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8c004c80 con 0x555790309cf0 2024-04-26T17:57:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7feea2ba4b80 1 -- 172.21.15.31:0/1893643140 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee88015a20 con 0x555790309cf0 2024-04-26T17:57:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7fee79ffb700 1 -- 172.21.15.31:0/1893643140 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fee8c004e20 con 0x555790309cf0 2024-04-26T17:57:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7fee79ffb700 1 -- 172.21.15.31:0/1893643140 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fee8c034400 con 0x555790309cf0 2024-04-26T17:57:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7feea2ba4b80 10 client.25091 fetch_fsmap learned FSMap version 28 2024-04-26T17:57:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7feea2ba4b80 1 -- 172.21.15.31:0/1893643140 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55579045ede0 con 0x555790309cf0 2024-04-26T17:57:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7fee79ffb700 1 -- 172.21.15.31:0/1893643140 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee88015a20 con 0x555790309cf0 2024-04-26T17:57:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7fee79ffb700 1 -- 172.21.15.31:0/1893643140 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fee8c046b80 con 0x555790309cf0 2024-04-26T17:57:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7feea2ba4b80 10 client.25091 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T17:57:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7feea2ba4b80 10 client.25091 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T17:57:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7feea2ba4b80 20 client.25091 populate_metadata read hostname 'smithi031' 2024-04-26T17:57:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/1893643140 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5557903e33f0 0x555790461fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7feea2ba4b80 4 client.25091 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T17:57:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7feea2ba4b80 1 -- 172.21.15.31:0/1893643140 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "status"}) -- 0x555790337170 con 0x5557903e33f0 2024-04-26T17:57:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.459+0000 7fee958ad700 1 --2- 172.21.15.31:0/1893643140 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5557903e33f0 0x555790461fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7fee958ad700 1 --2- 172.21.15.31:0/1893643140 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5557903e33f0 0x555790461fe0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d556931ef86c742 server_cookie=bc06108af43ab73c in_seq=0 out_seq=0 2024-04-26T17:57:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7fee79ffb700 10 client.25091 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7fee79ffb700 1 -- 172.21.15.31:0/1893643140 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x555790337170 con 0x5557903e33f0 2024-04-26T17:57:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7fee79ffb700 10 client.25091 handle_command_reply: tid=0 2024-04-26T17:57:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7feea2ba4b80 1 client.25091 shutdown 2024-04-26T17:57:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7feea2ba4b80 1 -- 172.21.15.31:0/1893643140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555790309cf0 msgr2=0x555790401960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/1893643140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555790309cf0 0x555790401960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fee8c000c00 tx=0x7fee8c009310 comp rx=0 tx=0).stop 2024-04-26T17:57:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7feea2ba4b80 1 -- 172.21.15.31:0/1893643140 shutdown_connections 2024-04-26T17:57:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/1893643140 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5557903e33f0 0x555790461fe0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/1893643140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555790309cf0 0x555790401960 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/1893643140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5557903e2ad0 0x5557903fcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7fee79ffb700 0 client.25091 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7feea2ba4b80 1 --2- 172.21.15.31:0/1893643140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557903dd2d0 0x5557903fc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7feea2ba4b80 1 -- 172.21.15.31:0/1893643140 >> 172.21.15.31:0/1893643140 conn(0x5557903dda20 msgr2=0x5557903e7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7feea2ba4b80 1 -- 172.21.15.31:0/1893643140 shutdown_connections 2024-04-26T17:57:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7feea2ba4b80 1 -- 172.21.15.31:0/1893643140 wait complete. 2024-04-26T17:57:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.460+0000 7feea2ba4b80 20 client.25091 trim_cache size 0 max 16384 2024-04-26T17:57:03.466 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T17:57:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "7e53e050-03f4-11ef-bc93-c7b262605968", 2024-04-26T17:57:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-04-26T17:57:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24473, 2024-04-26T17:57:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-26T17:57:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-26T17:57:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-26T17:57:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 237.84811357300001, 2024-04-26T17:57:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 28, 2024-04-26T17:57:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 91, 2024-04-26T17:57:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T17:57:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 252.71181789299999 2024-04-26T17:57:03.467 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T17:57:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.461+0000 7fee98312700 1 -- 172.21.15.31:0/589825927 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fee7c0a3ed0 msgr2=0x7fee7c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.461+0000 7fee98312700 1 --2- 172.21.15.31:0/589825927 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fee7c0a3ed0 0x7fee7c0a6390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee8c006010 tx=0x7fee8c01a040 comp rx=0 tx=0).stop 2024-04-26T17:57:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.461+0000 7fee98312700 1 -- 172.21.15.31:0/589825927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee90103e30 msgr2=0x7fee90075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.461+0000 7fee98312700 1 --2- 172.21.15.31:0/589825927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee90103e30 0x7fee90075c60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fee8000e9f0 tx=0x7fee8000ed00 comp rx=0 tx=0).stop 2024-04-26T17:57:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.461+0000 7fee98312700 1 -- 172.21.15.31:0/589825927 shutdown_connections 2024-04-26T17:57:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.461+0000 7fee98312700 1 --2- 172.21.15.31:0/589825927 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fee7c0a3ed0 0x7fee7c0a6390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.461+0000 7fee98312700 1 --2- 172.21.15.31:0/589825927 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee90103470 0x7fee90075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.461+0000 7fee98312700 1 --2- 172.21.15.31:0/589825927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee901094c0 0x7fee900761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.461+0000 7fee98312700 1 --2- 172.21.15.31:0/589825927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee90103e30 0x7fee90075c60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.461+0000 7fee98312700 1 -- 172.21.15.31:0/589825927 >> 172.21.15.31:0/589825927 conn(0x7fee9007a580 msgr2=0x7fee901021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.461+0000 7fee98312700 1 -- 172.21.15.31:0/589825927 shutdown_connections 2024-04-26T17:57:03.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:03.461+0000 7fee98312700 1 -- 172.21.15.31:0/589825927 wait complete. 2024-04-26T17:57:04.656 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-26T17:57:05.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:04 smithi031 ceph-mon[29164]: pgmap v20: 97 pgs: 97 active+clean; 161 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 22 MiB/s wr, 147 op/s 2024-04-26T17:57:05.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:04 smithi097 ceph-mon[32419]: pgmap v20: 97 pgs: 97 active+clean; 161 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 22 MiB/s wr, 147 op/s 2024-04-26T17:57:05.182 INFO:teuthology.orchestra.run.smithi031.stderr:1200000 of 3200000 tuples (37%) done (elapsed 6.37 s, remaining 10.61 s) 2024-04-26T17:57:05.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:04 smithi186 ceph-mon[32797]: pgmap v20: 97 pgs: 97 active+clean; 161 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 22 MiB/s wr, 147 op/s 2024-04-26T17:57:05.572 INFO:teuthology.orchestra.run.smithi031.stderr:1300000 of 3200000 tuples (40%) done (elapsed 6.76 s, remaining 9.87 s) 2024-04-26T17:57:05.774 INFO:teuthology.orchestra.run.smithi031.stderr:1400000 of 3200000 tuples (43%) done (elapsed 6.96 s, remaining 8.95 s) 2024-04-26T17:57:05.838 INFO:teuthology.orchestra.run.smithi031.stderr:1500000 of 3200000 tuples (46%) done (elapsed 7.02 s, remaining 7.96 s) 2024-04-26T17:57:06.012 INFO:teuthology.orchestra.run.smithi031.stderr:1600000 of 3200000 tuples (50%) done (elapsed 7.20 s, remaining 7.20 s) 2024-04-26T17:57:06.115 INFO:teuthology.orchestra.run.smithi031.stderr:1700000 of 3200000 tuples (53%) done (elapsed 7.30 s, remaining 6.44 s) 2024-04-26T17:57:06.261 INFO:teuthology.orchestra.run.smithi031.stderr:1800000 of 3200000 tuples (56%) done (elapsed 7.44 s, remaining 5.79 s) 2024-04-26T17:57:06.434 INFO:teuthology.orchestra.run.smithi031.stderr:1900000 of 3200000 tuples (59%) done (elapsed 7.62 s, remaining 5.21 s) 2024-04-26T17:57:06.608 INFO:teuthology.orchestra.run.smithi031.stderr:2000000 of 3200000 tuples (62%) done (elapsed 7.79 s, remaining 4.67 s) 2024-04-26T17:57:06.675 INFO:teuthology.orchestra.run.smithi031.stderr:2100000 of 3200000 tuples (65%) done (elapsed 7.86 s, remaining 4.12 s) 2024-04-26T17:57:06.858 INFO:teuthology.orchestra.run.smithi031.stderr:2200000 of 3200000 tuples (68%) done (elapsed 8.04 s, remaining 3.65 s) 2024-04-26T17:57:07.037 INFO:teuthology.orchestra.run.smithi031.stderr:2300000 of 3200000 tuples (71%) done (elapsed 8.22 s, remaining 3.22 s) 2024-04-26T17:57:07.117 INFO:teuthology.orchestra.run.smithi031.stderr:2400000 of 3200000 tuples (75%) done (elapsed 8.30 s, remaining 2.77 s) 2024-04-26T17:57:07.133 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:06 smithi097 ceph-mon[32419]: pgmap v21: 97 pgs: 97 active+clean; 215 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 29 MiB/s wr, 385 op/s 2024-04-26T17:57:07.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:06 smithi186 ceph-mon[32797]: pgmap v21: 97 pgs: 97 active+clean; 215 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 29 MiB/s wr, 385 op/s 2024-04-26T17:57:07.291 INFO:teuthology.orchestra.run.smithi031.stderr:2500000 of 3200000 tuples (78%) done (elapsed 8.47 s, remaining 2.37 s) 2024-04-26T17:57:07.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:06 smithi031 ceph-mon[29164]: pgmap v21: 97 pgs: 97 active+clean; 215 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 29 MiB/s wr, 385 op/s 2024-04-26T17:57:07.353 INFO:teuthology.orchestra.run.smithi031.stderr:2600000 of 3200000 tuples (81%) done (elapsed 8.54 s, remaining 1.97 s) 2024-04-26T17:57:07.520 INFO:teuthology.orchestra.run.smithi031.stderr:2700000 of 3200000 tuples (84%) done (elapsed 8.70 s, remaining 1.61 s) 2024-04-26T17:57:07.676 INFO:teuthology.orchestra.run.smithi031.stderr:2800000 of 3200000 tuples (87%) done (elapsed 8.86 s, remaining 1.27 s) 2024-04-26T17:57:07.756 INFO:teuthology.orchestra.run.smithi031.stderr:2900000 of 3200000 tuples (90%) done (elapsed 8.94 s, remaining 0.92 s) 2024-04-26T17:57:07.904 INFO:teuthology.orchestra.run.smithi031.stderr:3000000 of 3200000 tuples (93%) done (elapsed 9.09 s, remaining 0.61 s) 2024-04-26T17:57:07.991 INFO:teuthology.orchestra.run.smithi031.stderr:3100000 of 3200000 tuples (96%) done (elapsed 9.17 s, remaining 0.30 s) 2024-04-26T17:57:08.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:07 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:57:08.171 INFO:teuthology.orchestra.run.smithi031.stderr:3200000 of 3200000 tuples (100%) done (elapsed 9.35 s, remaining 0.00 s) 2024-04-26T17:57:08.210 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:07 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:57:08.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:07 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:57:09.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:08 smithi097 ceph-mon[32419]: pgmap v22: 97 pgs: 97 active+clean; 219 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 29 MiB/s wr, 463 op/s 2024-04-26T17:57:09.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:08 smithi186 ceph-mon[32797]: pgmap v22: 97 pgs: 97 active+clean; 219 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 29 MiB/s wr, 463 op/s 2024-04-26T17:57:09.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:08 smithi031 ceph-mon[29164]: pgmap v22: 97 pgs: 97 active+clean; 219 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 29 MiB/s wr, 463 op/s 2024-04-26T17:57:11.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:10 smithi186 ceph-mon[32797]: pgmap v23: 97 pgs: 97 active+clean; 219 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 29 MiB/s wr, 463 op/s 2024-04-26T17:57:11.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:10 smithi031 ceph-mon[29164]: pgmap v23: 97 pgs: 97 active+clean; 219 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 29 MiB/s wr, 463 op/s 2024-04-26T17:57:11.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:10 smithi097 ceph-mon[32419]: pgmap v23: 97 pgs: 97 active+clean; 219 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 29 MiB/s wr, 463 op/s 2024-04-26T17:57:11.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.642+0000 7fa384be4640 1 -- 172.21.15.31:0/746101259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3801032f0 msgr2=0x7fa38010f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:11.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.642+0000 7fa384be4640 1 --2- 172.21.15.31:0/746101259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3801032f0 0x7fa38010f800 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa378009cd0 tx=0x7fa37802a8e0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.642+0000 7fa384be4640 1 -- 172.21.15.31:0/746101259 shutdown_connections 2024-04-26T17:57:11.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.642+0000 7fa384be4640 1 --2- 172.21.15.31:0/746101259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3801032f0 0x7fa38010f800 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.642+0000 7fa384be4640 1 --2- 172.21.15.31:0/746101259 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa380102930 0x7fa380102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.642+0000 7fa384be4640 1 --2- 172.21.15.31:0/746101259 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa380108940 0x7fa380108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.642+0000 7fa384be4640 1 -- 172.21.15.31:0/746101259 >> 172.21.15.31:0/746101259 conn(0x7fa3800fe400 msgr2=0x7fa380100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.642+0000 7fa384be4640 1 -- 172.21.15.31:0/746101259 shutdown_connections 2024-04-26T17:57:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.642+0000 7fa384be4640 1 -- 172.21.15.31:0/746101259 wait complete. 2024-04-26T17:57:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa384be4640 1 Processor -- start 2024-04-26T17:57:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa384be4640 1 -- start start 2024-04-26T17:57:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa384be4640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa380102930 0x7fa3801990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa384be4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3801032f0 0x7fa3801995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa384be4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa380108940 0x7fa38019d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa384be4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa380112c00 con 0x7fa380108940 2024-04-26T17:57:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa384be4640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa380112a80 con 0x7fa380102930 2024-04-26T17:57:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa384be4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa380112d80 con 0x7fa3801032f0 2024-04-26T17:57:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa37ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa380108940 0x7fa38019d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa37ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa380108940 0x7fa38019d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34506/0 (socket says 172.21.15.31:34506) 2024-04-26T17:57:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa37ed9d640 1 -- 172.21.15.31:0/947490441 learned_addr learned my addr 172.21.15.31:0/947490441 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa37e59c640 1 --2- 172.21.15.31:0/947490441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa380102930 0x7fa3801990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa37dd9b640 1 --2- 172.21.15.31:0/947490441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3801032f0 0x7fa3801995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa37ed9d640 1 -- 172.21.15.31:0/947490441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3801032f0 msgr2=0x7fa3801995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa37ed9d640 1 --2- 172.21.15.31:0/947490441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3801032f0 0x7fa3801995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa37ed9d640 1 -- 172.21.15.31:0/947490441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa380102930 msgr2=0x7fa3801990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa37ed9d640 1 --2- 172.21.15.31:0/947490441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa380102930 0x7fa3801990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.643+0000 7fa37ed9d640 1 -- 172.21.15.31:0/947490441 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa38019e040 con 0x7fa380108940 2024-04-26T17:57:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.644+0000 7fa37ed9d640 1 --2- 172.21.15.31:0/947490441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa380108940 0x7fa38019d910 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa378033040 tx=0x7fa378009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.644+0000 7fa36f7fe640 1 -- 172.21.15.31:0/947490441 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa378004040 con 0x7fa380108940 2024-04-26T17:57:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.644+0000 7fa36f7fe640 1 -- 172.21.15.31:0/947490441 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa378004690 con 0x7fa380108940 2024-04-26T17:57:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.644+0000 7fa384be4640 1 -- 172.21.15.31:0/947490441 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa38019e2d0 con 0x7fa380108940 2024-04-26T17:57:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.644+0000 7fa36f7fe640 1 -- 172.21.15.31:0/947490441 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa378030d90 con 0x7fa380108940 2024-04-26T17:57:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.644+0000 7fa384be4640 1 -- 172.21.15.31:0/947490441 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3801a5bc0 con 0x7fa380108940 2024-04-26T17:57:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.646+0000 7fa36f7fe640 1 -- 172.21.15.31:0/947490441 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fa3780041e0 con 0x7fa380108940 2024-04-26T17:57:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa3879d1740 1 Processor -- start 2024-04-26T17:57:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa3879d1740 1 -- start start 2024-04-26T17:57:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa3879d1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68d9df00 0x555b68d9e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa3879d1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9e820 0x555b68da8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa3879d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68da90e0 0x555b68dab4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa3879d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555b68d04460 con 0x555b68da90e0 2024-04-26T17:57:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa3879d1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x555b68cbe590 con 0x555b68d9e820 2024-04-26T17:57:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa3879d1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555b68d00420 con 0x555b68d9df00 2024-04-26T17:57:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa36f7fe640 1 --2- 172.21.15.31:0/947490441 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa3680a3f60 0x7fa3680a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa37dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68da90e0 0x555b68dab4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa37dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68da90e0 0x555b68dab4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34518/0 (socket says 172.21.15.31:34518) 2024-04-26T17:57:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa37dd9b640 1 -- 172.21.15.31:0/3699712641 learned_addr learned my addr 172.21.15.31:0/3699712641 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa37ed9d640 1 --2- 172.21.15.31:0/947490441 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa3680a3f60 0x7fa3680a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa37e59c640 1 --2- 172.21.15.31:0/3699712641 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9e820 0x555b68da8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.647+0000 7fa37dd9b640 1 --2- 172.21.15.31:0/3699712641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68d9df00 0x555b68d9e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.648+0000 7fa37dd9b640 1 -- 172.21.15.31:0/3699712641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68d9df00 msgr2=0x555b68d9e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.648+0000 7fa37dd9b640 1 --2- 172.21.15.31:0/3699712641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68d9df00 0x555b68d9e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.648+0000 7fa36f7fe640 1 -- 172.21.15.31:0/947490441 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fa37802e030 con 0x7fa380108940 2024-04-26T17:57:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.648+0000 7fa37dd9b640 1 -- 172.21.15.31:0/3699712641 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9e820 msgr2=0x555b68da8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.648+0000 7fa37dd9b640 1 --2- 172.21.15.31:0/3699712641 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9e820 0x555b68da8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.648+0000 7fa37dd9b640 1 -- 172.21.15.31:0/3699712641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b68ccb0d0 con 0x555b68da90e0 2024-04-26T17:57:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.648+0000 7fa37e59c640 1 --2- 172.21.15.31:0/3699712641 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9e820 0x555b68da8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:57:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.648+0000 7fa37dd9b640 1 --2- 172.21.15.31:0/3699712641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68da90e0 0x555b68dab4d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa374009f90 tx=0x7fa374009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1080e770bbe7a4d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.648+0000 7fa37ed9d640 1 --2- 172.21.15.31:0/947490441 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa3680a3f60 0x7fa3680a6420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa3780ef080 tx=0x7fa3780a8040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.648+0000 7fa36cff9640 1 -- 172.21.15.31:0/3699712641 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa374004250 con 0x555b68da90e0 2024-04-26T17:57:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.648+0000 7fa36cff9640 1 -- 172.21.15.31:0/3699712641 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3740043f0 con 0x555b68da90e0 2024-04-26T17:57:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.648+0000 7fa36cff9640 1 -- 172.21.15.31:0/3699712641 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3740108f0 con 0x555b68da90e0 2024-04-26T17:57:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.649+0000 7fa3879d1740 1 -- 172.21.15.31:0/3699712641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68da90e0 msgr2=0x555b68dab4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.649+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3699712641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68da90e0 0x555b68dab4d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa374009f90 tx=0x7fa374009790 comp rx=0 tx=0).stop 2024-04-26T17:57:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.649+0000 7fa3879d1740 1 -- 172.21.15.31:0/3699712641 shutdown_connections 2024-04-26T17:57:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.649+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3699712641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68da90e0 0x555b68dab4d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.649+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3699712641 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9e820 0x555b68da8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.649+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3699712641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68d9df00 0x555b68d9e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.649+0000 7fa3879d1740 1 -- 172.21.15.31:0/3699712641 >> 172.21.15.31:0/3699712641 conn(0x555b68d9a410 msgr2=0x555b68d9a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.649+0000 7fa3879d1740 1 -- 172.21.15.31:0/3699712641 shutdown_connections 2024-04-26T17:57:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.649+0000 7fa3879d1740 1 -- 172.21.15.31:0/3699712641 wait complete. 2024-04-26T17:57:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.649+0000 7fa3879d1740 1 -- start start 2024-04-26T17:57:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.649+0000 7fa3879d1740 1 Processor -- start 2024-04-26T17:57:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa3879d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9df00 0x555b68da60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa3879d1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9e820 0x555b68da65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa3879d1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68da90e0 0x555b68da6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa3879d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555b68db0120 con 0x555b68d9df00 2024-04-26T17:57:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa3879d1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x555b68c8bb60 con 0x555b68d9e820 2024-04-26T17:57:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa3879d1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555b68db02a0 con 0x555b68da90e0 2024-04-26T17:57:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa3879d1740 10 client.0 _ll_register_callbacks cb 0x555b68cb02b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:57:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa37dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9df00 0x555b68da60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa37dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9df00 0x555b68da60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34524/0 (socket says 172.21.15.31:34524) 2024-04-26T17:57:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa37dd9b640 1 -- 172.21.15.31:0/3908814854 learned_addr learned my addr 172.21.15.31:0/3908814854 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa37e59c640 1 --2- 172.21.15.31:0/3908814854 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9e820 0x555b68da65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa37dd9b640 1 --2- 172.21.15.31:0/3908814854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68da90e0 0x555b68da6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa37dd9b640 1 -- 172.21.15.31:0/3908814854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68da90e0 msgr2=0x555b68da6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa37dd9b640 1 --2- 172.21.15.31:0/3908814854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68da90e0 0x555b68da6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa37dd9b640 1 -- 172.21.15.31:0/3908814854 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9e820 msgr2=0x555b68da65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa37dd9b640 1 --2- 172.21.15.31:0/3908814854 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9e820 0x555b68da65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa37dd9b640 1 -- 172.21.15.31:0/3908814854 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b68a16210 con 0x555b68d9df00 2024-04-26T17:57:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa37e59c640 1 --2- 172.21.15.31:0/3908814854 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9e820 0x555b68da65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa37dd9b640 1 --2- 172.21.15.31:0/3908814854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9df00 0x555b68da60a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa374003f90 tx=0x7fa3740095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f4dddb0a4f572fe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.650+0000 7fa365ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:57:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.651+0000 7fa365ffb640 1 -- 172.21.15.31:0/3908814854 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa374010690 con 0x555b68d9df00 2024-04-26T17:57:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.651+0000 7fa365ffb640 1 -- 172.21.15.31:0/3908814854 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fa374010830 con 0x555b68d9df00 2024-04-26T17:57:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.651+0000 7fa3879d1740 1 -- 172.21.15.31:0/3908814854 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3740043f0 con 0x555b68d9df00 2024-04-26T17:57:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.651+0000 7fa365ffb640 1 -- 172.21.15.31:0/3908814854 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa374035050 con 0x555b68d9df00 2024-04-26T17:57:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.651+0000 7fa3879d1740 10 client.15378 fetch_fsmap learned FSMap version 29 2024-04-26T17:57:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.651+0000 7fa3879d1740 1 -- 172.21.15.31:0/3908814854 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555b68ae2620 con 0x555b68d9df00 2024-04-26T17:57:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.651+0000 7fa365ffb640 1 -- 172.21.15.31:0/3908814854 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3740043f0 con 0x555b68d9df00 2024-04-26T17:57:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.651+0000 7fa365ffb640 1 -- 172.21.15.31:0/3908814854 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fa37401c020 con 0x555b68d9df00 2024-04-26T17:57:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.651+0000 7fa3879d1740 10 client.15378 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:57:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.651+0000 7fa3879d1740 10 client.15378 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:57:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.651+0000 7fa3879d1740 20 client.15378 populate_metadata read hostname 'smithi031' 2024-04-26T17:57:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.651+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3908814854 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555b68dbd5d0 0x555b68dbf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.651+0000 7fa3879d1740 4 client.15378 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:57:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.651+0000 7fa3879d1740 1 -- 172.21.15.31:0/3908814854 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b68d12070 con 0x555b68dbd5d0 2024-04-26T17:57:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.652+0000 7fa37e59c640 1 --2- 172.21.15.31:0/3908814854 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555b68dbd5d0 0x555b68dbf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.652+0000 7fa365ffb640 10 client.15378 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.652+0000 7fa37e59c640 1 --2- 172.21.15.31:0/3908814854 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555b68dbd5d0 0x555b68dbf9c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=746eb8c8d09204c7 server_cookie=388f783443fc59fe in_seq=0 out_seq=0 2024-04-26T17:57:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa365ffb640 1 -- 172.21.15.31:0/3908814854 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555b68d12070 con 0x555b68dbd5d0 2024-04-26T17:57:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa365ffb640 10 client.15378 handle_command_reply: tid=0 2024-04-26T17:57:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa3879d1740 1 client.15378 shutdown 2024-04-26T17:57:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa3879d1740 1 -- 172.21.15.31:0/3908814854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9df00 msgr2=0x555b68da60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3908814854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9df00 0x555b68da60a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa374003f90 tx=0x7fa3740095e0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa3879d1740 1 -- 172.21.15.31:0/3908814854 shutdown_connections 2024-04-26T17:57:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3908814854 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555b68dbd5d0 0x555b68dbf9c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3908814854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68da90e0 0x555b68da6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3908814854 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9e820 0x555b68da65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa365ffb640 0 client.15378 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3908814854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9df00 0x555b68da60a0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa3879d1740 1 -- 172.21.15.31:0/3908814854 >> 172.21.15.31:0/3908814854 conn(0x555b68d9a410 msgr2=0x555b68da0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa3879d1740 1 -- 172.21.15.31:0/3908814854 shutdown_connections 2024-04-26T17:57:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa3879d1740 1 -- 172.21.15.31:0/3908814854 wait complete. 2024-04-26T17:57:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.653+0000 7fa3879d1740 20 client.15378 trim_cache size 0 max 16384 2024-04-26T17:57:11.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.668+0000 7fa3879d1740 1 Processor -- start 2024-04-26T17:57:11.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.668+0000 7fa3879d1740 1 -- start start 2024-04-26T17:57:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa3879d1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68d9df00 0x555b68db8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa3879d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9e820 0x555b68da8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa3879d1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68da90e0 0x555b68da08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa3879d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa3801ae9f0 con 0x555b68d9e820 2024-04-26T17:57:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa3879d1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa3801ac260 con 0x555b68da90e0 2024-04-26T17:57:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa3879d1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa3801aa0d0 con 0x555b68d9df00 2024-04-26T17:57:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa37e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9e820 0x555b68da8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa37e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9e820 0x555b68da8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34536/0 (socket says 172.21.15.31:34536) 2024-04-26T17:57:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa37e59c640 1 -- 172.21.15.31:0/126902692 learned_addr learned my addr 172.21.15.31:0/126902692 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa37dd9b640 1 --2- 172.21.15.31:0/126902692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68d9df00 0x555b68db8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa37e59c640 1 -- 172.21.15.31:0/126902692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68d9df00 msgr2=0x555b68db8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa37e59c640 1 --2- 172.21.15.31:0/126902692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68d9df00 0x555b68db8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa37e59c640 1 -- 172.21.15.31:0/126902692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68da90e0 msgr2=0x555b68da08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa37e59c640 1 --2- 172.21.15.31:0/126902692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68da90e0 0x555b68da08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa37e59c640 1 -- 172.21.15.31:0/126902692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b68d95d60 con 0x555b68d9e820 2024-04-26T17:57:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa37e59c640 1 --2- 172.21.15.31:0/126902692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9e820 0x555b68da8160 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x555b68dc00f0 tx=0x7fa370005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646ebcbcb298b340 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa367fff640 1 -- 172.21.15.31:0/126902692 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa370005020 con 0x555b68d9e820 2024-04-26T17:57:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa367fff640 1 -- 172.21.15.31:0/126902692 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa37000f810 con 0x555b68d9e820 2024-04-26T17:57:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.669+0000 7fa367fff640 1 -- 172.21.15.31:0/126902692 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3700049e0 con 0x555b68d9e820 2024-04-26T17:57:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.670+0000 7fa3879d1740 1 -- 172.21.15.31:0/126902692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9e820 msgr2=0x555b68da8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.670+0000 7fa3879d1740 1 --2- 172.21.15.31:0/126902692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9e820 0x555b68da8160 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x555b68dc00f0 tx=0x7fa370005f90 comp rx=0 tx=0).stop 2024-04-26T17:57:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.670+0000 7fa3879d1740 1 -- 172.21.15.31:0/126902692 shutdown_connections 2024-04-26T17:57:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.670+0000 7fa3879d1740 1 --2- 172.21.15.31:0/126902692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68da90e0 0x555b68da08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.670+0000 7fa3879d1740 1 --2- 172.21.15.31:0/126902692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9e820 0x555b68da8160 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.670+0000 7fa3879d1740 1 --2- 172.21.15.31:0/126902692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68d9df00 0x555b68db8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.670+0000 7fa3879d1740 1 -- 172.21.15.31:0/126902692 >> 172.21.15.31:0/126902692 conn(0x555b68d9a410 msgr2=0x555b68da6060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.670+0000 7fa3879d1740 1 -- 172.21.15.31:0/126902692 shutdown_connections 2024-04-26T17:57:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.670+0000 7fa3879d1740 1 -- 172.21.15.31:0/126902692 wait complete. 2024-04-26T17:57:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.670+0000 7fa3879d1740 1 -- start start 2024-04-26T17:57:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa3879d1740 1 Processor -- start 2024-04-26T17:57:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa3879d1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9df00 0x555b68da3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa3879d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9e820 0x555b68da35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa3879d1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68da90e0 0x555b68da3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa3879d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa3801aa720 con 0x555b68d9e820 2024-04-26T17:57:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa3879d1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa3801abb90 con 0x555b68d9df00 2024-04-26T17:57:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa3879d1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa3801aadf0 con 0x555b68da90e0 2024-04-26T17:57:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa3879d1740 10 client.0 _ll_register_callbacks cb 0x7fa374012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:57:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa37e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9e820 0x555b68da35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa37dd9b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9df00 0x555b68da3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa37e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9e820 0x555b68da35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34548/0 (socket says 172.21.15.31:34548) 2024-04-26T17:57:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa37dd9b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9df00 0x555b68da3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:44134/0 (socket says 172.21.15.31:44134) 2024-04-26T17:57:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa37e59c640 1 -- 172.21.15.31:0/3694684942 learned_addr learned my addr 172.21.15.31:0/3694684942 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa37dd9b640 1 --2- 172.21.15.31:0/3694684942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68da90e0 0x555b68da3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa37dd9b640 1 -- 172.21.15.31:0/3694684942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68da90e0 msgr2=0x555b68da3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa37dd9b640 1 --2- 172.21.15.31:0/3694684942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68da90e0 0x555b68da3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa37dd9b640 1 -- 172.21.15.31:0/3694684942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9e820 msgr2=0x555b68da35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa37dd9b640 1 --2- 172.21.15.31:0/3694684942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9e820 0x555b68da35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa37dd9b640 1 -- 172.21.15.31:0/3694684942 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b68e22c90 con 0x555b68d9df00 2024-04-26T17:57:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa37dd9b640 1 --2- 172.21.15.31:0/3694684942 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9df00 0x555b68da3080 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa374010420 tx=0x7fa374009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31f1a84298f56140 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.671+0000 7fa365ffb640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T17:57:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa365ffb640 1 -- 172.21.15.31:0/3694684942 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37401e440 con 0x555b68d9df00 2024-04-26T17:57:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa3879d1740 1 -- 172.21.15.31:0/3694684942 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa37000f810 con 0x555b68d9df00 2024-04-26T17:57:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa365ffb640 1 -- 172.21.15.31:0/3694684942 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fa37401e5e0 con 0x555b68d9df00 2024-04-26T17:57:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa365ffb640 1 -- 172.21.15.31:0/3694684942 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa37401e800 con 0x555b68d9df00 2024-04-26T17:57:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa3879d1740 10 client.25106 fetch_fsmap learned FSMap version 29 2024-04-26T17:57:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa3879d1740 1 -- 172.21.15.31:0/3694684942 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555b68e23fa0 con 0x555b68d9df00 2024-04-26T17:57:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa365ffb640 1 -- 172.21.15.31:0/3694684942 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa37000f810 con 0x555b68d9df00 2024-04-26T17:57:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa365ffb640 1 -- 172.21.15.31:0/3694684942 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fa37401c040 con 0x555b68d9df00 2024-04-26T17:57:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa3879d1740 10 client.25106 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:57:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa3879d1740 10 client.25106 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:57:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa3879d1740 20 client.25106 populate_metadata read hostname 'smithi031' 2024-04-26T17:57:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3694684942 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555b68e24500 0x555b68e268f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa3879d1740 4 client.25106 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:57:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa3879d1740 1 -- 172.21.15.31:0/3694684942 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555b68d0bee0 con 0x555b68e24500 2024-04-26T17:57:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.672+0000 7fa37e59c640 1 --2- 172.21.15.31:0/3694684942 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555b68e24500 0x555b68e268f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.673+0000 7fa365ffb640 10 client.25106 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.673+0000 7fa37e59c640 1 --2- 172.21.15.31:0/3694684942 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555b68e24500 0x555b68e268f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f8b19452c24f522 server_cookie=bf5c1afaf2710f08 in_seq=0 out_seq=0 2024-04-26T17:57:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa365ffb640 1 -- 172.21.15.31:0/3694684942 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555b68d0bee0 con 0x555b68e24500 2024-04-26T17:57:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa365ffb640 10 client.25106 handle_command_reply: tid=0 2024-04-26T17:57:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa3879d1740 1 client.25106 shutdown 2024-04-26T17:57:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa3879d1740 1 -- 172.21.15.31:0/3694684942 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9df00 msgr2=0x555b68da3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3694684942 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9df00 0x555b68da3080 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa374010420 tx=0x7fa374009ed0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa3879d1740 1 -- 172.21.15.31:0/3694684942 shutdown_connections 2024-04-26T17:57:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3694684942 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555b68e24500 0x555b68e268f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3694684942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b68da90e0 0x555b68da3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3694684942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b68d9e820 0x555b68da35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa365ffb640 0 client.25106 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa3879d1740 1 --2- 172.21.15.31:0/3694684942 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555b68d9df00 0x555b68da3080 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa3879d1740 1 -- 172.21.15.31:0/3694684942 >> 172.21.15.31:0/3694684942 conn(0x555b68d9a410 msgr2=0x555b68da7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa3879d1740 1 -- 172.21.15.31:0/3694684942 shutdown_connections 2024-04-26T17:57:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa3879d1740 1 -- 172.21.15.31:0/3694684942 wait complete. 2024-04-26T17:57:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.743+0000 7fa3879d1740 20 client.25106 trim_cache size 0 max 16384 2024-04-26T17:57:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:57:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:57:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:57:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:57:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.744+0000 7fa384be4640 1 -- 172.21.15.31:0/947490441 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa3680a3f60 msgr2=0x7fa3680a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.744+0000 7fa384be4640 1 --2- 172.21.15.31:0/947490441 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa3680a3f60 0x7fa3680a6420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa3780ef080 tx=0x7fa3780a8040 comp rx=0 tx=0).stop 2024-04-26T17:57:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.744+0000 7fa384be4640 1 -- 172.21.15.31:0/947490441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa380108940 msgr2=0x7fa38019d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.744+0000 7fa384be4640 1 --2- 172.21.15.31:0/947490441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa380108940 0x7fa38019d910 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa378033040 tx=0x7fa378009360 comp rx=0 tx=0).stop 2024-04-26T17:57:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.744+0000 7fa384be4640 1 -- 172.21.15.31:0/947490441 shutdown_connections 2024-04-26T17:57:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.744+0000 7fa384be4640 1 --2- 172.21.15.31:0/947490441 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa3680a3f60 0x7fa3680a6420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.744+0000 7fa384be4640 1 --2- 172.21.15.31:0/947490441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa380108940 0x7fa38019d910 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.744+0000 7fa384be4640 1 --2- 172.21.15.31:0/947490441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3801032f0 0x7fa3801995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.744+0000 7fa384be4640 1 --2- 172.21.15.31:0/947490441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa380102930 0x7fa3801990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.744+0000 7fa384be4640 1 -- 172.21.15.31:0/947490441 >> 172.21.15.31:0/947490441 conn(0x7fa3800fe400 msgr2=0x7fa3800fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.744+0000 7fa384be4640 1 -- 172.21.15.31:0/947490441 shutdown_connections 2024-04-26T17:57:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:11.744+0000 7fa384be4640 1 -- 172.21.15.31:0/947490441 wait complete. 2024-04-26T17:57:13.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:12 smithi186 ceph-mon[32797]: pgmap v24: 97 pgs: 97 active+clean; 395 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 59 MiB/s wr, 485 op/s 2024-04-26T17:57:13.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:12 smithi031 ceph-mon[29164]: pgmap v24: 97 pgs: 97 active+clean; 395 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 59 MiB/s wr, 485 op/s 2024-04-26T17:57:13.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:12 smithi097 ceph-mon[32419]: pgmap v24: 97 pgs: 97 active+clean; 395 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 59 MiB/s wr, 485 op/s 2024-04-26T17:57:15.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:14 smithi186 ceph-mon[32797]: pgmap v25: 97 pgs: 97 active+clean; 395 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 37 MiB/s wr, 344 op/s 2024-04-26T17:57:15.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:14 smithi031 ceph-mon[29164]: pgmap v25: 97 pgs: 97 active+clean; 395 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 37 MiB/s wr, 344 op/s 2024-04-26T17:57:15.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:14 smithi097 ceph-mon[32419]: pgmap v25: 97 pgs: 97 active+clean; 395 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 37 MiB/s wr, 344 op/s 2024-04-26T17:57:17.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:16 smithi186 ceph-mon[32797]: pgmap v26: 97 pgs: 97 active+clean; 396 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 37 MiB/s wr, 401 op/s 2024-04-26T17:57:17.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:16 smithi031 ceph-mon[29164]: pgmap v26: 97 pgs: 97 active+clean; 396 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 37 MiB/s wr, 401 op/s 2024-04-26T17:57:17.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:16 smithi097 ceph-mon[32419]: pgmap v26: 97 pgs: 97 active+clean; 396 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 37 MiB/s wr, 401 op/s 2024-04-26T17:57:19.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:18 smithi186 ceph-mon[32797]: pgmap v27: 97 pgs: 97 active+clean; 396 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 31 MiB/s wr, 170 op/s 2024-04-26T17:57:19.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:18 smithi031 ceph-mon[29164]: pgmap v27: 97 pgs: 97 active+clean; 396 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 31 MiB/s wr, 170 op/s 2024-04-26T17:57:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:18 smithi097 ceph-mon[32419]: pgmap v27: 97 pgs: 97 active+clean; 396 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 31 MiB/s wr, 170 op/s 2024-04-26T17:57:19.657 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-26T17:57:19.657 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-26T17:57:19.658 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:57:19.920 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:57:20.779 INFO:teuthology.orchestra.run.smithi031.stderr:vacuuming... 2024-04-26T17:57:21.270 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:20 smithi186 ceph-mon[32797]: pgmap v28: 97 pgs: 97 active+clean; 396 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 30 MiB/s wr, 84 op/s 2024-04-26T17:57:21.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:20 smithi031 ceph-mon[29164]: pgmap v28: 97 pgs: 97 active+clean; 396 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 30 MiB/s wr, 84 op/s 2024-04-26T17:57:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:20 smithi097 ceph-mon[32419]: pgmap v28: 97 pgs: 97 active+clean; 396 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 30 MiB/s wr, 84 op/s 2024-04-26T17:57:21.688 INFO:teuthology.orchestra.run.smithi031.stderr:creating primary keys... 2024-04-26T17:57:21.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.689+0000 7feecd30e700 1 -- 172.21.15.31:0/2929995193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec81024a0 msgr2=0x7feec810e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:21.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.689+0000 7feecd30e700 1 --2- 172.21.15.31:0/2929995193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec81024a0 0x7feec810e960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7feec0009990 tx=0x7feec0009ca0 comp rx=0 tx=0).stop 2024-04-26T17:57:21.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.689+0000 7feecd30e700 1 -- 172.21.15.31:0/2929995193 shutdown_connections 2024-04-26T17:57:21.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.689+0000 7feecd30e700 1 --2- 172.21.15.31:0/2929995193 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec8107a90 0x7feec8107e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:21.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.689+0000 7feecd30e700 1 --2- 172.21.15.31:0/2929995193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec81024a0 0x7feec810e960 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:21.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.689+0000 7feecd30e700 1 --2- 172.21.15.31:0/2929995193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feec8101ae0 0x7feec8101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:21.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.689+0000 7feecd30e700 1 -- 172.21.15.31:0/2929995193 >> 172.21.15.31:0/2929995193 conn(0x7feec8078990 msgr2=0x7feec80ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:21.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.689+0000 7feecd30e700 1 -- 172.21.15.31:0/2929995193 shutdown_connections 2024-04-26T17:57:21.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.689+0000 7feecd30e700 1 -- 172.21.15.31:0/2929995193 wait complete. 2024-04-26T17:57:21.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.690+0000 7feecd30e700 1 Processor -- start 2024-04-26T17:57:21.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.690+0000 7feecd30e700 1 -- start start 2024-04-26T17:57:21.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.690+0000 7feecd30e700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec8101ae0 0x7feec80fde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:21.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.690+0000 7feecd30e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feec81024a0 0x7feec80fe340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:21.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.690+0000 7feecd30e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec8107a90 0x7feec80fe880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:21.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.690+0000 7feecd30e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feec81124d0 con 0x7feec8107a90 2024-04-26T17:57:21.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.690+0000 7feecd30e700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7feec8112350 con 0x7feec8101ae0 2024-04-26T17:57:21.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.690+0000 7feecd30e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feec8112650 con 0x7feec81024a0 2024-04-26T17:57:21.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.690+0000 7feec6ffd700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec8101ae0 0x7feec80fde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:21.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feec77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec8107a90 0x7feec80fe880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:21.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feec6ffd700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec8101ae0 0x7feec80fde00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:54434/0 (socket says 172.21.15.31:54434) 2024-04-26T17:57:21.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feec67fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feec81024a0 0x7feec80fe340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:21.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feec6ffd700 1 -- 172.21.15.31:0/1079290345 learned_addr learned my addr 172.21.15.31:0/1079290345 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:21.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feec77fe700 1 -- 172.21.15.31:0/1079290345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feec81024a0 msgr2=0x7feec80fe340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:21.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feec77fe700 1 --2- 172.21.15.31:0/1079290345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feec81024a0 0x7feec80fe340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:21.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feec77fe700 1 -- 172.21.15.31:0/1079290345 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec8101ae0 msgr2=0x7feec80fde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:21.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feec77fe700 1 --2- 172.21.15.31:0/1079290345 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec8101ae0 0x7feec80fde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:21.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feec77fe700 1 -- 172.21.15.31:0/1079290345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec81a7f80 con 0x7feec8107a90 2024-04-26T17:57:21.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feec6ffd700 1 --2- 172.21.15.31:0/1079290345 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec8101ae0 0x7feec80fde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:21.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feec77fe700 1 --2- 172.21.15.31:0/1079290345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec8107a90 0x7feec80fe880 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7feec00098e0 tx=0x7feec00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:21.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feebffff700 1 -- 172.21.15.31:0/1079290345 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec0027070 con 0x7feec8107a90 2024-04-26T17:57:21.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feebffff700 1 -- 172.21.15.31:0/1079290345 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feec0022470 con 0x7feec8107a90 2024-04-26T17:57:21.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feecd30e700 1 -- 172.21.15.31:0/1079290345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec81a8210 con 0x7feec8107a90 2024-04-26T17:57:21.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feebffff700 1 -- 172.21.15.31:0/1079290345 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec0017650 con 0x7feec8107a90 2024-04-26T17:57:21.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.691+0000 7feecd30e700 1 -- 172.21.15.31:0/1079290345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec81a86c0 con 0x7feec8107a90 2024-04-26T17:57:21.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.692+0000 7feecd30e700 1 -- 172.21.15.31:0/1079290345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feec81031d0 con 0x7feec8107a90 2024-04-26T17:57:21.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.694+0000 7feebffff700 1 -- 172.21.15.31:0/1079290345 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7feec002a070 con 0x7feec8107a90 2024-04-26T17:57:21.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.695+0000 7feebffff700 1 --2- 172.21.15.31:0/1079290345 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7feeb40a3f50 0x7feeb40a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:21.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.695+0000 7feebffff700 1 -- 172.21.15.31:0/1079290345 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7feec00d1da0 con 0x7feec8107a90 2024-04-26T17:57:21.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.695+0000 7feec6ffd700 1 --2- 172.21.15.31:0/1079290345 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7feeb40a3f50 0x7feeb40a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:21.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.695+0000 7feec6ffd700 1 --2- 172.21.15.31:0/1079290345 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7feeb40a3f50 0x7feeb40a6410 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feeb8009c10 tx=0x7feeb800b040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:21.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.699+0000 7feebffff700 1 -- 172.21.15.31:0/1079290345 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feec0031070 con 0x7feec8107a90 2024-04-26T17:57:21.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.977+0000 7feecd30e700 1 -- 172.21.15.31:0/1079290345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feec802e490 con 0x7feec8107a90 2024-04-26T17:57:21.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.978+0000 7feebffff700 1 -- 172.21.15.31:0/1079290345 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11875 (secure 0 0 0) 0x7feec0013070 con 0x7feec8107a90 2024-04-26T17:57:21.979 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:57:21.979 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T17:57:06.843229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24446,24457,24473]},"id":1}]} 2024-04-26T17:57:21.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.982+0000 7feecd30e700 1 -- 172.21.15.31:0/1079290345 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7feeb40a3f50 msgr2=0x7feeb40a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:21.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.982+0000 7feecd30e700 1 --2- 172.21.15.31:0/1079290345 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7feeb40a3f50 0x7feeb40a6410 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feeb8009c10 tx=0x7feeb800b040 comp rx=0 tx=0).stop 2024-04-26T17:57:21.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.982+0000 7feecd30e700 1 -- 172.21.15.31:0/1079290345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec8107a90 msgr2=0x7feec80fe880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:21.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.982+0000 7feecd30e700 1 --2- 172.21.15.31:0/1079290345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec8107a90 0x7feec80fe880 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7feec00098e0 tx=0x7feec00047b0 comp rx=0 tx=0).stop 2024-04-26T17:57:21.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.982+0000 7feecd30e700 1 -- 172.21.15.31:0/1079290345 shutdown_connections 2024-04-26T17:57:21.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.982+0000 7feecd30e700 1 --2- 172.21.15.31:0/1079290345 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7feeb40a3f50 0x7feeb40a6410 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:21.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.982+0000 7feecd30e700 1 --2- 172.21.15.31:0/1079290345 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feec8101ae0 0x7feec80fde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:21.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.982+0000 7feecd30e700 1 --2- 172.21.15.31:0/1079290345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec8107a90 0x7feec80fe880 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:21.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.982+0000 7feecd30e700 1 --2- 172.21.15.31:0/1079290345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feec81024a0 0x7feec80fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:21.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.982+0000 7feecd30e700 1 -- 172.21.15.31:0/1079290345 >> 172.21.15.31:0/1079290345 conn(0x7feec8078990 msgr2=0x7feec810f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:21.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.982+0000 7feecd30e700 1 -- 172.21.15.31:0/1079290345 shutdown_connections 2024-04-26T17:57:21.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:21.982+0000 7feecd30e700 1 -- 172.21.15.31:0/1079290345 wait complete. 2024-04-26T17:57:21.983 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-26T17:57:22.807 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24473 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-26T17:57:22.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24473 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-26T17:57:23.008 INFO:teuthology.orchestra.run.smithi031.stderr:done in 24.22 s (drop tables 0.00 s, create tables 0.01 s, client-side generate 21.98 s, vacuum 0.91 s, primary keys 1.32 s). 2024-04-26T17:57:23.011 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900' 2024-04-26T17:57:23.033 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:22 smithi031 ceph-mon[29164]: pgmap v29: 97 pgs: 97 active+clean; 804 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 64 MiB/s wr, 96 op/s 2024-04-26T17:57:23.033 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:22 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1079290345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:57:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:starting vacuum...end. 2024-04-26T17:57:23.140 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:57:23.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:22 smithi186 ceph-mon[32797]: pgmap v29: 97 pgs: 97 active+clean; 804 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 64 MiB/s wr, 96 op/s 2024-04-26T17:57:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:22 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1079290345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:57:23.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:22 smithi097 ceph-mon[32419]: pgmap v29: 97 pgs: 97 active+clean; 804 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 64 MiB/s wr, 96 op/s 2024-04-26T17:57:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:22 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1079290345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:57:24.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.852+0000 7f4721a62640 1 -- 172.21.15.31:0/2232982731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c112630 msgr2=0x7f471c11a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:24.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.852+0000 7f4721a62640 1 --2- 172.21.15.31:0/2232982731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c112630 0x7f471c11a200 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4714009ca0 tx=0x7f471402a8c0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.852+0000 7f4721a62640 1 -- 172.21.15.31:0/2232982731 shutdown_connections 2024-04-26T17:57:24.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.852+0000 7f4721a62640 1 --2- 172.21.15.31:0/2232982731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c112630 0x7f471c11a200 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.852+0000 7f4721a62640 1 --2- 172.21.15.31:0/2232982731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f471c10df10 0x7f471c111f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.852+0000 7f4721a62640 1 --2- 172.21.15.31:0/2232982731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f471c10d560 0x7f471c10d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.852+0000 7f4721a62640 1 -- 172.21.15.31:0/2232982731 >> 172.21.15.31:0/2232982731 conn(0x7f471c06db90 msgr2=0x7f471c10b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:24.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.852+0000 7f4721a62640 1 -- 172.21.15.31:0/2232982731 shutdown_connections 2024-04-26T17:57:24.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.852+0000 7f4721a62640 1 -- 172.21.15.31:0/2232982731 wait complete. 2024-04-26T17:57:24.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.852+0000 7f4721a62640 1 Processor -- start 2024-04-26T17:57:24.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f4721a62640 1 -- start start 2024-04-26T17:57:24.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f4721a62640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c10d560 0x7f471c1a97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f4721a62640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f471c112630 0x7f471c1a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f4721a62640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f471c1a3950 0x7f471c1a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f4721a62640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f471c11d5f0 con 0x7f471c10d560 2024-04-26T17:57:24.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f4721a62640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f471c11d470 con 0x7f471c1a3950 2024-04-26T17:57:24.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f4721a62640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f471c11d770 con 0x7f471c112630 2024-04-26T17:57:24.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f471b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c10d560 0x7f471c1a97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f471b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c10d560 0x7f471c1a97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57288/0 (socket says 172.21.15.31:57288) 2024-04-26T17:57:24.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f471b7fe640 1 -- 172.21.15.31:0/766967318 learned_addr learned my addr 172.21.15.31:0/766967318 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:24.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f471bfff640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f471c1a3950 0x7f471c1a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f471affd640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f471c112630 0x7f471c1a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f471b7fe640 1 -- 172.21.15.31:0/766967318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f471c112630 msgr2=0x7f471c1a9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:24.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f471b7fe640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f471c112630 0x7f471c1a9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f471b7fe640 1 -- 172.21.15.31:0/766967318 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f471c1a3950 msgr2=0x7f471c1a3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f471b7fe640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f471c1a3950 0x7f471c1a3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f471b7fe640 1 -- 172.21.15.31:0/766967318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f471c1a4470 con 0x7f471c10d560 2024-04-26T17:57:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f471bfff640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f471c1a3950 0x7f471c1a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:57:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.853+0000 7f471b7fe640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c10d560 0x7f471c1a97a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f470c00b4a0 tx=0x7f470c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.854+0000 7f4718ff9640 1 -- 172.21.15.31:0/766967318 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c014070 con 0x7f471c10d560 2024-04-26T17:57:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.854+0000 7f4718ff9640 1 -- 172.21.15.31:0/766967318 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f470c010ac0 con 0x7f471c10d560 2024-04-26T17:57:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.854+0000 7f4718ff9640 1 -- 172.21.15.31:0/766967318 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c017c30 con 0x7f471c10d560 2024-04-26T17:57:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.854+0000 7f4721a62640 1 -- 172.21.15.31:0/766967318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f471c1a4760 con 0x7f471c10d560 2024-04-26T17:57:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.854+0000 7f4721a62640 1 -- 172.21.15.31:0/766967318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f471c1b09b0 con 0x7f471c10d560 2024-04-26T17:57:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.856+0000 7f4718ff9640 1 -- 172.21.15.31:0/766967318 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f470c017070 con 0x7f471c10d560 2024-04-26T17:57:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.856+0000 7f4724776740 1 Processor -- start 2024-04-26T17:57:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.856+0000 7f4724776740 1 -- start start 2024-04-26T17:57:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f4724776740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d577112e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f4724776740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d5771bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f4724776740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 0x561d5771e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f4724776740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561d57677460 con 0x561d57711820 2024-04-26T17:57:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f4724776740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x561d57631590 con 0x561d57710f00 2024-04-26T17:57:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f4724776740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561d57673420 con 0x561d5771c0e0 2024-04-26T17:57:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f471b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d5771bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f471b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d5771bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57292/0 (socket says 172.21.15.31:57292) 2024-04-26T17:57:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f471b7fe640 1 -- 172.21.15.31:0/2470188701 learned_addr learned my addr 172.21.15.31:0/2470188701 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f471affd640 1 --2- 172.21.15.31:0/2470188701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 0x561d5771e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f4718ff9640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f47040a3f60 0x7f47040a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f471bfff640 1 --2- 172.21.15.31:0/2470188701 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d577112e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f471b7fe640 1 -- 172.21.15.31:0/2470188701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 msgr2=0x561d5771e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f471b7fe640 1 --2- 172.21.15.31:0/2470188701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 0x561d5771e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f471b7fe640 1 -- 172.21.15.31:0/2470188701 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 msgr2=0x561d577112e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f471b7fe640 1 --2- 172.21.15.31:0/2470188701 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d577112e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f471bfff640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f47040a3f60 0x7f47040a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f471b7fe640 1 -- 172.21.15.31:0/2470188701 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d5763e0d0 con 0x561d57711820 2024-04-26T17:57:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f4718ff9640 1 -- 172.21.15.31:0/766967318 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f470c0cfdc0 con 0x7f471c10d560 2024-04-26T17:57:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f471b7fe640 1 --2- 172.21.15.31:0/2470188701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d5771bba0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f470c098070 tx=0x7f470c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=106ef00e84e92233 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.857+0000 7f4701ffb640 1 -- 172.21.15.31:0/2470188701 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c017360 con 0x561d57711820 2024-04-26T17:57:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.858+0000 7f4701ffb640 1 -- 172.21.15.31:0/2470188701 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f470c09c310 con 0x561d57711820 2024-04-26T17:57:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.858+0000 7f4701ffb640 1 -- 172.21.15.31:0/2470188701 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c09ffc0 con 0x561d57711820 2024-04-26T17:57:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.858+0000 7f4724776740 1 -- 172.21.15.31:0/2470188701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 msgr2=0x561d5771bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.858+0000 7f4724776740 1 --2- 172.21.15.31:0/2470188701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d5771bba0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f470c098070 tx=0x7f470c020000 comp rx=0 tx=0).stop 2024-04-26T17:57:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.858+0000 7f471bfff640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f47040a3f60 0x7f47040a6420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4714009430 tx=0x7f47140095c0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.858+0000 7f4724776740 1 -- 172.21.15.31:0/2470188701 shutdown_connections 2024-04-26T17:57:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.858+0000 7f4724776740 1 --2- 172.21.15.31:0/2470188701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 0x561d5771e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.858+0000 7f4724776740 1 --2- 172.21.15.31:0/2470188701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d5771bba0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.858+0000 7f4724776740 1 --2- 172.21.15.31:0/2470188701 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d577112e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.858+0000 7f4724776740 1 -- 172.21.15.31:0/2470188701 >> 172.21.15.31:0/2470188701 conn(0x561d5770d410 msgr2=0x561d5770d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.858+0000 7f4724776740 1 -- 172.21.15.31:0/2470188701 shutdown_connections 2024-04-26T17:57:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.858+0000 7f4724776740 1 -- 172.21.15.31:0/2470188701 wait complete. 2024-04-26T17:57:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.859+0000 7f4724776740 1 -- start start 2024-04-26T17:57:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.859+0000 7f4724776740 1 Processor -- start 2024-04-26T17:57:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.859+0000 7f4724776740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d57710f00 0x561d57719090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.859+0000 7f4724776740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57711820 0x561d577195d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.859+0000 7f4724776740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d5771c0e0 0x561d57719b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.859+0000 7f4724776740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561d5771ec50 con 0x561d5771c0e0 2024-04-26T17:57:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.859+0000 7f4724776740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x561d575feb60 con 0x561d57711820 2024-04-26T17:57:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.859+0000 7f4724776740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561d5771edd0 con 0x561d57710f00 2024-04-26T17:57:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.859+0000 7f4724776740 10 client.0 _ll_register_callbacks cb 0x561d576232b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:57:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.859+0000 7f471bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d5771c0e0 0x561d57719b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.859+0000 7f471bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d5771c0e0 0x561d57719b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57296/0 (socket says 172.21.15.31:57296) 2024-04-26T17:57:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.859+0000 7f471bfff640 1 -- 172.21.15.31:0/2174108888 learned_addr learned my addr 172.21.15.31:0/2174108888 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.859+0000 7f471affd640 1 --2- 172.21.15.31:0/2174108888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57711820 0x561d577195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.859+0000 7f471b7fe640 1 --2- 172.21.15.31:0/2174108888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d57710f00 0x561d57719090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f471bfff640 1 -- 172.21.15.31:0/2174108888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d57710f00 msgr2=0x561d57719090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f471bfff640 1 --2- 172.21.15.31:0/2174108888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d57710f00 0x561d57719090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f471bfff640 1 -- 172.21.15.31:0/2174108888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57711820 msgr2=0x561d577195d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f471bfff640 1 --2- 172.21.15.31:0/2174108888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57711820 0x561d577195d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f471bfff640 1 -- 172.21.15.31:0/2174108888 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d57389210 con 0x561d5771c0e0 2024-04-26T17:57:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f471affd640 1 --2- 172.21.15.31:0/2174108888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57711820 0x561d577195d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f471bfff640 1 --2- 172.21.15.31:0/2174108888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d5771c0e0 0x561d57719b10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f47140098e0 tx=0x7f471402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3341828456ba3fad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f46faffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:57:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f46faffd640 1 -- 172.21.15.31:0/2174108888 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4714003e00 con 0x561d5771c0e0 2024-04-26T17:57:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f46faffd640 1 -- 172.21.15.31:0/2174108888 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f47140047a0 con 0x561d5771c0e0 2024-04-26T17:57:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f4724776740 1 -- 172.21.15.31:0/2174108888 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f470c09c310 con 0x561d5771c0e0 2024-04-26T17:57:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f46faffd640 1 -- 172.21.15.31:0/2174108888 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f471402f660 con 0x561d5771c0e0 2024-04-26T17:57:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f4724776740 10 client.15402 fetch_fsmap learned FSMap version 29 2024-04-26T17:57:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f4724776740 1 -- 172.21.15.31:0/2174108888 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561d57455620 con 0x561d5771c0e0 2024-04-26T17:57:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f46faffd640 1 -- 172.21.15.31:0/2174108888 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f470c09c310 con 0x561d5771c0e0 2024-04-26T17:57:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.860+0000 7f46faffd640 1 -- 172.21.15.31:0/2174108888 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f4714030a00 con 0x561d5771c0e0 2024-04-26T17:57:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.861+0000 7f4724776740 10 client.15402 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:57:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.861+0000 7f4724776740 10 client.15402 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:57:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.861+0000 7f4724776740 20 client.15402 populate_metadata read hostname 'smithi031' 2024-04-26T17:57:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.861+0000 7f4724776740 1 --2- 172.21.15.31:0/2174108888 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x561d5772c240 0x561d5772e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.861+0000 7f4724776740 4 client.15402 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:57:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.861+0000 7f4724776740 1 -- 172.21.15.31:0/2174108888 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d57685070 con 0x561d5772c240 2024-04-26T17:57:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.861+0000 7f471b7fe640 1 --2- 172.21.15.31:0/2174108888 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x561d5772c240 0x561d5772e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.861+0000 7f471b7fe640 1 --2- 172.21.15.31:0/2174108888 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x561d5772c240 0x561d5772e650 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c715a6e7a0ae121e server_cookie=9416e27984840711 in_seq=0 out_seq=0 2024-04-26T17:57:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.861+0000 7f46faffd640 10 client.15402 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.862+0000 7f46faffd640 1 -- 172.21.15.31:0/2174108888 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561d57685070 con 0x561d5772c240 2024-04-26T17:57:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.862+0000 7f46faffd640 10 client.15402 handle_command_reply: tid=0 2024-04-26T17:57:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.862+0000 7f4724776740 1 client.15402 shutdown 2024-04-26T17:57:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.862+0000 7f4724776740 1 -- 172.21.15.31:0/2174108888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d5771c0e0 msgr2=0x561d57719b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.862+0000 7f4724776740 1 --2- 172.21.15.31:0/2174108888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d5771c0e0 0x561d57719b10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f47140098e0 tx=0x7f471402a8c0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.862+0000 7f4724776740 1 -- 172.21.15.31:0/2174108888 shutdown_connections 2024-04-26T17:57:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.862+0000 7f4724776740 1 --2- 172.21.15.31:0/2174108888 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x561d5772c240 0x561d5772e650 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.862+0000 7f4724776740 1 --2- 172.21.15.31:0/2174108888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d5771c0e0 0x561d57719b10 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.862+0000 7f46faffd640 0 client.15402 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.862+0000 7f4724776740 1 --2- 172.21.15.31:0/2174108888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57711820 0x561d577195d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.862+0000 7f4724776740 1 --2- 172.21.15.31:0/2174108888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d57710f00 0x561d57719090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.862+0000 7f4724776740 1 -- 172.21.15.31:0/2174108888 >> 172.21.15.31:0/2174108888 conn(0x561d5770d410 msgr2=0x561d57713190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.862+0000 7f4724776740 1 -- 172.21.15.31:0/2174108888 shutdown_connections 2024-04-26T17:57:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.862+0000 7f4724776740 1 -- 172.21.15.31:0/2174108888 wait complete. 2024-04-26T17:57:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.863+0000 7f4724776740 20 client.15402 trim_cache size 0 max 16384 2024-04-26T17:57:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.890+0000 7f4724776740 1 Processor -- start 2024-04-26T17:57:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.891+0000 7f4724776740 1 -- start start 2024-04-26T17:57:24.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.891+0000 7f4724776740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d57727b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.891+0000 7f4724776740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d57711d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.891+0000 7f4724776740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 0x561d57733fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.891+0000 7f4724776740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f471c1bb720 con 0x561d57711820 2024-04-26T17:57:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.891+0000 7f4724776740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f471c1b8f90 con 0x561d57710f00 2024-04-26T17:57:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.891+0000 7f4724776740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f471c1b6e00 con 0x561d5771c0e0 2024-04-26T17:57:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.891+0000 7f471b7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d57727b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.891+0000 7f471affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d57711d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.891+0000 7f471b7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d57727b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:53216/0 (socket says 172.21.15.31:53216) 2024-04-26T17:57:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.891+0000 7f471b7fe640 1 -- 172.21.15.31:0/1106143391 learned_addr learned my addr 172.21.15.31:0/1106143391 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.892+0000 7f471bfff640 1 --2- 172.21.15.31:0/1106143391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 0x561d57733fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.892+0000 7f471b7fe640 1 -- 172.21.15.31:0/1106143391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 msgr2=0x561d57733fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.892+0000 7f471b7fe640 1 --2- 172.21.15.31:0/1106143391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 0x561d57733fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.892+0000 7f471b7fe640 1 -- 172.21.15.31:0/1106143391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 msgr2=0x561d57711d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.892+0000 7f471b7fe640 1 --2- 172.21.15.31:0/1106143391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d57711d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.892+0000 7f471b7fe640 1 -- 172.21.15.31:0/1106143391 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d57714c70 con 0x561d57710f00 2024-04-26T17:57:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.892+0000 7f471b7fe640 1 --2- 172.21.15.31:0/1106143391 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d57727b10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f470c09b8c0 tx=0x7f470c010f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=560f379329d7ea03 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.892+0000 7f47017fa640 1 -- 172.21.15.31:0/1106143391 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c017360 con 0x561d57710f00 2024-04-26T17:57:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.892+0000 7f47017fa640 1 -- 172.21.15.31:0/1106143391 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f470c0a4880 con 0x561d57710f00 2024-04-26T17:57:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.892+0000 7f47017fa640 1 -- 172.21.15.31:0/1106143391 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c0d3600 con 0x561d57710f00 2024-04-26T17:57:24.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.893+0000 7f4724776740 1 -- 172.21.15.31:0/1106143391 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 msgr2=0x561d57727b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:24.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.893+0000 7f4724776740 1 --2- 172.21.15.31:0/1106143391 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d57727b10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f470c09b8c0 tx=0x7f470c010f90 comp rx=0 tx=0).stop 2024-04-26T17:57:24.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.894+0000 7f4724776740 1 -- 172.21.15.31:0/1106143391 shutdown_connections 2024-04-26T17:57:24.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.894+0000 7f4724776740 1 --2- 172.21.15.31:0/1106143391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 0x561d57733fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.894+0000 7f4724776740 1 --2- 172.21.15.31:0/1106143391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d57711d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.894+0000 7f4724776740 1 --2- 172.21.15.31:0/1106143391 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d57727b10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.894+0000 7f4724776740 1 -- 172.21.15.31:0/1106143391 >> 172.21.15.31:0/1106143391 conn(0x561d5770d410 msgr2=0x561d5770d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.894+0000 7f4724776740 1 -- 172.21.15.31:0/1106143391 shutdown_connections 2024-04-26T17:57:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.894+0000 7f4724776740 1 -- 172.21.15.31:0/1106143391 wait complete. 2024-04-26T17:57:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f4724776740 1 -- start start 2024-04-26T17:57:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f4724776740 1 Processor -- start 2024-04-26T17:57:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f4724776740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d57724410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f4724776740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d57715a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f4724776740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 0x561d57715fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f4724776740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f471c1b7450 con 0x561d57711820 2024-04-26T17:57:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f4724776740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f471c1b88c0 con 0x561d57710f00 2024-04-26T17:57:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f4724776740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f471c1b7b20 con 0x561d5771c0e0 2024-04-26T17:57:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f4724776740 10 client.0 _ll_register_callbacks cb 0x7f4710005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:57:24.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f471affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d57715a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f471b7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d57724410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f471affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d57715a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57314/0 (socket says 172.21.15.31:57314) 2024-04-26T17:57:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f471b7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d57724410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:53230/0 (socket says 172.21.15.31:53230) 2024-04-26T17:57:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f471affd640 1 -- 172.21.15.31:0/1229294820 learned_addr learned my addr 172.21.15.31:0/1229294820 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.895+0000 7f471bfff640 1 --2- 172.21.15.31:0/1229294820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 0x561d57715fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.896+0000 7f471b7fe640 1 -- 172.21.15.31:0/1229294820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 msgr2=0x561d57715fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.896+0000 7f471b7fe640 1 --2- 172.21.15.31:0/1229294820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 0x561d57715fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.896+0000 7f471b7fe640 1 -- 172.21.15.31:0/1229294820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 msgr2=0x561d57715a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.896+0000 7f471b7fe640 1 --2- 172.21.15.31:0/1229294820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d57715a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.896+0000 7f471b7fe640 1 -- 172.21.15.31:0/1229294820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d57716790 con 0x561d57710f00 2024-04-26T17:57:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.896+0000 7f471affd640 1 --2- 172.21.15.31:0/1229294820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d57715a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.896+0000 7f471b7fe640 1 --2- 172.21.15.31:0/1229294820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d57724410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f470c09b910 tx=0x7f470c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab91abbea300ddbe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.896+0000 7f46faffd640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T17:57:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.896+0000 7f46faffd640 1 -- 172.21.15.31:0/1229294820 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c0b4070 con 0x561d57710f00 2024-04-26T17:57:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.897+0000 7f46faffd640 1 -- 172.21.15.31:0/1229294820 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f470c0d34e0 con 0x561d57710f00 2024-04-26T17:57:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.897+0000 7f4724776740 1 -- 172.21.15.31:0/1229294820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f470c0a4880 con 0x561d57710f00 2024-04-26T17:57:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.897+0000 7f46faffd640 1 -- 172.21.15.31:0/1229294820 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f470c017360 con 0x561d57710f00 2024-04-26T17:57:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.897+0000 7f4724776740 10 client.25130 fetch_fsmap learned FSMap version 29 2024-04-26T17:57:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.897+0000 7f4724776740 1 -- 172.21.15.31:0/1229294820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561d5779b050 con 0x561d57710f00 2024-04-26T17:57:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.897+0000 7f46faffd640 1 -- 172.21.15.31:0/1229294820 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f470c0a4880 con 0x561d57710f00 2024-04-26T17:57:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.897+0000 7f46faffd640 1 -- 172.21.15.31:0/1229294820 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f470c09be70 con 0x561d57710f00 2024-04-26T17:57:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.897+0000 7f4724776740 10 client.25130 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:57:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.897+0000 7f4724776740 10 client.25130 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:57:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.897+0000 7f4724776740 20 client.25130 populate_metadata read hostname 'smithi031' 2024-04-26T17:57:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.897+0000 7f4724776740 1 --2- 172.21.15.31:0/1229294820 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x561d5779b280 0x561d5779d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.897+0000 7f4724776740 4 client.25130 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:57:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.897+0000 7f4724776740 1 -- 172.21.15.31:0/1229294820 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561d5767eee0 con 0x561d5779b280 2024-04-26T17:57:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.898+0000 7f471affd640 1 --2- 172.21.15.31:0/1229294820 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x561d5779b280 0x561d5779d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.898+0000 7f46faffd640 10 client.25130 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.898+0000 7f471affd640 1 --2- 172.21.15.31:0/1229294820 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x561d5779b280 0x561d5779d670 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b07419c86ac6a1d server_cookie=5ed2a282320d2916 in_seq=0 out_seq=0 2024-04-26T17:57:24.909 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:24 smithi031 ceph-mon[29164]: pgmap v30: 97 pgs: 97 active+clean; 804 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 MiB/s wr, 74 op/s 2024-04-26T17:57:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.922+0000 7f46faffd640 1 -- 172.21.15.31:0/1229294820 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561d5767eee0 con 0x561d5779b280 2024-04-26T17:57:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.922+0000 7f46faffd640 10 client.25130 handle_command_reply: tid=0 2024-04-26T17:57:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.922+0000 7f4724776740 1 client.25130 shutdown 2024-04-26T17:57:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4724776740 1 -- 172.21.15.31:0/1229294820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 msgr2=0x561d57724410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4724776740 1 --2- 172.21.15.31:0/1229294820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d57724410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f470c09b910 tx=0x7f470c0046c0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4724776740 1 -- 172.21.15.31:0/1229294820 shutdown_connections 2024-04-26T17:57:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4724776740 1 --2- 172.21.15.31:0/1229294820 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x561d5779b280 0x561d5779d670 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4724776740 1 --2- 172.21.15.31:0/1229294820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d5771c0e0 0x561d57715fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4724776740 1 --2- 172.21.15.31:0/1229294820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d57711820 0x561d57715a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4724776740 1 --2- 172.21.15.31:0/1229294820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561d57710f00 0x561d57724410 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4724776740 1 -- 172.21.15.31:0/1229294820 >> 172.21.15.31:0/1229294820 conn(0x561d5770d410 msgr2=0x561d57711d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f46faffd640 0 client.25130 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4724776740 1 -- 172.21.15.31:0/1229294820 shutdown_connections 2024-04-26T17:57:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4724776740 1 -- 172.21.15.31:0/1229294820 wait complete. 2024-04-26T17:57:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4724776740 20 client.25130 trim_cache size 0 max 16384 2024-04-26T17:57:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:57:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:57:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:57:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:57:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4721a62640 1 -- 172.21.15.31:0/766967318 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f47040a3f60 msgr2=0x7f47040a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4721a62640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f47040a3f60 0x7f47040a6420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4714009430 tx=0x7f47140095c0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4721a62640 1 -- 172.21.15.31:0/766967318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c10d560 msgr2=0x7f471c1a97a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.923+0000 7f4721a62640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c10d560 0x7f471c1a97a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f470c00b4a0 tx=0x7f470c00b960 comp rx=0 tx=0).stop 2024-04-26T17:57:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.924+0000 7f4721a62640 1 -- 172.21.15.31:0/766967318 shutdown_connections 2024-04-26T17:57:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.924+0000 7f4721a62640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f471c1a3950 0x7f471c1a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.924+0000 7f4721a62640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f471c112630 0x7f471c1a9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.924+0000 7f4721a62640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f47040a3f60 0x7f47040a6420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.924+0000 7f4721a62640 1 --2- 172.21.15.31:0/766967318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c10d560 0x7f471c1a97a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.924+0000 7f4721a62640 1 -- 172.21.15.31:0/766967318 >> 172.21.15.31:0/766967318 conn(0x7f471c06db90 msgr2=0x7f471c110790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.924+0000 7f4721a62640 1 -- 172.21.15.31:0/766967318 shutdown_connections 2024-04-26T17:57:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:24.924+0000 7f4721a62640 1 -- 172.21.15.31:0/766967318 wait complete. 2024-04-26T17:57:25.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:24 smithi186 ceph-mon[32797]: pgmap v30: 97 pgs: 97 active+clean; 804 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 MiB/s wr, 74 op/s 2024-04-26T17:57:25.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:24 smithi097 ceph-mon[32419]: pgmap v30: 97 pgs: 97 active+clean; 804 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 MiB/s wr, 74 op/s 2024-04-26T17:57:25.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.368+0000 7f7d3f7fe700 1 -- 172.21.15.31:0/2562061224 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c00bb00 con 0x7f7d4006b9d0 2024-04-26T17:57:25.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.369+0000 7f7d46c1e700 1 -- 172.21.15.31:0/2562061224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d4006b9d0 msgr2=0x7f7d4006be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:25.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.369+0000 7f7d46c1e700 1 --2- 172.21.15.31:0/2562061224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d4006b9d0 0x7f7d4006be30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009990 tx=0x7f7d2c009ca0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.369+0000 7f7d46c1e700 1 -- 172.21.15.31:0/2562061224 shutdown_connections 2024-04-26T17:57:25.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.369+0000 7f7d46c1e700 1 --2- 172.21.15.31:0/2562061224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d4006c370 0x7f7d40110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.369+0000 7f7d46c1e700 1 --2- 172.21.15.31:0/2562061224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d401094c0 0x7f7d401098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.369+0000 7f7d46c1e700 1 --2- 172.21.15.31:0/2562061224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d4006b9d0 0x7f7d4006be30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.369+0000 7f7d46c1e700 1 -- 172.21.15.31:0/2562061224 >> 172.21.15.31:0/2562061224 conn(0x7f7d40079360 msgr2=0x7f7d40079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:25.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.370+0000 7f7d46c1e700 1 -- 172.21.15.31:0/2562061224 shutdown_connections 2024-04-26T17:57:25.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.370+0000 7f7d46c1e700 1 -- 172.21.15.31:0/2562061224 wait complete. 2024-04-26T17:57:25.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.371+0000 7f7d46c1e700 1 Processor -- start 2024-04-26T17:57:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.371+0000 7f7d46c1e700 1 -- start start 2024-04-26T17:57:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.371+0000 7f7d46c1e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d4006b9d0 0x7f7d4019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.371+0000 7f7d46c1e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d4006c370 0x7f7d4019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.371+0000 7f7d46c1e700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d401094c0 0x7f7d40199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.371+0000 7f7d46c1e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d401135b0 con 0x7f7d4006b9d0 2024-04-26T17:57:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.371+0000 7f7d46c1e700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7d40113430 con 0x7f7d401094c0 2024-04-26T17:57:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.371+0000 7f7d46c1e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7d40113730 con 0x7f7d4006c370 2024-04-26T17:57:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.371+0000 7f7d449ba700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d4006b9d0 0x7f7d4019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.371+0000 7f7d449ba700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d4006b9d0 0x7f7d4019f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57342/0 (socket says 172.21.15.31:57342) 2024-04-26T17:57:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.371+0000 7f7d449ba700 1 -- 172.21.15.31:0/1421527245 learned_addr learned my addr 172.21.15.31:0/1421527245 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.371+0000 7f7d451bb700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d401094c0 0x7f7d40199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.371+0000 7f7d3ffff700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d4006c370 0x7f7d4019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.372+0000 7f7d449ba700 1 -- 172.21.15.31:0/1421527245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d4006c370 msgr2=0x7f7d4019fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.372+0000 7f7d449ba700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d4006c370 0x7f7d4019fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.372+0000 7f7d449ba700 1 -- 172.21.15.31:0/1421527245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d401094c0 msgr2=0x7f7d40199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:25.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.372+0000 7f7d449ba700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d401094c0 0x7f7d40199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.372+0000 7f7d449ba700 1 -- 172.21.15.31:0/1421527245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d40199ff0 con 0x7f7d4006b9d0 2024-04-26T17:57:25.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.372+0000 7f7d451bb700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d401094c0 0x7f7d40199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:25.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.372+0000 7f7d449ba700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d4006b9d0 0x7f7d4019f690 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400e9a0 tx=0x7f7d3400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:25.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.372+0000 7f7d3dffb700 1 -- 172.21.15.31:0/1421527245 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3400cbd0 con 0x7f7d4006b9d0 2024-04-26T17:57:25.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.372+0000 7f7d46c1e700 1 -- 172.21.15.31:0/1421527245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d4019a2e0 con 0x7f7d4006b9d0 2024-04-26T17:57:25.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.372+0000 7f7d3dffb700 1 -- 172.21.15.31:0/1421527245 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d3400cd70 con 0x7f7d4006b9d0 2024-04-26T17:57:25.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.372+0000 7f7d3dffb700 1 -- 172.21.15.31:0/1421527245 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d34018680 con 0x7f7d4006b9d0 2024-04-26T17:57:25.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.372+0000 7f7d46c1e700 1 -- 172.21.15.31:0/1421527245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d401a64f0 con 0x7f7d4006b9d0 2024-04-26T17:57:25.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.376+0000 7f7d3dffb700 1 -- 172.21.15.31:0/1421527245 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f7d34010430 con 0x7f7d4006b9d0 2024-04-26T17:57:25.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.377+0000 7f7d3dffb700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7d300a3f70 0x7f7d300a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.377+0000 7f7d3dffb700 1 -- 172.21.15.31:0/1421527245 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7d340d0c90 con 0x7f7d4006b9d0 2024-04-26T17:57:25.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.377+0000 7f7d3ffff700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7d300a3f70 0x7f7d300a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.378+0000 7f7d3ffff700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7d300a3f70 0x7f7d300a6430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c006010 tx=0x7f7d2c01a040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:25.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.379+0000 7f7d514b0b80 1 Processor -- start 2024-04-26T17:57:25.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.379+0000 7f7d514b0b80 1 -- start start 2024-04-26T17:57:25.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.379+0000 7f7d514b0b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0168d0 0x563bdb016cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.379+0000 7f7d514b0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb0171f0 0x563bdb01af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.379+0000 7f7d514b0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb01b4b0 0x563bdb01b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.379+0000 7f7d514b0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563bdaf649c0 con 0x563bdb0171f0 2024-04-26T17:57:25.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.379+0000 7f7d514b0b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x563bdaecaac0 con 0x563bdb0168d0 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.379+0000 7f7d514b0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563bdaf60980 con 0x563bdb01b4b0 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.379+0000 7f7d3ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb0171f0 0x563bdb01af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.379+0000 7f7d3ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb0171f0 0x563bdb01af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57346/0 (socket says 172.21.15.31:57346) 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.379+0000 7f7d3ffff700 1 -- 172.21.15.31:0/3191100222 learned_addr learned my addr 172.21.15.31:0/3191100222 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.380+0000 7f7d449ba700 1 --2- 172.21.15.31:0/3191100222 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0168d0 0x563bdb016cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.380+0000 7f7d451bb700 1 --2- 172.21.15.31:0/3191100222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb01b4b0 0x563bdb01b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.380+0000 7f7d3ffff700 1 -- 172.21.15.31:0/3191100222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb01b4b0 msgr2=0x563bdb01b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.380+0000 7f7d3ffff700 1 --2- 172.21.15.31:0/3191100222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb01b4b0 0x563bdb01b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.380+0000 7f7d3ffff700 1 -- 172.21.15.31:0/3191100222 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0168d0 msgr2=0x563bdb016cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.380+0000 7f7d3ffff700 1 --2- 172.21.15.31:0/3191100222 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0168d0 0x563bdb016cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.380+0000 7f7d3ffff700 1 -- 172.21.15.31:0/3191100222 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bdaeb1a60 con 0x563bdb0171f0 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.380+0000 7f7d3ffff700 1 --2- 172.21.15.31:0/3191100222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb0171f0 0x563bdb01af70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009750 tx=0x7f7d2c01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170930db5ca8cb3b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.380+0000 7f7d2affd700 1 -- 172.21.15.31:0/3191100222 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c00f870 con 0x563bdb0171f0 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.380+0000 7f7d2affd700 1 -- 172.21.15.31:0/3191100222 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d2c032070 con 0x563bdb0171f0 2024-04-26T17:57:25.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.380+0000 7f7d2affd700 1 -- 172.21.15.31:0/3191100222 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c023d70 con 0x563bdb0171f0 2024-04-26T17:57:25.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.381+0000 7f7d514b0b80 1 -- 172.21.15.31:0/3191100222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb0171f0 msgr2=0x563bdb01af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:25.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.381+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/3191100222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb0171f0 0x563bdb01af70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009750 tx=0x7f7d2c01e000 comp rx=0 tx=0).stop 2024-04-26T17:57:25.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.381+0000 7f7d514b0b80 1 -- 172.21.15.31:0/3191100222 shutdown_connections 2024-04-26T17:57:25.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.381+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/3191100222 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0168d0 0x563bdb016cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.381+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/3191100222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb01b4b0 0x563bdb01b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.381+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/3191100222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb0171f0 0x563bdb01af70 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.381+0000 7f7d514b0b80 1 -- 172.21.15.31:0/3191100222 >> 172.21.15.31:0/3191100222 conn(0x563bdaf40750 msgr2=0x563bdb00d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:25.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.381+0000 7f7d514b0b80 1 -- 172.21.15.31:0/3191100222 shutdown_connections 2024-04-26T17:57:25.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.381+0000 7f7d514b0b80 1 -- 172.21.15.31:0/3191100222 wait complete. 2024-04-26T17:57:25.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.382+0000 7f7d514b0b80 1 -- start start 2024-04-26T17:57:25.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.383+0000 7f7d514b0b80 1 Processor -- start 2024-04-26T17:57:25.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.383+0000 7f7d514b0b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0168d0 0x563bdb022920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d514b0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb0171f0 0x563bdb01fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d514b0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb01b4b0 0x563bdb020200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d514b0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563bdb023680 con 0x563bdb01b4b0 2024-04-26T17:57:25.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d449ba700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0168d0 0x563bdb022920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d514b0b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x563bdaf1ec10 con 0x563bdb0168d0 2024-04-26T17:57:25.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d451bb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb01b4b0 0x563bdb020200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d449ba700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0168d0 0x563bdb022920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:53264/0 (socket says 172.21.15.31:53264) 2024-04-26T17:57:25.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d449ba700 1 -- 172.21.15.31:0/1910705151 learned_addr learned my addr 172.21.15.31:0/1910705151 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:25.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d3ffff700 1 --2- 172.21.15.31:0/1910705151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb0171f0 0x563bdb01fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d514b0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563bdb023800 con 0x563bdb0171f0 2024-04-26T17:57:25.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d514b0b80 10 client.0 _ll_register_callbacks cb 0x563bdaefcae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:57:25.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d449ba700 1 -- 172.21.15.31:0/1910705151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb0171f0 msgr2=0x563bdb01fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:25.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d449ba700 1 --2- 172.21.15.31:0/1910705151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb0171f0 0x563bdb01fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d449ba700 1 -- 172.21.15.31:0/1910705151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb01b4b0 msgr2=0x563bdb020200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:25.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d449ba700 1 --2- 172.21.15.31:0/1910705151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb01b4b0 0x563bdb020200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d449ba700 1 -- 172.21.15.31:0/1910705151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bdaf22610 con 0x563bdb0168d0 2024-04-26T17:57:25.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.384+0000 7f7d449ba700 1 --2- 172.21.15.31:0/1910705151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0168d0 0x563bdb022920 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d34099400 tx=0x7f7d3409cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6e1b44a5c83cad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:25.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d1ffff700 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T17:57:25.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d1ffff700 1 -- 172.21.15.31:0/1910705151 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d340bd0b0 con 0x563bdb0168d0 2024-04-26T17:57:25.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d1ffff700 1 -- 172.21.15.31:0/1910705151 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7d34010720 con 0x563bdb0168d0 2024-04-26T17:57:25.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d514b0b80 1 -- 172.21.15.31:0/1910705151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d2c032070 con 0x563bdb0168d0 2024-04-26T17:57:25.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d1ffff700 1 -- 172.21.15.31:0/1910705151 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7d3409d130 con 0x563bdb0168d0 2024-04-26T17:57:25.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d514b0b80 10 client.25142 fetch_fsmap learned FSMap version 29 2024-04-26T17:57:25.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d514b0b80 1 -- 172.21.15.31:0/1910705151 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563bdb02ded0 con 0x563bdb0168d0 2024-04-26T17:57:25.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d1ffff700 1 -- 172.21.15.31:0/1910705151 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d2c032070 con 0x563bdb0168d0 2024-04-26T17:57:25.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d1ffff700 1 -- 172.21.15.31:0/1910705151 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f7d340ceb80 con 0x563bdb0168d0 2024-04-26T17:57:25.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d514b0b80 10 client.25142 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:57:25.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d514b0b80 10 client.25142 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T17:57:25.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d514b0b80 20 client.25142 populate_metadata read hostname 'smithi031' 2024-04-26T17:57:25.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/1910705151 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563bdb030460 0x563bdb032850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d514b0b80 4 client.25142 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:57:25.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.385+0000 7f7d514b0b80 1 -- 172.21.15.31:0/1910705151 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563bdaf723a0 con 0x563bdb030460 2024-04-26T17:57:25.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.386+0000 7f7d3ffff700 1 --2- 172.21.15.31:0/1910705151 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563bdb030460 0x563bdb032850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.386+0000 7f7d3ffff700 1 --2- 172.21.15.31:0/1910705151 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563bdb030460 0x563bdb032850 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cfcc25998865688 server_cookie=ca1a535de44892f8 in_seq=0 out_seq=0 2024-04-26T17:57:25.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.386+0000 7f7d1ffff700 10 client.25142 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:25.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.387+0000 7f7d1ffff700 1 -- 172.21.15.31:0/1910705151 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563bdaf723a0 con 0x563bdb030460 2024-04-26T17:57:25.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.387+0000 7f7d1ffff700 10 client.25142 handle_command_reply: tid=0 2024-04-26T17:57:25.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.387+0000 7f7d514b0b80 1 client.25142 shutdown 2024-04-26T17:57:25.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.387+0000 7f7d514b0b80 1 -- 172.21.15.31:0/1910705151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0168d0 msgr2=0x563bdb022920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:25.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.387+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/1910705151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0168d0 0x563bdb022920 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d34099400 tx=0x7f7d3409cf70 comp rx=0 tx=0).stop 2024-04-26T17:57:25.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.388+0000 7f7d514b0b80 1 -- 172.21.15.31:0/1910705151 shutdown_connections 2024-04-26T17:57:25.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.388+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/1910705151 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563bdb030460 0x563bdb032850 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.388+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/1910705151 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0168d0 0x563bdb022920 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d34099400 tx=0x7f7d3409cf70 comp rx=0 tx=0).stop 2024-04-26T17:57:25.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.388+0000 7f7d1ffff700 0 client.25142 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:25.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.388+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/1910705151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb01b4b0 0x563bdb020200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.389+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/1910705151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb0171f0 0x563bdb01fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.389+0000 7f7d514b0b80 1 -- 172.21.15.31:0/1910705151 >> 172.21.15.31:0/1910705151 conn(0x563bdaf40750 msgr2=0x563bdb006d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:25.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.389+0000 7f7d514b0b80 1 -- 172.21.15.31:0/1910705151 shutdown_connections 2024-04-26T17:57:25.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.389+0000 7f7d514b0b80 1 -- 172.21.15.31:0/1910705151 wait complete. 2024-04-26T17:57:25.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.389+0000 7f7d514b0b80 20 client.25142 trim_cache size 0 max 16384 2024-04-26T17:57:25.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.420+0000 7f7d514b0b80 1 Processor -- start 2024-04-26T17:57:25.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.420+0000 7f7d514b0b80 1 -- start start 2024-04-26T17:57:25.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.421+0000 7f7d514b0b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0171f0 0x563bdb006d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.421+0000 7f7d514b0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb01b4b0 0x563bdb011d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.421+0000 7f7d514b0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb030460 0x563bdb02c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.421+0000 7f7d449ba700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0171f0 0x563bdb006d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.421+0000 7f7d514b0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d401ae1e0 con 0x563bdb030460 2024-04-26T17:57:25.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.421+0000 7f7d514b0b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7d401aba10 con 0x563bdb0171f0 2024-04-26T17:57:25.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.421+0000 7f7d514b0b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7d401a9860 con 0x563bdb01b4b0 2024-04-26T17:57:25.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.421+0000 7f7d3ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb01b4b0 0x563bdb011d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.421+0000 7f7d449ba700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0171f0 0x563bdb006d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:53278/0 (socket says 172.21.15.31:53278) 2024-04-26T17:57:25.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.421+0000 7f7d449ba700 1 -- 172.21.15.31:0/3570385069 learned_addr learned my addr 172.21.15.31:0/3570385069 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:25.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.421+0000 7f7d451bb700 1 --2- 172.21.15.31:0/3570385069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb030460 0x563bdb02c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.422+0000 7f7d3ffff700 1 -- 172.21.15.31:0/3570385069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0171f0 msgr2=0x563bdb006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:25.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.422+0000 7f7d3ffff700 1 --2- 172.21.15.31:0/3570385069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0171f0 0x563bdb006d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.422+0000 7f7d3ffff700 1 -- 172.21.15.31:0/3570385069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb030460 msgr2=0x563bdb02c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:25.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.422+0000 7f7d3ffff700 1 --2- 172.21.15.31:0/3570385069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb030460 0x563bdb02c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.422+0000 7f7d3ffff700 1 -- 172.21.15.31:0/3570385069 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bdb0205b0 con 0x563bdb01b4b0 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.422+0000 7f7d3ffff700 1 --2- 172.21.15.31:0/3570385069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb01b4b0 0x563bdb011d10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c023950 tx=0x7f7d2c01e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c085370c51b0636f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.423+0000 7f7d2a7fc700 1 -- 172.21.15.31:0/3570385069 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c043070 con 0x563bdb01b4b0 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.423+0000 7f7d2a7fc700 1 -- 172.21.15.31:0/3570385069 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d2c03b030 con 0x563bdb01b4b0 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.423+0000 7f7d2a7fc700 1 -- 172.21.15.31:0/3570385069 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c00fad0 con 0x563bdb01b4b0 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.424+0000 7f7d514b0b80 1 -- 172.21.15.31:0/3570385069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb01b4b0 msgr2=0x563bdb011d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.424+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/3570385069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb01b4b0 0x563bdb011d10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c023950 tx=0x7f7d2c01e000 comp rx=0 tx=0).stop 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.424+0000 7f7d514b0b80 1 -- 172.21.15.31:0/3570385069 shutdown_connections 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.424+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/3570385069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0171f0 0x563bdb006d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.424+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/3570385069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb030460 0x563bdb02c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.424+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/3570385069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb01b4b0 0x563bdb011d10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.424+0000 7f7d514b0b80 1 -- 172.21.15.31:0/3570385069 >> 172.21.15.31:0/3570385069 conn(0x563bdaf40750 msgr2=0x563bdb021c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.424+0000 7f7d514b0b80 1 -- 172.21.15.31:0/3570385069 shutdown_connections 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.424+0000 7f7d514b0b80 1 -- 172.21.15.31:0/3570385069 wait complete. 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.424+0000 7f7d514b0b80 1 -- start start 2024-04-26T17:57:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.425+0000 7f7d514b0b80 1 Processor -- start 2024-04-26T17:57:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.425+0000 7f7d514b0b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0171f0 0x563bdb025840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.425+0000 7f7d514b0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb01b4b0 0x563bdb025d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.425+0000 7f7d514b0b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb030460 0x563bdb096930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.425+0000 7f7d449ba700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0171f0 0x563bdb025840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.425+0000 7f7d3ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb01b4b0 0x563bdb025d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.425+0000 7f7d449ba700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0171f0 0x563bdb025840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:53280/0 (socket says 172.21.15.31:53280) 2024-04-26T17:57:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.425+0000 7f7d449ba700 1 -- 172.21.15.31:0/2384613940 learned_addr learned my addr 172.21.15.31:0/2384613940 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.425+0000 7f7d514b0b80 1 -- 172.21.15.31:0/2384613940 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d401a9ed0 con 0x563bdb01b4b0 2024-04-26T17:57:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.425+0000 7f7d514b0b80 1 -- 172.21.15.31:0/2384613940 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7d401ab340 con 0x563bdb0171f0 2024-04-26T17:57:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.425+0000 7f7d514b0b80 1 -- 172.21.15.31:0/2384613940 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7d401aa5a0 con 0x563bdb030460 2024-04-26T17:57:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.425+0000 7f7d514b0b80 10 client.0 _ll_register_callbacks cb 0x563bdaefcae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:57:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d451bb700 1 --2- 172.21.15.31:0/2384613940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb030460 0x563bdb096930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d449ba700 1 -- 172.21.15.31:0/2384613940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb030460 msgr2=0x563bdb096930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d449ba700 1 --2- 172.21.15.31:0/2384613940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb030460 0x563bdb096930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d449ba700 1 -- 172.21.15.31:0/2384613940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb01b4b0 msgr2=0x563bdb025d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d449ba700 1 --2- 172.21.15.31:0/2384613940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb01b4b0 0x563bdb025d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d449ba700 1 -- 172.21.15.31:0/2384613940 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bdb096e70 con 0x563bdb0171f0 2024-04-26T17:57:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d449ba700 1 --2- 172.21.15.31:0/2384613940 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0171f0 0x563bdb025840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7d3409c9c0 tx=0x7f7d3400ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=635cc28eb060d6db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d1ffff700 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T17:57:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d1ffff700 1 -- 172.21.15.31:0/2384613940 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d34010920 con 0x563bdb0171f0 2024-04-26T17:57:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d1ffff700 1 -- 172.21.15.31:0/2384613940 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7d3409be80 con 0x563bdb0171f0 2024-04-26T17:57:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d514b0b80 1 -- 172.21.15.31:0/2384613940 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d2c03b030 con 0x563bdb0171f0 2024-04-26T17:57:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d1ffff700 1 -- 172.21.15.31:0/2384613940 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7d340ce1f0 con 0x563bdb0171f0 2024-04-26T17:57:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d514b0b80 10 client.25148 fetch_fsmap learned FSMap version 29 2024-04-26T17:57:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d514b0b80 1 -- 172.21.15.31:0/2384613940 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563bdb0122e0 con 0x563bdb0171f0 2024-04-26T17:57:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.426+0000 7f7d1ffff700 1 -- 172.21.15.31:0/2384613940 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d2c03b030 con 0x563bdb0171f0 2024-04-26T17:57:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.428+0000 7f7d1ffff700 1 -- 172.21.15.31:0/2384613940 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f7d340672c0 con 0x563bdb0171f0 2024-04-26T17:57:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.428+0000 7f7d514b0b80 10 client.25148 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:57:25.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.428+0000 7f7d514b0b80 10 client.25148 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T17:57:25.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.428+0000 7f7d514b0b80 20 client.25148 populate_metadata read hostname 'smithi031' 2024-04-26T17:57:25.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.428+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/2384613940 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563bdb012a30 0x563bdb014e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:25.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.428+0000 7f7d514b0b80 4 client.25148 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-26T17:57:25.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.428+0000 7f7d514b0b80 1 -- 172.21.15.31:0/2384613940 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x563bdaf6c440 con 0x563bdb012a30 2024-04-26T17:57:25.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.428+0000 7f7d3ffff700 1 --2- 172.21.15.31:0/2384613940 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563bdb012a30 0x563bdb014e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:25.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.429+0000 7f7d3ffff700 1 --2- 172.21.15.31:0/2384613940 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563bdb012a30 0x563bdb014e50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e984f29b08973379 server_cookie=a099996105e5aaf7 in_seq=0 out_seq=0 2024-04-26T17:57:25.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:25.429+0000 7f7d1ffff700 10 client.25148 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:26.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d1ffff700 1 -- 172.21.15.31:0/2384613940 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x563bdaf6c440 con 0x563bdb012a30 2024-04-26T17:57:26.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d1ffff700 10 client.25148 handle_command_reply: tid=0 2024-04-26T17:57:26.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d514b0b80 1 client.25148 shutdown 2024-04-26T17:57:26.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d514b0b80 1 -- 172.21.15.31:0/2384613940 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0171f0 msgr2=0x563bdb025840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:26.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/2384613940 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0171f0 0x563bdb025840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7d3409c9c0 tx=0x7f7d3400ef80 comp rx=0 tx=0).stop 2024-04-26T17:57:26.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d514b0b80 1 -- 172.21.15.31:0/2384613940 shutdown_connections 2024-04-26T17:57:26.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/2384613940 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563bdb012a30 0x563bdb014e50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:26.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/2384613940 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563bdb0171f0 0x563bdb025840 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:26.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/2384613940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563bdb030460 0x563bdb096930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:26.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d514b0b80 1 --2- 172.21.15.31:0/2384613940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563bdb01b4b0 0x563bdb025d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d1ffff700 0 client.25148 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d514b0b80 1 -- 172.21.15.31:0/2384613940 >> 172.21.15.31:0/2384613940 conn(0x563bdaf40750 msgr2=0x563bdb01f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d514b0b80 1 -- 172.21.15.31:0/2384613940 shutdown_connections 2024-04-26T17:57:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d514b0b80 1 -- 172.21.15.31:0/2384613940 wait complete. 2024-04-26T17:57:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.064+0000 7f7d514b0b80 20 client.25148 trim_cache size 0 max 16384 2024-04-26T17:57:26.066 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T17:57:26.066 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 29, 2024-04-26T17:57:26.066 INFO:teuthology.orchestra.run.smithi031.stdout: "leader": 24473, 2024-04-26T17:57:26.067 INFO:teuthology.orchestra.run.smithi031.stdout: "set_version": 2, 2024-04-26T17:57:26.067 INFO:teuthology.orchestra.run.smithi031.stdout: "sets": { 2024-04-26T17:57:26.067 INFO:teuthology.orchestra.run.smithi031.stdout: "10d0f28a": { 2024-04-26T17:57:26.067 INFO:teuthology.orchestra.run.smithi031.stdout: "version": 2, 2024-04-26T17:57:26.067 INFO:teuthology.orchestra.run.smithi031.stdout: "age_ref": 0.0, 2024-04-26T17:57:26.067 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T17:57:26.067 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "QUIESCED", 2024-04-26T17:57:26.067 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T17:57:26.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T17:57:26.067 INFO:teuthology.orchestra.run.smithi031.stdout: "timeout": 60.0, 2024-04-26T17:57:26.067 INFO:teuthology.orchestra.run.smithi031.stdout: "expiration": 90.0, 2024-04-26T17:57:26.067 INFO:teuthology.orchestra.run.smithi031.stdout: "members": { 2024-04-26T17:57:26.067 INFO:teuthology.orchestra.run.smithi031.stdout: "file:/": { 2024-04-26T17:57:26.067 INFO:teuthology.orchestra.run.smithi031.stdout: "excluded": false, 2024-04-26T17:57:26.068 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T17:57:26.068 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "QUIESCED", 2024-04-26T17:57:26.068 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T17:57:26.068 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:57:26.068 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:57:26.068 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:57:26.068 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:57:26.068 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:57:26.068 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T17:57:26.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.065+0000 7f7d46c1e700 1 -- 172.21.15.31:0/1421527245 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7d300a3f70 msgr2=0x7f7d300a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:26.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.065+0000 7f7d46c1e700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7d300a3f70 0x7f7d300a6430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c006010 tx=0x7f7d2c01a040 comp rx=0 tx=0).stop 2024-04-26T17:57:26.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.065+0000 7f7d46c1e700 1 -- 172.21.15.31:0/1421527245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d4006b9d0 msgr2=0x7f7d4019f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:26.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.065+0000 7f7d46c1e700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d4006b9d0 0x7f7d4019f690 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400e9a0 tx=0x7f7d3400ecb0 comp rx=0 tx=0).stop 2024-04-26T17:57:26.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.065+0000 7f7d46c1e700 1 -- 172.21.15.31:0/1421527245 shutdown_connections 2024-04-26T17:57:26.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.065+0000 7f7d46c1e700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7d300a3f70 0x7f7d300a6430 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:26.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.065+0000 7f7d46c1e700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d401094c0 0x7f7d40199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:26.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.065+0000 7f7d46c1e700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d4006b9d0 0x7f7d4019f690 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:26.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.065+0000 7f7d46c1e700 1 --2- 172.21.15.31:0/1421527245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d4006c370 0x7f7d4019fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:26.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.065+0000 7f7d46c1e700 1 -- 172.21.15.31:0/1421527245 >> 172.21.15.31:0/1421527245 conn(0x7f7d40079360 msgr2=0x7f7d40100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:26.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.065+0000 7f7d46c1e700 1 -- 172.21.15.31:0/1421527245 shutdown_connections 2024-04-26T17:57:26.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:26.065+0000 7f7d46c1e700 1 -- 172.21.15.31:0/1421527245 wait complete. 2024-04-26T17:57:26.791 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '10d0f28a', quiesce took 7.1 seconds. Will release after: 30.0 2024-04-26T17:57:27.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:26 smithi031 ceph-mon[29164]: pgmap v31: 97 pgs: 97 active+clean; 941 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 51 MiB/s wr, 205 op/s 2024-04-26T17:57:27.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:26 smithi186 ceph-mon[32797]: pgmap v31: 97 pgs: 97 active+clean; 941 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 51 MiB/s wr, 205 op/s 2024-04-26T17:57:27.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:26 smithi097 ceph-mon[32419]: pgmap v31: 97 pgs: 97 active+clean; 941 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 51 MiB/s wr, 205 op/s 2024-04-26T17:57:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 5.0 s, 1565.1 tps, lat 29.323 ms stddev 30.590 2024-04-26T17:57:29.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:28 smithi186 ceph-mon[32797]: pgmap v32: 97 pgs: 97 active+clean; 954 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 53 MiB/s wr, 150 op/s 2024-04-26T17:57:29.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:28 smithi031 ceph-mon[29164]: pgmap v32: 97 pgs: 97 active+clean; 954 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 53 MiB/s wr, 150 op/s 2024-04-26T17:57:29.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:28 smithi097 ceph-mon[32419]: pgmap v32: 97 pgs: 97 active+clean; 954 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 53 MiB/s wr, 150 op/s 2024-04-26T17:57:31.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:30 smithi186 ceph-mon[32797]: pgmap v33: 97 pgs: 97 active+clean; 954 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 53 MiB/s wr, 144 op/s 2024-04-26T17:57:31.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:30 smithi031 ceph-mon[29164]: pgmap v33: 97 pgs: 97 active+clean; 954 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 53 MiB/s wr, 144 op/s 2024-04-26T17:57:31.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:30 smithi097 ceph-mon[32419]: pgmap v33: 97 pgs: 97 active+clean; 954 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 53 MiB/s wr, 144 op/s 2024-04-26T17:57:33.099 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 10.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T17:57:33.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:32 smithi186 ceph-mon[32797]: pgmap v34: 97 pgs: 97 active+clean; 954 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 53 MiB/s wr, 179 op/s 2024-04-26T17:57:33.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:32 smithi031 ceph-mon[29164]: pgmap v34: 97 pgs: 97 active+clean; 954 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 53 MiB/s wr, 179 op/s 2024-04-26T17:57:33.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:32 smithi097 ceph-mon[32419]: pgmap v34: 97 pgs: 97 active+clean; 954 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 53 MiB/s wr, 179 op/s 2024-04-26T17:57:35.186 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:34 smithi097 ceph-mon[32419]: pgmap v35: 97 pgs: 97 active+clean; 954 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 19 MiB/s wr, 167 op/s 2024-04-26T17:57:35.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:34 smithi186 ceph-mon[32797]: pgmap v35: 97 pgs: 97 active+clean; 954 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 19 MiB/s wr, 167 op/s 2024-04-26T17:57:35.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:34 smithi031 ceph-mon[29164]: pgmap v35: 97 pgs: 97 active+clean; 954 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 19 MiB/s wr, 167 op/s 2024-04-26T17:57:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:35 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:57:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:35 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:57:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:35 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:57:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:35 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:57:36.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:35 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:57:36.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:35 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:57:36.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:35 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:57:36.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:35 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:57:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:35 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:57:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:35 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:57:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:35 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:57:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:35 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:57:37.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:36 smithi186 ceph-mon[32797]: pgmap v36: 97 pgs: 97 active+clean; 955 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 19 MiB/s wr, 172 op/s 2024-04-26T17:57:37.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:36 smithi031 ceph-mon[29164]: pgmap v36: 97 pgs: 97 active+clean; 955 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 19 MiB/s wr, 172 op/s 2024-04-26T17:57:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:36 smithi097 ceph-mon[32419]: pgmap v36: 97 pgs: 97 active+clean; 955 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 19 MiB/s wr, 172 op/s 2024-04-26T17:57:38.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.026+0000 7fc73cc09640 1 -- 172.21.15.31:0/2542810623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7381032f0 msgr2=0x7fc73810f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:38.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.026+0000 7fc73cc09640 1 --2- 172.21.15.31:0/2542810623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7381032f0 0x7fc73810f800 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc730009cd0 tx=0x7fc73002a8e0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.027+0000 7fc73cc09640 1 -- 172.21.15.31:0/2542810623 shutdown_connections 2024-04-26T17:57:38.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.027+0000 7fc73cc09640 1 --2- 172.21.15.31:0/2542810623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7381032f0 0x7fc73810f800 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.027+0000 7fc73cc09640 1 --2- 172.21.15.31:0/2542810623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc738102930 0x7fc738102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.027+0000 7fc73cc09640 1 --2- 172.21.15.31:0/2542810623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc738108940 0x7fc738108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.027+0000 7fc73cc09640 1 -- 172.21.15.31:0/2542810623 >> 172.21.15.31:0/2542810623 conn(0x7fc7380fe400 msgr2=0x7fc738100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:38.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.027+0000 7fc73cc09640 1 -- 172.21.15.31:0/2542810623 shutdown_connections 2024-04-26T17:57:38.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.027+0000 7fc73cc09640 1 -- 172.21.15.31:0/2542810623 wait complete. 2024-04-26T17:57:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.027+0000 7fc73cc09640 1 Processor -- start 2024-04-26T17:57:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.027+0000 7fc73cc09640 1 -- start start 2024-04-26T17:57:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc73cc09640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc738102930 0x7fc738074c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc73cc09640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7381032f0 0x7fc738075170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc73cc09640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc738108940 0x7fc7380756b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc73cc09640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc738112c20 con 0x7fc738108940 2024-04-26T17:57:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc73cc09640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc738112aa0 con 0x7fc7381032f0 2024-04-26T17:57:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc73cc09640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc738112da0 con 0x7fc738102930 2024-04-26T17:57:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc736d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc738108940 0x7fc7380756b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc736d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc738108940 0x7fc7380756b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50644/0 (socket says 172.21.15.31:50644) 2024-04-26T17:57:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc736d9d640 1 -- 172.21.15.31:0/1206150350 learned_addr learned my addr 172.21.15.31:0/1206150350 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc735d9b640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7381032f0 0x7fc738075170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc73659c640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc738102930 0x7fc738074c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc736d9d640 1 -- 172.21.15.31:0/1206150350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc738102930 msgr2=0x7fc738074c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc736d9d640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc738102930 0x7fc738074c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc736d9d640 1 -- 172.21.15.31:0/1206150350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7381032f0 msgr2=0x7fc738075170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc736d9d640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7381032f0 0x7fc738075170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc736d9d640 1 -- 172.21.15.31:0/1206150350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7381a7580 con 0x7fc738108940 2024-04-26T17:57:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc73659c640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc738102930 0x7fc738074c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:57:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.028+0000 7fc736d9d640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc738108940 0x7fc7380756b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc730005e30 tx=0x7fc730003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.029+0000 7fc7277fe640 1 -- 172.21.15.31:0/1206150350 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc730004240 con 0x7fc738108940 2024-04-26T17:57:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.029+0000 7fc73cc09640 1 -- 172.21.15.31:0/1206150350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7381a7810 con 0x7fc738108940 2024-04-26T17:57:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.029+0000 7fc7277fe640 1 -- 172.21.15.31:0/1206150350 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc730031d00 con 0x7fc738108940 2024-04-26T17:57:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.029+0000 7fc7277fe640 1 -- 172.21.15.31:0/1206150350 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc730030e30 con 0x7fc738108940 2024-04-26T17:57:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.029+0000 7fc73cc09640 1 -- 172.21.15.31:0/1206150350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7381a7db0 con 0x7fc738108940 2024-04-26T17:57:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.031+0000 7fc7277fe640 1 -- 172.21.15.31:0/1206150350 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fc7300305a0 con 0x7fc738108940 2024-04-26T17:57:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.031+0000 7fc73f9ff740 1 Processor -- start 2024-04-26T17:57:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.031+0000 7fc73f9ff740 1 -- start start 2024-04-26T17:57:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc7277fe640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc7200a3f10 0x7fc7200a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc73f9ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d57f00 0x55f573d582e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc73f9ff740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d58820 0x55f573d62ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc73f9ff740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d630e0 0x55f573d654d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc73f9ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f573cbe460 con 0x55f573d57f00 2024-04-26T17:57:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc73f9ff740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55f573c78590 con 0x55f573d58820 2024-04-26T17:57:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc73f9ff740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f573cba420 con 0x55f573d630e0 2024-04-26T17:57:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc735d9b640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc7200a3f10 0x7fc7200a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc73659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d57f00 0x55f573d582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc73659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d57f00 0x55f573d582e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50648/0 (socket says 172.21.15.31:50648) 2024-04-26T17:57:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc73659c640 1 -- 172.21.15.31:0/4140856708 learned_addr learned my addr 172.21.15.31:0/4140856708 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc7277fe640 1 -- 172.21.15.31:0/1206150350 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fc73002e030 con 0x7fc738108940 2024-04-26T17:57:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc736d9d640 1 --2- 172.21.15.31:0/4140856708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d630e0 0x55f573d654d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc735d9b640 1 --2- 172.21.15.31:0/4140856708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d58820 0x55f573d62ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc73659c640 1 -- 172.21.15.31:0/4140856708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d630e0 msgr2=0x55f573d654d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc73659c640 1 --2- 172.21.15.31:0/4140856708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d630e0 0x55f573d654d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc73659c640 1 -- 172.21.15.31:0/4140856708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d58820 msgr2=0x55f573d62ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc73659c640 1 --2- 172.21.15.31:0/4140856708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d58820 0x55f573d62ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc73659c640 1 -- 172.21.15.31:0/4140856708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f573c850d0 con 0x55f573d57f00 2024-04-26T17:57:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.032+0000 7fc735d9b640 1 --2- 172.21.15.31:0/4140856708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d58820 0x55f573d62ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:57:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.033+0000 7fc735d9b640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc7200a3f10 0x7fc7200a63d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc738076670 tx=0x7fc72c013880 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.033+0000 7fc73659c640 1 --2- 172.21.15.31:0/4140856708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d57f00 0x55f573d582e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc72800dc40 tx=0x7fc72800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44480fc09d1f920 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.033+0000 7fc724ff9640 1 -- 172.21.15.31:0/4140856708 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc728005020 con 0x55f573d57f00 2024-04-26T17:57:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.033+0000 7fc724ff9640 1 -- 172.21.15.31:0/4140856708 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7280047a0 con 0x55f573d57f00 2024-04-26T17:57:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.033+0000 7fc724ff9640 1 -- 172.21.15.31:0/4140856708 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc728011640 con 0x55f573d57f00 2024-04-26T17:57:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 -- 172.21.15.31:0/4140856708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d57f00 msgr2=0x55f573d582e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/4140856708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d57f00 0x55f573d582e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc72800dc40 tx=0x7fc72800f730 comp rx=0 tx=0).stop 2024-04-26T17:57:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 -- 172.21.15.31:0/4140856708 shutdown_connections 2024-04-26T17:57:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/4140856708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d630e0 0x55f573d654d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/4140856708 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d58820 0x55f573d62ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/4140856708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d57f00 0x55f573d582e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 -- 172.21.15.31:0/4140856708 >> 172.21.15.31:0/4140856708 conn(0x55f573d54410 msgr2=0x55f573d547f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 -- 172.21.15.31:0/4140856708 shutdown_connections 2024-04-26T17:57:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 -- 172.21.15.31:0/4140856708 wait complete. 2024-04-26T17:57:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 -- start start 2024-04-26T17:57:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 Processor -- start 2024-04-26T17:57:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d57f00 0x55f573d5fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d58820 0x55f573d60430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 0x55f573d60970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f573d65d10 con 0x55f573d58820 2024-04-26T17:57:38.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55f573c45b60 con 0x55f573d630e0 2024-04-26T17:57:38.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f573d65e90 con 0x55f573d57f00 2024-04-26T17:57:38.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.034+0000 7fc73f9ff740 10 client.0 _ll_register_callbacks cb 0x55f573c6a2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:57:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc735d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d58820 0x55f573d60430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc735d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d58820 0x55f573d60430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50654/0 (socket says 172.21.15.31:50654) 2024-04-26T17:57:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc735d9b640 1 -- 172.21.15.31:0/3651035917 learned_addr learned my addr 172.21.15.31:0/3651035917 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc73659c640 1 --2- 172.21.15.31:0/3651035917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d57f00 0x55f573d5fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc736d9d640 1 --2- 172.21.15.31:0/3651035917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 0x55f573d60970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc735d9b640 1 -- 172.21.15.31:0/3651035917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d57f00 msgr2=0x55f573d5fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc735d9b640 1 --2- 172.21.15.31:0/3651035917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d57f00 0x55f573d5fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc735d9b640 1 -- 172.21.15.31:0/3651035917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 msgr2=0x55f573d60970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc735d9b640 1 --2- 172.21.15.31:0/3651035917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 0x55f573d60970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc735d9b640 1 -- 172.21.15.31:0/3651035917 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5739d0210 con 0x55f573d58820 2024-04-26T17:57:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc735d9b640 1 --2- 172.21.15.31:0/3651035917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d58820 0x55f573d60430 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0038a0 tx=0x7fc72c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e52a146fc6a2b3ee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc71dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:57:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc71dffb640 1 -- 172.21.15.31:0/3651035917 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c024070 con 0x55f573d58820 2024-04-26T17:57:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc71dffb640 1 -- 172.21.15.31:0/3651035917 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fc72c00fac0 con 0x55f573d58820 2024-04-26T17:57:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc73f9ff740 1 -- 172.21.15.31:0/3651035917 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7280047a0 con 0x55f573d58820 2024-04-26T17:57:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.035+0000 7fc71dffb640 1 -- 172.21.15.31:0/3651035917 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc72c030050 con 0x55f573d58820 2024-04-26T17:57:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.036+0000 7fc73f9ff740 10 client.15438 fetch_fsmap learned FSMap version 29 2024-04-26T17:57:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.036+0000 7fc73f9ff740 1 -- 172.21.15.31:0/3651035917 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f573a9c620 con 0x55f573d58820 2024-04-26T17:57:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.036+0000 7fc71dffb640 1 -- 172.21.15.31:0/3651035917 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7280047a0 con 0x55f573d58820 2024-04-26T17:57:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.036+0000 7fc71dffb640 1 -- 172.21.15.31:0/3651035917 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fc72c01f070 con 0x55f573d58820 2024-04-26T17:57:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.036+0000 7fc73f9ff740 10 client.15438 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:57:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.036+0000 7fc73f9ff740 10 client.15438 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:57:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.036+0000 7fc73f9ff740 20 client.15438 populate_metadata read hostname 'smithi031' 2024-04-26T17:57:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.036+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/3651035917 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55f573d731c0 0x55f573d75600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.036+0000 7fc73f9ff740 4 client.15438 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:57:38.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.036+0000 7fc73f9ff740 1 -- 172.21.15.31:0/3651035917 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f573ccc070 con 0x55f573d731c0 2024-04-26T17:57:38.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.036+0000 7fc73659c640 1 --2- 172.21.15.31:0/3651035917 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55f573d731c0 0x55f573d75600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.037+0000 7fc73659c640 1 --2- 172.21.15.31:0/3651035917 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55f573d731c0 0x55f573d75600 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3e823424b27fe74 server_cookie=95cf8a1d01ad2898 in_seq=0 out_seq=0 2024-04-26T17:57:38.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.037+0000 7fc71dffb640 10 client.15438 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:38.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.037+0000 7fc71dffb640 1 -- 172.21.15.31:0/3651035917 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f573ccc070 con 0x55f573d731c0 2024-04-26T17:57:38.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.037+0000 7fc71dffb640 10 client.15438 handle_command_reply: tid=0 2024-04-26T17:57:38.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.038+0000 7fc73f9ff740 1 client.15438 shutdown 2024-04-26T17:57:38.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.038+0000 7fc73f9ff740 1 -- 172.21.15.31:0/3651035917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d58820 msgr2=0x55f573d60430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.038+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/3651035917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d58820 0x55f573d60430 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0038a0 tx=0x7fc72c004730 comp rx=0 tx=0).stop 2024-04-26T17:57:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.038+0000 7fc73f9ff740 1 -- 172.21.15.31:0/3651035917 shutdown_connections 2024-04-26T17:57:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.038+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/3651035917 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55f573d731c0 0x55f573d75600 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.038+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/3651035917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 0x55f573d60970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.038+0000 7fc71dffb640 0 client.15438 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.038+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/3651035917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d58820 0x55f573d60430 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.038+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/3651035917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d57f00 0x55f573d5fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.038+0000 7fc73f9ff740 1 -- 172.21.15.31:0/3651035917 >> 172.21.15.31:0/3651035917 conn(0x55f573d54410 msgr2=0x55f573d5a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.038+0000 7fc73f9ff740 1 -- 172.21.15.31:0/3651035917 shutdown_connections 2024-04-26T17:57:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.038+0000 7fc73f9ff740 1 -- 172.21.15.31:0/3651035917 wait complete. 2024-04-26T17:57:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.038+0000 7fc73f9ff740 20 client.15438 trim_cache size 0 max 16384 2024-04-26T17:57:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.053+0000 7fc73f9ff740 1 Processor -- start 2024-04-26T17:57:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.053+0000 7fc73f9ff740 1 -- start start 2024-04-26T17:57:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.053+0000 7fc73f9ff740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d57f00 0x55f573d5f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.053+0000 7fc73f9ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d58820 0x55f573d7af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.053+0000 7fc73f9ff740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 0x55f573d7b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.053+0000 7fc73f9ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381b2e20 con 0x55f573d58820 2024-04-26T17:57:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.053+0000 7fc73f9ff740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381b0690 con 0x55f573d630e0 2024-04-26T17:57:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.053+0000 7fc73f9ff740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381ae500 con 0x55f573d57f00 2024-04-26T17:57:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.053+0000 7fc735d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d58820 0x55f573d7af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.053+0000 7fc735d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d58820 0x55f573d7af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50658/0 (socket says 172.21.15.31:50658) 2024-04-26T17:57:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.053+0000 7fc735d9b640 1 -- 172.21.15.31:0/2311430627 learned_addr learned my addr 172.21.15.31:0/2311430627 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.054+0000 7fc73659c640 1 --2- 172.21.15.31:0/2311430627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d57f00 0x55f573d5f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.054+0000 7fc736d9d640 1 --2- 172.21.15.31:0/2311430627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 0x55f573d7b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.054+0000 7fc735d9b640 1 -- 172.21.15.31:0/2311430627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d57f00 msgr2=0x55f573d5f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.054+0000 7fc735d9b640 1 --2- 172.21.15.31:0/2311430627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d57f00 0x55f573d5f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.054+0000 7fc735d9b640 1 -- 172.21.15.31:0/2311430627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 msgr2=0x55f573d7b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.054+0000 7fc735d9b640 1 --2- 172.21.15.31:0/2311430627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 0x55f573d7b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.054+0000 7fc735d9b640 1 -- 172.21.15.31:0/2311430627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f573d58610 con 0x55f573d58820 2024-04-26T17:57:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.054+0000 7fc73659c640 1 --2- 172.21.15.31:0/2311430627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d57f00 0x55f573d5f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:57:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.054+0000 7fc736d9d640 1 --2- 172.21.15.31:0/2311430627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 0x55f573d7b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:57:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.054+0000 7fc735d9b640 1 --2- 172.21.15.31:0/2311430627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d58820 0x55f573d7af10 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc72c013e20 tx=0x7fc72c020490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7371785e6f9cbfb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.054+0000 7fc71ffff640 1 -- 172.21.15.31:0/2311430627 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c02d070 con 0x55f573d58820 2024-04-26T17:57:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.054+0000 7fc71ffff640 1 -- 172.21.15.31:0/2311430627 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc72c00fdc0 con 0x55f573d58820 2024-04-26T17:57:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.054+0000 7fc71ffff640 1 -- 172.21.15.31:0/2311430627 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c024040 con 0x55f573d58820 2024-04-26T17:57:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 -- 172.21.15.31:0/2311430627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d58820 msgr2=0x55f573d7af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/2311430627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d58820 0x55f573d7af10 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc72c013e20 tx=0x7fc72c020490 comp rx=0 tx=0).stop 2024-04-26T17:57:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 -- 172.21.15.31:0/2311430627 shutdown_connections 2024-04-26T17:57:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/2311430627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 0x55f573d7b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/2311430627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d58820 0x55f573d7af10 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/2311430627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d57f00 0x55f573d5f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 -- 172.21.15.31:0/2311430627 >> 172.21.15.31:0/2311430627 conn(0x55f573d54410 msgr2=0x55f573d5feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 -- 172.21.15.31:0/2311430627 shutdown_connections 2024-04-26T17:57:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 -- 172.21.15.31:0/2311430627 wait complete. 2024-04-26T17:57:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 -- start start 2024-04-26T17:57:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 Processor -- start 2024-04-26T17:57:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d57f00 0x55f573d5ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d58820 0x55f573d5cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 0x55f573d5d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381aeb50 con 0x55f573d57f00 2024-04-26T17:57:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381affc0 con 0x55f573d630e0 2024-04-26T17:57:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381af220 con 0x55f573d58820 2024-04-26T17:57:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73f9ff740 10 client.0 _ll_register_callbacks cb 0x7fc72c004410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:57:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d57f00 0x55f573d5ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d57f00 0x55f573d5ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50670/0 (socket says 172.21.15.31:50670) 2024-04-26T17:57:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc735d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d58820 0x55f573d5cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.055+0000 7fc73659c640 1 -- 172.21.15.31:0/2913202051 learned_addr learned my addr 172.21.15.31:0/2913202051 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.056+0000 7fc736d9d640 1 --2- 172.21.15.31:0/2913202051 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 0x55f573d5d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.056+0000 7fc73659c640 1 -- 172.21.15.31:0/2913202051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d58820 msgr2=0x55f573d5cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.056+0000 7fc73659c640 1 --2- 172.21.15.31:0/2913202051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d58820 0x55f573d5cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.056+0000 7fc73659c640 1 -- 172.21.15.31:0/2913202051 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 msgr2=0x55f573d5d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.056+0000 7fc73659c640 1 --2- 172.21.15.31:0/2913202051 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 0x55f573d5d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.056+0000 7fc73659c640 1 -- 172.21.15.31:0/2913202051 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f573d691b0 con 0x55f573d57f00 2024-04-26T17:57:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.056+0000 7fc73659c640 1 --2- 172.21.15.31:0/2913202051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d57f00 0x55f573d5ca70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc72800d7c0 tx=0x7fc72800d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec710226acc8174b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.056+0000 7fc71dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:57:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.056+0000 7fc71dffb640 1 -- 172.21.15.31:0/2913202051 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72802e020 con 0x55f573d57f00 2024-04-26T17:57:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.056+0000 7fc73f9ff740 1 -- 172.21.15.31:0/2913202051 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc72c00fdc0 con 0x55f573d57f00 2024-04-26T17:57:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.056+0000 7fc71dffb640 1 -- 172.21.15.31:0/2913202051 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fc728004450 con 0x55f573d57f00 2024-04-26T17:57:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.056+0000 7fc71dffb640 1 -- 172.21.15.31:0/2913202051 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc728039050 con 0x55f573d57f00 2024-04-26T17:57:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.056+0000 7fc73f9ff740 10 client.15444 fetch_fsmap learned FSMap version 29 2024-04-26T17:57:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.056+0000 7fc73f9ff740 1 -- 172.21.15.31:0/2913202051 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f573dd9a50 con 0x55f573d57f00 2024-04-26T17:57:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.057+0000 7fc71dffb640 1 -- 172.21.15.31:0/2913202051 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc72c00fdc0 con 0x55f573d57f00 2024-04-26T17:57:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.057+0000 7fc71dffb640 1 -- 172.21.15.31:0/2913202051 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fc7280048c0 con 0x55f573d57f00 2024-04-26T17:57:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.057+0000 7fc73f9ff740 10 client.15444 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:57:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.057+0000 7fc73f9ff740 10 client.15444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:57:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.057+0000 7fc73f9ff740 20 client.15444 populate_metadata read hostname 'smithi031' 2024-04-26T17:57:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.057+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/2913202051 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55f573d731c0 0x55f573ddbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.057+0000 7fc73f9ff740 4 client.15444 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:57:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.057+0000 7fc73f9ff740 1 -- 172.21.15.31:0/2913202051 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f573cc5ee0 con 0x55f573d731c0 2024-04-26T17:57:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.057+0000 7fc735d9b640 1 --2- 172.21.15.31:0/2913202051 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55f573d731c0 0x55f573ddbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.057+0000 7fc71dffb640 10 client.15444 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.057+0000 7fc735d9b640 1 --2- 172.21.15.31:0/2913202051 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55f573d731c0 0x55f573ddbfe0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d78562c58e270b7 server_cookie=8c280c2581cd0ea6 in_seq=0 out_seq=0 2024-04-26T17:57:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.081+0000 7fc71dffb640 1 -- 172.21.15.31:0/2913202051 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f573cc5ee0 con 0x55f573d731c0 2024-04-26T17:57:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.081+0000 7fc71dffb640 10 client.15444 handle_command_reply: tid=0 2024-04-26T17:57:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.081+0000 7fc73f9ff740 1 client.15444 shutdown 2024-04-26T17:57:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.081+0000 7fc73f9ff740 1 -- 172.21.15.31:0/2913202051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d57f00 msgr2=0x55f573d5ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.081+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/2913202051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d57f00 0x55f573d5ca70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc72800d7c0 tx=0x7fc72800d9b0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73f9ff740 1 -- 172.21.15.31:0/2913202051 shutdown_connections 2024-04-26T17:57:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/2913202051 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55f573d731c0 0x55f573ddbfe0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/2913202051 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55f573d630e0 0x55f573d5d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/2913202051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f573d58820 0x55f573d5cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73f9ff740 1 --2- 172.21.15.31:0/2913202051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f573d57f00 0x55f573d5ca70 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc71dffb640 0 client.15444 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73f9ff740 1 -- 172.21.15.31:0/2913202051 >> 172.21.15.31:0/2913202051 conn(0x55f573d54410 msgr2=0x55f573d5feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73f9ff740 1 -- 172.21.15.31:0/2913202051 shutdown_connections 2024-04-26T17:57:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73f9ff740 1 -- 172.21.15.31:0/2913202051 wait complete. 2024-04-26T17:57:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73f9ff740 20 client.15444 trim_cache size 0 max 16384 2024-04-26T17:57:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:57:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:57:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:57:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:57:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73cc09640 1 -- 172.21.15.31:0/1206150350 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc7200a3f10 msgr2=0x7fc7200a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73cc09640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc7200a3f10 0x7fc7200a63d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc738076670 tx=0x7fc72c013880 comp rx=0 tx=0).stop 2024-04-26T17:57:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73cc09640 1 -- 172.21.15.31:0/1206150350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc738108940 msgr2=0x7fc7380756b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73cc09640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc738108940 0x7fc7380756b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc730005e30 tx=0x7fc730003ba0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73cc09640 1 -- 172.21.15.31:0/1206150350 shutdown_connections 2024-04-26T17:57:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73cc09640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc7200a3f10 0x7fc7200a63d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73cc09640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc738108940 0x7fc7380756b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73cc09640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc7381032f0 0x7fc738075170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73cc09640 1 --2- 172.21.15.31:0/1206150350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc738102930 0x7fc738074c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73cc09640 1 -- 172.21.15.31:0/1206150350 >> 172.21.15.31:0/1206150350 conn(0x7fc7380fe400 msgr2=0x7fc7380fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73cc09640 1 -- 172.21.15.31:0/1206150350 shutdown_connections 2024-04-26T17:57:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:38.082+0000 7fc73cc09640 1 -- 172.21.15.31:0/1206150350 wait complete. 2024-04-26T17:57:38.098 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 15.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T17:57:39.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:38 smithi186 ceph-mon[32797]: pgmap v37: 97 pgs: 97 active+clean; 955 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 42 op/s 2024-04-26T17:57:39.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:38 smithi031 ceph-mon[29164]: pgmap v37: 97 pgs: 97 active+clean; 955 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 42 op/s 2024-04-26T17:57:39.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:38 smithi097 ceph-mon[32419]: pgmap v37: 97 pgs: 97 active+clean; 955 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 42 op/s 2024-04-26T17:57:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:40 smithi186 ceph-mon[32797]: pgmap v38: 97 pgs: 97 active+clean; 955 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 89 KiB/s wr, 39 op/s 2024-04-26T17:57:41.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:40 smithi031 ceph-mon[29164]: pgmap v38: 97 pgs: 97 active+clean; 955 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 89 KiB/s wr, 39 op/s 2024-04-26T17:57:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:40 smithi097 ceph-mon[32419]: pgmap v38: 97 pgs: 97 active+clean; 955 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 89 KiB/s wr, 39 op/s 2024-04-26T17:57:43.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 20.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T17:57:43.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:42 smithi186 ceph-mon[32797]: pgmap v39: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 109 KiB/s wr, 49 op/s 2024-04-26T17:57:43.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:42 smithi031 ceph-mon[29164]: pgmap v39: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 109 KiB/s wr, 49 op/s 2024-04-26T17:57:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:42 smithi097 ceph-mon[32419]: pgmap v39: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 109 KiB/s wr, 49 op/s 2024-04-26T17:57:45.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:44 smithi186 ceph-mon[32797]: pgmap v40: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-04-26T17:57:45.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:44 smithi031 ceph-mon[29164]: pgmap v40: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-04-26T17:57:45.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:44 smithi097 ceph-mon[32419]: pgmap v40: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 14 op/s 2024-04-26T17:57:47.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:46 smithi186 ceph-mon[32797]: pgmap v41: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 29 KiB/s wr, 17 op/s 2024-04-26T17:57:47.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:46 smithi031 ceph-mon[29164]: pgmap v41: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 29 KiB/s wr, 17 op/s 2024-04-26T17:57:47.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:46 smithi097 ceph-mon[32419]: pgmap v41: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 29 KiB/s wr, 17 op/s 2024-04-26T17:57:48.099 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 25.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T17:57:49.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:48 smithi186 ceph-mon[32797]: pgmap v42: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-04-26T17:57:49.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:48 smithi031 ceph-mon[29164]: pgmap v42: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-04-26T17:57:49.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:48 smithi097 ceph-mon[32419]: pgmap v42: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-04-26T17:57:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.184+0000 7f75e1dac640 1 -- 172.21.15.31:0/3127210487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc1044e0 msgr2=0x7f75dc10c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.184+0000 7f75e1dac640 1 --2- 172.21.15.31:0/3127210487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc1044e0 0x7f75dc10c9c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009900 tx=0x7f75d002a510 comp rx=0 tx=0).stop 2024-04-26T17:57:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.184+0000 7f75e1dac640 1 -- 172.21.15.31:0/3127210487 shutdown_connections 2024-04-26T17:57:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.184+0000 7f75e1dac640 1 --2- 172.21.15.31:0/3127210487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75dc10cf00 0x7f75dc10f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.184+0000 7f75e1dac640 1 --2- 172.21.15.31:0/3127210487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc1044e0 0x7f75dc10c9c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.184+0000 7f75e1dac640 1 --2- 172.21.15.31:0/3127210487 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75dc103bc0 0x7f75dc103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.184+0000 7f75e1dac640 1 -- 172.21.15.31:0/3127210487 >> 172.21.15.31:0/3127210487 conn(0x7f75dc0789b0 msgr2=0x7f75dc078dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.184+0000 7f75e1dac640 1 -- 172.21.15.31:0/3127210487 shutdown_connections 2024-04-26T17:57:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.184+0000 7f75e1dac640 1 -- 172.21.15.31:0/3127210487 wait complete. 2024-04-26T17:57:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.184+0000 7f75e1dac640 1 Processor -- start 2024-04-26T17:57:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75e1dac640 1 -- start start 2024-04-26T17:57:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75e1dac640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc103bc0 0x7f75dc1990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75e1dac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75dc1044e0 0x7f75dc199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75e1dac640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75dc10cf00 0x7f75dc19d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75e1dac640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc112c00 con 0x7f75dc103bc0 2024-04-26T17:57:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75e1dac640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc112a80 con 0x7f75dc10cf00 2024-04-26T17:57:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75e1dac640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc112d80 con 0x7f75dc1044e0 2024-04-26T17:57:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75db7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc103bc0 0x7f75dc1990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75db7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc103bc0 0x7f75dc1990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60092/0 (socket says 172.21.15.31:60092) 2024-04-26T17:57:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75db7fe640 1 -- 172.21.15.31:0/2721381342 learned_addr learned my addr 172.21.15.31:0/2721381342 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75daffd640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75dc1044e0 0x7f75dc199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75dbfff640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75dc10cf00 0x7f75dc19d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75db7fe640 1 -- 172.21.15.31:0/2721381342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75dc1044e0 msgr2=0x7f75dc199600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75db7fe640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75dc1044e0 0x7f75dc199600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75db7fe640 1 -- 172.21.15.31:0/2721381342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75dc10cf00 msgr2=0x7f75dc19d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.185+0000 7f75db7fe640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75dc10cf00 0x7f75dc19d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.186+0000 7f75db7fe640 1 -- 172.21.15.31:0/2721381342 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75dc19e0b0 con 0x7f75dc103bc0 2024-04-26T17:57:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.186+0000 7f75daffd640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75dc1044e0 0x7f75dc199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:57:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.186+0000 7f75db7fe640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc103bc0 0x7f75dc1990c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f75cc00c920 tx=0x7f75cc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.186+0000 7f75d8ff9640 1 -- 172.21.15.31:0/2721381342 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc007c70 con 0x7f75dc103bc0 2024-04-26T17:57:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.186+0000 7f75d8ff9640 1 -- 172.21.15.31:0/2721381342 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75cc007e10 con 0x7f75dc103bc0 2024-04-26T17:57:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.186+0000 7f75e1dac640 1 -- 172.21.15.31:0/2721381342 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75dc19e3a0 con 0x7f75dc103bc0 2024-04-26T17:57:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.186+0000 7f75d8ff9640 1 -- 172.21.15.31:0/2721381342 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc018820 con 0x7f75dc103bc0 2024-04-26T17:57:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.186+0000 7f75e1dac640 1 -- 172.21.15.31:0/2721381342 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75dc1a5c30 con 0x7f75dc103bc0 2024-04-26T17:57:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.188+0000 7f75d8ff9640 1 -- 172.21.15.31:0/2721381342 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f75cc00f450 con 0x7f75dc103bc0 2024-04-26T17:57:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75e4b9b740 1 Processor -- start 2024-04-26T17:57:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75e4b9b740 1 -- start start 2024-04-26T17:57:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75e4b9b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7728f00 0x5641f77292e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75e4b9b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7729820 0x5641f7733ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75e4b9b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f77340e0 0x5641f77364d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75e4b9b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5641f768f460 con 0x5641f7728f00 2024-04-26T17:57:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75e4b9b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5641f7649590 con 0x5641f77340e0 2024-04-26T17:57:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75e4b9b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5641f768b420 con 0x5641f7729820 2024-04-26T17:57:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75dbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7728f00 0x5641f77292e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75d8ff9640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75c40a3ee0 0x7f75c40a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75dbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7728f00 0x5641f77292e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60096/0 (socket says 172.21.15.31:60096) 2024-04-26T17:57:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75dbfff640 1 -- 172.21.15.31:0/1310176505 learned_addr learned my addr 172.21.15.31:0/1310176505 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75daffd640 1 --2- 172.21.15.31:0/1310176505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f77340e0 0x5641f77364d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75db7fe640 1 --2- 172.21.15.31:0/1310176505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7729820 0x5641f7733ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75dbfff640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75c40a3ee0 0x7f75c40a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.189+0000 7f75d8ff9640 1 -- 172.21.15.31:0/2721381342 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f75cc0d0c90 con 0x7f75dc103bc0 2024-04-26T17:57:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.190+0000 7f75dbfff640 1 -- 172.21.15.31:0/1310176505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7729820 msgr2=0x5641f7733ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.190+0000 7f75dbfff640 1 --2- 172.21.15.31:0/1310176505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7729820 0x5641f7733ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.190+0000 7f75dbfff640 1 -- 172.21.15.31:0/1310176505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f77340e0 msgr2=0x5641f77364d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.190+0000 7f75dbfff640 1 --2- 172.21.15.31:0/1310176505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f77340e0 0x5641f77364d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.190+0000 7f75dbfff640 1 -- 172.21.15.31:0/1310176505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641f76560d0 con 0x5641f7728f00 2024-04-26T17:57:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.190+0000 7f75daffd640 1 --2- 172.21.15.31:0/1310176505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f77340e0 0x5641f77364d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.190+0000 7f75dbfff640 1 --2- 172.21.15.31:0/1310176505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7728f00 0x5641f77292e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f75d4014940 tx=0x7f75d4014e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86e980b47c84e23 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.190+0000 7f75c1ffb640 1 -- 172.21.15.31:0/1310176505 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d4014050 con 0x5641f7728f00 2024-04-26T17:57:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.190+0000 7f75c1ffb640 1 -- 172.21.15.31:0/1310176505 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75d40141f0 con 0x5641f7728f00 2024-04-26T17:57:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.190+0000 7f75c1ffb640 1 -- 172.21.15.31:0/1310176505 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d401e650 con 0x5641f7728f00 2024-04-26T17:57:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.190+0000 7f75dbfff640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75c40a3ee0 0x7f75c40a63a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f75d4021000 tx=0x7f75d4020000 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.191+0000 7f75e4b9b740 1 -- 172.21.15.31:0/1310176505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7728f00 msgr2=0x5641f77292e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.191+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/1310176505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7728f00 0x5641f77292e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f75d4014940 tx=0x7f75d4014e00 comp rx=0 tx=0).stop 2024-04-26T17:57:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.191+0000 7f75e4b9b740 1 -- 172.21.15.31:0/1310176505 shutdown_connections 2024-04-26T17:57:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.191+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/1310176505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f77340e0 0x5641f77364d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.191+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/1310176505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7729820 0x5641f7733ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.191+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/1310176505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7728f00 0x5641f77292e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.191+0000 7f75e4b9b740 1 -- 172.21.15.31:0/1310176505 >> 172.21.15.31:0/1310176505 conn(0x5641f7725410 msgr2=0x5641f77257f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.191+0000 7f75e4b9b740 1 -- 172.21.15.31:0/1310176505 shutdown_connections 2024-04-26T17:57:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.191+0000 7f75e4b9b740 1 -- 172.21.15.31:0/1310176505 wait complete. 2024-04-26T17:57:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.191+0000 7f75e4b9b740 1 -- start start 2024-04-26T17:57:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.191+0000 7f75e4b9b740 1 Processor -- start 2024-04-26T17:57:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75e4b9b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f7728f00 0x5641f7731020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75e4b9b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7729820 0x5641f7731560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75e4b9b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f77340e0 0x5641f7731aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75e4b9b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5641f7736d10 con 0x5641f77340e0 2024-04-26T17:57:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75e4b9b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5641f7616b60 con 0x5641f7728f00 2024-04-26T17:57:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75e4b9b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5641f7736e90 con 0x5641f7729820 2024-04-26T17:57:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75e4b9b740 10 client.0 _ll_register_callbacks cb 0x5641f763b2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:57:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75dbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f77340e0 0x5641f7731aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75db7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f7728f00 0x5641f7731020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75dbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f77340e0 0x5641f7731aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60100/0 (socket says 172.21.15.31:60100) 2024-04-26T17:57:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75dbfff640 1 -- 172.21.15.31:0/1058450458 learned_addr learned my addr 172.21.15.31:0/1058450458 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75daffd640 1 --2- 172.21.15.31:0/1058450458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7729820 0x5641f7731560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75dbfff640 1 -- 172.21.15.31:0/1058450458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7729820 msgr2=0x5641f7731560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75dbfff640 1 --2- 172.21.15.31:0/1058450458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7729820 0x5641f7731560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75dbfff640 1 -- 172.21.15.31:0/1058450458 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f7728f00 msgr2=0x5641f7731020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75dbfff640 1 --2- 172.21.15.31:0/1058450458 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f7728f00 0x5641f7731020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75dbfff640 1 -- 172.21.15.31:0/1058450458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641f73a1210 con 0x5641f77340e0 2024-04-26T17:57:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75db7fe640 1 --2- 172.21.15.31:0/1058450458 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f7728f00 0x5641f7731020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:57:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75dbfff640 1 --2- 172.21.15.31:0/1058450458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f77340e0 0x5641f7731aa0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f75d40095c0 tx=0x7f75d4005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1ec80d2b451bed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.192+0000 7f75baffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:57:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.193+0000 7f75baffd640 1 -- 172.21.15.31:0/1058450458 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d40143a0 con 0x5641f77340e0 2024-04-26T17:57:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.193+0000 7f75baffd640 1 -- 172.21.15.31:0/1058450458 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f75d401e3f0 con 0x5641f77340e0 2024-04-26T17:57:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.193+0000 7f75e4b9b740 1 -- 172.21.15.31:0/1058450458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75d40141f0 con 0x5641f77340e0 2024-04-26T17:57:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.193+0000 7f75baffd640 1 -- 172.21.15.31:0/1058450458 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f75d4036020 con 0x5641f77340e0 2024-04-26T17:57:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.193+0000 7f75e4b9b740 10 client.15456 fetch_fsmap learned FSMap version 29 2024-04-26T17:57:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.193+0000 7f75e4b9b740 1 -- 172.21.15.31:0/1058450458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5641f746d620 con 0x5641f77340e0 2024-04-26T17:57:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.193+0000 7f75baffd640 1 -- 172.21.15.31:0/1058450458 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75d40141f0 con 0x5641f77340e0 2024-04-26T17:57:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.193+0000 7f75baffd640 1 -- 172.21.15.31:0/1058450458 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f75d400f040 con 0x5641f77340e0 2024-04-26T17:57:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.193+0000 7f75e4b9b740 10 client.15456 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:57:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.193+0000 7f75e4b9b740 10 client.15456 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:57:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.193+0000 7f75e4b9b740 20 client.15456 populate_metadata read hostname 'smithi031' 2024-04-26T17:57:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.193+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/1058450458 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5641f7744180 0x5641f7746570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.193+0000 7f75e4b9b740 4 client.15456 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:57:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.193+0000 7f75e4b9b740 1 -- 172.21.15.31:0/1058450458 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641f769d070 con 0x5641f7744180 2024-04-26T17:57:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.194+0000 7f75db7fe640 1 --2- 172.21.15.31:0/1058450458 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5641f7744180 0x5641f7746570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.194+0000 7f75baffd640 10 client.15456 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.194+0000 7f75db7fe640 1 --2- 172.21.15.31:0/1058450458 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5641f7744180 0x5641f7746570 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5be4f00db705f2e8 server_cookie=71540261bf8292d2 in_seq=0 out_seq=0 2024-04-26T17:57:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75baffd640 1 -- 172.21.15.31:0/1058450458 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5641f769d070 con 0x5641f7744180 2024-04-26T17:57:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75baffd640 10 client.15456 handle_command_reply: tid=0 2024-04-26T17:57:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75e4b9b740 1 client.15456 shutdown 2024-04-26T17:57:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75e4b9b740 1 -- 172.21.15.31:0/1058450458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f77340e0 msgr2=0x5641f7731aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/1058450458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f77340e0 0x5641f7731aa0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f75d40095c0 tx=0x7f75d4005f10 comp rx=0 tx=0).stop 2024-04-26T17:57:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75e4b9b740 1 -- 172.21.15.31:0/1058450458 shutdown_connections 2024-04-26T17:57:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/1058450458 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5641f7744180 0x5641f7746570 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/1058450458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f77340e0 0x5641f7731aa0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/1058450458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7729820 0x5641f7731560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75baffd640 0 client.15456 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/1058450458 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f7728f00 0x5641f7731020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75e4b9b740 1 -- 172.21.15.31:0/1058450458 >> 172.21.15.31:0/1058450458 conn(0x5641f7725410 msgr2=0x5641f772b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75e4b9b740 1 -- 172.21.15.31:0/1058450458 shutdown_connections 2024-04-26T17:57:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75e4b9b740 1 -- 172.21.15.31:0/1058450458 wait complete. 2024-04-26T17:57:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.195+0000 7f75e4b9b740 20 client.15456 trim_cache size 0 max 16384 2024-04-26T17:57:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75e4b9b740 1 Processor -- start 2024-04-26T17:57:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75e4b9b740 1 -- start start 2024-04-26T17:57:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75e4b9b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7728f00 0x5641f772b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75e4b9b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7729820 0x5641f7724430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75e4b9b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f77340e0 0x5641f774bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75e4b9b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc1b0d10 con 0x5641f7729820 2024-04-26T17:57:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75e4b9b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc1ae580 con 0x5641f77340e0 2024-04-26T17:57:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75e4b9b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc1ac3d0 con 0x5641f7728f00 2024-04-26T17:57:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75daffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7729820 0x5641f7724430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75daffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7729820 0x5641f7724430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60110/0 (socket says 172.21.15.31:60110) 2024-04-26T17:57:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75daffd640 1 -- 172.21.15.31:0/3968557376 learned_addr learned my addr 172.21.15.31:0/3968557376 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75dbfff640 1 --2- 172.21.15.31:0/3968557376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f77340e0 0x5641f774bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75db7fe640 1 --2- 172.21.15.31:0/3968557376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7728f00 0x5641f772b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75daffd640 1 -- 172.21.15.31:0/3968557376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7728f00 msgr2=0x5641f772b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75daffd640 1 --2- 172.21.15.31:0/3968557376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7728f00 0x5641f772b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75daffd640 1 -- 172.21.15.31:0/3968557376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f77340e0 msgr2=0x5641f774bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75daffd640 1 --2- 172.21.15.31:0/3968557376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f77340e0 0x5641f774bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.211+0000 7f75daffd640 1 -- 172.21.15.31:0/3968557376 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641f7729650 con 0x5641f7729820 2024-04-26T17:57:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.212+0000 7f75daffd640 1 --2- 172.21.15.31:0/3968557376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7729820 0x5641f7724430 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009230 tx=0x7f75d00043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726d13f364bf1c81 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.212+0000 7f75c17fa640 1 -- 172.21.15.31:0/3968557376 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0035040 con 0x5641f7729820 2024-04-26T17:57:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.212+0000 7f75c17fa640 1 -- 172.21.15.31:0/3968557376 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75d00046c0 con 0x5641f7729820 2024-04-26T17:57:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.212+0000 7f75c17fa640 1 -- 172.21.15.31:0/3968557376 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0003850 con 0x5641f7729820 2024-04-26T17:57:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.212+0000 7f75e4b9b740 1 -- 172.21.15.31:0/3968557376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7729820 msgr2=0x5641f7724430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.212+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/3968557376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7729820 0x5641f7724430 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009230 tx=0x7f75d00043c0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.212+0000 7f75e4b9b740 1 -- 172.21.15.31:0/3968557376 shutdown_connections 2024-04-26T17:57:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.212+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/3968557376 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f77340e0 0x5641f774bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.212+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/3968557376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7729820 0x5641f7724430 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.212+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/3968557376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f7728f00 0x5641f772b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.212+0000 7f75e4b9b740 1 -- 172.21.15.31:0/3968557376 >> 172.21.15.31:0/3968557376 conn(0x5641f7725410 msgr2=0x5641f7723d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.212+0000 7f75e4b9b740 1 -- 172.21.15.31:0/3968557376 shutdown_connections 2024-04-26T17:57:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.212+0000 7f75e4b9b740 1 -- 172.21.15.31:0/3968557376 wait complete. 2024-04-26T17:57:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.213+0000 7f75e4b9b740 1 -- start start 2024-04-26T17:57:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.213+0000 7f75e4b9b740 1 Processor -- start 2024-04-26T17:57:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.213+0000 7f75e4b9b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f7728f00 0x5641f773c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.213+0000 7f75e4b9b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7729820 0x5641f773c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.213+0000 7f75e4b9b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f77340e0 0x5641f77390d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.213+0000 7f75e4b9b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc1aca40 con 0x5641f7729820 2024-04-26T17:57:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.213+0000 7f75e4b9b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc1adeb0 con 0x5641f7728f00 2024-04-26T17:57:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.213+0000 7f75e4b9b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc1ad110 con 0x5641f77340e0 2024-04-26T17:57:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.213+0000 7f75e4b9b740 10 client.0 _ll_register_callbacks cb 0x7f75d0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:57:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.213+0000 7f75daffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7729820 0x5641f773c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.213+0000 7f75daffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7729820 0x5641f773c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60122/0 (socket says 172.21.15.31:60122) 2024-04-26T17:57:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.213+0000 7f75daffd640 1 -- 172.21.15.31:0/4281731609 learned_addr learned my addr 172.21.15.31:0/4281731609 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75db7fe640 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f7728f00 0x5641f773c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75daffd640 1 -- 172.21.15.31:0/4281731609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f77340e0 msgr2=0x5641f77390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75dbfff640 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f77340e0 0x5641f77390d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75daffd640 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f77340e0 0x5641f77390d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75daffd640 1 -- 172.21.15.31:0/4281731609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f7728f00 msgr2=0x5641f773c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:57:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75daffd640 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f7728f00 0x5641f773c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75daffd640 1 -- 172.21.15.31:0/4281731609 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641f7739900 con 0x5641f7729820 2024-04-26T17:57:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75db7fe640 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f7728f00 0x5641f773c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:57:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75dbfff640 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f77340e0 0x5641f77390d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:57:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75daffd640 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7729820 0x5641f773c620 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f75d0000c00 tx=0x7f75d0005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adaae122249c3928 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75baffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:57:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75baffd640 1 -- 172.21.15.31:0/4281731609 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0035070 con 0x5641f7729820 2024-04-26T17:57:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75baffd640 1 -- 172.21.15.31:0/4281731609 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f75d00037f0 con 0x5641f7729820 2024-04-26T17:57:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75e4b9b740 1 -- 172.21.15.31:0/4281731609 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75d00046c0 con 0x5641f7729820 2024-04-26T17:57:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75baffd640 1 -- 172.21.15.31:0/4281731609 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f75d0056050 con 0x5641f7729820 2024-04-26T17:57:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75e4b9b740 10 client.15462 fetch_fsmap learned FSMap version 29 2024-04-26T17:57:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75e4b9b740 1 -- 172.21.15.31:0/4281731609 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5641f77ab110 con 0x5641f7729820 2024-04-26T17:57:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.214+0000 7f75baffd640 1 -- 172.21.15.31:0/4281731609 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75d00046c0 con 0x5641f7729820 2024-04-26T17:57:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.215+0000 7f75baffd640 1 -- 172.21.15.31:0/4281731609 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f75d0043020 con 0x5641f7729820 2024-04-26T17:57:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.215+0000 7f75e4b9b740 10 client.15462 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:57:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.215+0000 7f75e4b9b740 10 client.15462 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:57:51.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.215+0000 7f75e4b9b740 20 client.15462 populate_metadata read hostname 'smithi031' 2024-04-26T17:57:51.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.215+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5641f77ab810 0x5641f77adc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:51.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.215+0000 7f75e4b9b740 4 client.15462 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:57:51.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.215+0000 7f75e4b9b740 1 -- 172.21.15.31:0/4281731609 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5641f7696ee0 con 0x5641f77ab810 2024-04-26T17:57:51.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.215+0000 7f75db7fe640 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5641f77ab810 0x5641f77adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:51.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.215+0000 7f75db7fe640 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5641f77ab810 0x5641f77adc30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9739cefe4ba477bb server_cookie=7d4aae1d6eb135ca in_seq=0 out_seq=0 2024-04-26T17:57:51.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.215+0000 7f75baffd640 10 client.15462 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:51.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75baffd640 1 -- 172.21.15.31:0/4281731609 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5641f7696ee0 con 0x5641f77ab810 2024-04-26T17:57:51.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75baffd640 10 client.15462 handle_command_reply: tid=0 2024-04-26T17:57:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75e4b9b740 1 client.15462 shutdown 2024-04-26T17:57:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75e4b9b740 1 -- 172.21.15.31:0/4281731609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7729820 msgr2=0x5641f773c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7729820 0x5641f773c620 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f75d0000c00 tx=0x7f75d0005230 comp rx=0 tx=0).stop 2024-04-26T17:57:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75e4b9b740 1 -- 172.21.15.31:0/4281731609 shutdown_connections 2024-04-26T17:57:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5641f77ab810 0x5641f77adc30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641f77340e0 0x5641f77390d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641f7729820 0x5641f773c620 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75baffd640 0 client.15462 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:57:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75e4b9b740 1 --2- 172.21.15.31:0/4281731609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5641f7728f00 0x5641f773c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75e4b9b740 1 -- 172.21.15.31:0/4281731609 >> 172.21.15.31:0/4281731609 conn(0x5641f7725410 msgr2=0x5641f77245b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75e4b9b740 1 -- 172.21.15.31:0/4281731609 shutdown_connections 2024-04-26T17:57:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75e4b9b740 1 -- 172.21.15.31:0/4281731609 wait complete. 2024-04-26T17:57:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.224+0000 7f75e4b9b740 20 client.15462 trim_cache size 0 max 16384 2024-04-26T17:57:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:57:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:57:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:57:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:57:51.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.225+0000 7f75e1dac640 1 -- 172.21.15.31:0/2721381342 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75c40a3ee0 msgr2=0x7f75c40a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:51.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.225+0000 7f75e1dac640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75c40a3ee0 0x7f75c40a63a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f75d4021000 tx=0x7f75d4020000 comp rx=0 tx=0).stop 2024-04-26T17:57:51.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.225+0000 7f75e1dac640 1 -- 172.21.15.31:0/2721381342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc103bc0 msgr2=0x7f75dc1990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:51.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.225+0000 7f75e1dac640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc103bc0 0x7f75dc1990c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f75cc00c920 tx=0x7f75cc00cde0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.225+0000 7f75e1dac640 1 -- 172.21.15.31:0/2721381342 shutdown_connections 2024-04-26T17:57:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.225+0000 7f75e1dac640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75dc10cf00 0x7f75dc19d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.225+0000 7f75e1dac640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75dc1044e0 0x7f75dc199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.225+0000 7f75e1dac640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75c40a3ee0 0x7f75c40a63a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.225+0000 7f75e1dac640 1 --2- 172.21.15.31:0/2721381342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc103bc0 0x7f75dc1990c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.225+0000 7f75e1dac640 1 -- 172.21.15.31:0/2721381342 >> 172.21.15.31:0/2721381342 conn(0x7f75dc0789b0 msgr2=0x7f75dc0ffad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.225+0000 7f75e1dac640 1 -- 172.21.15.31:0/2721381342 shutdown_connections 2024-04-26T17:57:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:57:51.225+0000 7f75e1dac640 1 -- 172.21.15.31:0/2721381342 wait complete. 2024-04-26T17:57:51.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:50 smithi186 ceph-mon[32797]: pgmap v43: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-04-26T17:57:51.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:50 smithi031 ceph-mon[29164]: pgmap v43: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-04-26T17:57:51.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:50 smithi097 ceph-mon[32419]: pgmap v43: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-04-26T17:57:53.099 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 30.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T17:57:53.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:52 smithi186 ceph-mon[32797]: pgmap v44: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 22 KiB/s wr, 15 op/s 2024-04-26T17:57:53.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:52 smithi031 ceph-mon[29164]: pgmap v44: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 22 KiB/s wr, 15 op/s 2024-04-26T17:57:53.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:52 smithi097 ceph-mon[32419]: pgmap v44: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 22 KiB/s wr, 15 op/s 2024-04-26T17:57:55.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:54 smithi186 ceph-mon[32797]: pgmap v45: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-26T17:57:55.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:54 smithi031 ceph-mon[29164]: pgmap v45: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-26T17:57:55.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:54 smithi097 ceph-mon[32419]: pgmap v45: 97 pgs: 97 active+clean; 955 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-26T17:57:56.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:57:57.062 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:57:57.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:56 smithi031 ceph-mon[29164]: pgmap v46: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 16 MiB/s wr, 16 op/s 2024-04-26T17:57:57.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:56 smithi186 ceph-mon[32797]: pgmap v46: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 16 MiB/s wr, 16 op/s 2024-04-26T17:57:57.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:56 smithi097 ceph-mon[32419]: pgmap v46: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 16 MiB/s wr, 16 op/s 2024-04-26T17:57:58.099 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 35.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T17:57:58.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.885+0000 7fabf200c700 1 -- 172.21.15.31:0/543233999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec1034c0 msgr2=0x7fabec103940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:58.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.885+0000 7fabf200c700 1 --2- 172.21.15.31:0/543233999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec1034c0 0x7fabec103940 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fabd40099c0 tx=0x7fabd4009cd0 comp rx=0 tx=0).stop 2024-04-26T17:57:58.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.885+0000 7fabf200c700 1 -- 172.21.15.31:0/543233999 shutdown_connections 2024-04-26T17:57:58.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.885+0000 7fabf200c700 1 --2- 172.21.15.31:0/543233999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabec103e80 0x7fabec110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:58.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.885+0000 7fabf200c700 1 --2- 172.21.15.31:0/543233999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabec109560 0x7fabec109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:58.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.885+0000 7fabf200c700 1 --2- 172.21.15.31:0/543233999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec1034c0 0x7fabec103940 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:58.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.885+0000 7fabf200c700 1 -- 172.21.15.31:0/543233999 >> 172.21.15.31:0/543233999 conn(0x7fabec07a5d0 msgr2=0x7fabec07a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:58.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.885+0000 7fabf200c700 1 -- 172.21.15.31:0/543233999 shutdown_connections 2024-04-26T17:57:58.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.885+0000 7fabf200c700 1 -- 172.21.15.31:0/543233999 wait complete. 2024-04-26T17:57:58.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.886+0000 7fabf200c700 1 Processor -- start 2024-04-26T17:57:58.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.886+0000 7fabf200c700 1 -- start start 2024-04-26T17:57:58.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabf200c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabec1034c0 0x7fabec19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:58.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabf200c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec103e80 0x7fabec19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:58.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabf200c700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabec109560 0x7fabec1998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:58.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabf200c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fabec113660 con 0x7fabec103e80 2024-04-26T17:57:58.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabf200c700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fabec1134e0 con 0x7fabec109560 2024-04-26T17:57:58.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabf200c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fabec1137e0 con 0x7fabec1034c0 2024-04-26T17:57:58.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabeaffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec103e80 0x7fabec19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:58.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabeaffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec103e80 0x7fabec19fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41266/0 (socket says 172.21.15.31:41266) 2024-04-26T17:57:58.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabeaffd700 1 -- 172.21.15.31:0/2236053305 learned_addr learned my addr 172.21.15.31:0/2236053305 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:57:58.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabeb7fe700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabec1034c0 0x7fabec19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:58.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabebfff700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabec109560 0x7fabec1998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:58.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabeaffd700 1 -- 172.21.15.31:0/2236053305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabec1034c0 msgr2=0x7fabec19f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:58.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabeaffd700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabec1034c0 0x7fabec19f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:58.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabeaffd700 1 -- 172.21.15.31:0/2236053305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabec109560 msgr2=0x7fabec1998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:58.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabeaffd700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabec109560 0x7fabec1998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:58.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabeaffd700 1 -- 172.21.15.31:0/2236053305 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec19a040 con 0x7fabec103e80 2024-04-26T17:57:58.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabebfff700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabec109560 0x7fabec1998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:57:58.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabeaffd700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec103e80 0x7fabec19fc60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fabd400b580 tx=0x7fabd40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:58.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabe8ff9700 1 -- 172.21.15.31:0/2236053305 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4027070 con 0x7fabec103e80 2024-04-26T17:57:58.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabf200c700 1 -- 172.21.15.31:0/2236053305 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabec19a2d0 con 0x7fabec103e80 2024-04-26T17:57:58.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabe8ff9700 1 -- 172.21.15.31:0/2236053305 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabd4004020 con 0x7fabec103e80 2024-04-26T17:57:58.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.887+0000 7fabe8ff9700 1 -- 172.21.15.31:0/2236053305 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4022e30 con 0x7fabec103e80 2024-04-26T17:57:58.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.888+0000 7fabf200c700 1 -- 172.21.15.31:0/2236053305 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabec1a65f0 con 0x7fabec103e80 2024-04-26T17:57:58.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.888+0000 7fabf200c700 1 -- 172.21.15.31:0/2236053305 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabec10b9f0 con 0x7fabec103e80 2024-04-26T17:57:58.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.890+0000 7fabe8ff9700 1 -- 172.21.15.31:0/2236053305 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fabd4022590 con 0x7fabec103e80 2024-04-26T17:57:58.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.890+0000 7fabe8ff9700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fabd80a3ed0 0x7fabd80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:57:58.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.890+0000 7fabe8ff9700 1 -- 172.21.15.31:0/2236053305 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fabd40d1710 con 0x7fabec103e80 2024-04-26T17:57:58.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.890+0000 7fabeb7fe700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fabd80a3ed0 0x7fabd80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:57:58.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.891+0000 7fabeb7fe700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fabd80a3ed0 0x7fabd80a6390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fabec19aa20 tx=0x7fabe4009040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:57:58.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:58.895+0000 7fabe8ff9700 1 -- 172.21.15.31:0/2236053305 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fabd405dff0 con 0x7fabec103e80 2024-04-26T17:57:59.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:58 smithi031 ceph-mon[29164]: pgmap v47: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 16 MiB/s wr, 13 op/s 2024-04-26T17:57:59.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:59.151+0000 7fabf200c700 1 -- 172.21.15.31:0/2236053305 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabec104bd0 con 0x7fabec103e80 2024-04-26T17:57:59.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:59.151+0000 7fabe8ff9700 1 -- 172.21.15.31:0/2236053305 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11875 (secure 0 0 0) 0x7fabd4013070 con 0x7fabec103e80 2024-04-26T17:57:59.152 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:57:59.152 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T17:57:06.843229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24446,24457,24473]},"id":1}]} 2024-04-26T17:57:59.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:59.153+0000 7fabf200c700 1 -- 172.21.15.31:0/2236053305 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fabd80a3ed0 msgr2=0x7fabd80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:59.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:59.153+0000 7fabf200c700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fabd80a3ed0 0x7fabd80a6390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fabec19aa20 tx=0x7fabe4009040 comp rx=0 tx=0).stop 2024-04-26T17:57:59.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:59.153+0000 7fabf200c700 1 -- 172.21.15.31:0/2236053305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec103e80 msgr2=0x7fabec19fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:57:59.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:59.153+0000 7fabf200c700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec103e80 0x7fabec19fc60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fabd400b580 tx=0x7fabd40047b0 comp rx=0 tx=0).stop 2024-04-26T17:57:59.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:59.154+0000 7fabf200c700 1 -- 172.21.15.31:0/2236053305 shutdown_connections 2024-04-26T17:57:59.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:59.154+0000 7fabf200c700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fabd80a3ed0 0x7fabd80a6390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:59.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:59.154+0000 7fabf200c700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fabec109560 0x7fabec1998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:59.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:59.154+0000 7fabf200c700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabec1034c0 0x7fabec19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:59.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:59.154+0000 7fabf200c700 1 --2- 172.21.15.31:0/2236053305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabec103e80 0x7fabec19fc60 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:57:59.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:59.154+0000 7fabf200c700 1 -- 172.21.15.31:0/2236053305 >> 172.21.15.31:0/2236053305 conn(0x7fabec07a5d0 msgr2=0x7fabec1020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:57:59.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:59.154+0000 7fabf200c700 1 -- 172.21.15.31:0/2236053305 shutdown_connections 2024-04-26T17:57:59.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:57:59.154+0000 7fabf200c700 1 -- 172.21.15.31:0/2236053305 wait complete. 2024-04-26T17:57:59.155 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-26T17:57:59.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:58 smithi186 ceph-mon[32797]: pgmap v47: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 16 MiB/s wr, 13 op/s 2024-04-26T17:57:59.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:58 smithi097 ceph-mon[32419]: pgmap v47: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 16 MiB/s wr, 13 op/s 2024-04-26T17:57:59.987 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24473 quiesce db --set-id 10d0f28a --release --await' 2024-04-26T17:57:59.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24473 quiesce db --set-id 10d0f28a --release --await 2024-04-26T17:58:00.248 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:58:00.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:57:59 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2236053305' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:58:00.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:57:59 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2236053305' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:58:00.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:57:59 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2236053305' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:58:01.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:00 smithi186 ceph-mon[32797]: pgmap v48: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 16 MiB/s wr, 13 op/s 2024-04-26T17:58:01.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:00 smithi031 ceph-mon[29164]: pgmap v48: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 16 MiB/s wr, 13 op/s 2024-04-26T17:58:01.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:00 smithi097 ceph-mon[32419]: pgmap v48: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 16 MiB/s wr, 13 op/s 2024-04-26T17:58:02.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.015+0000 7f7c5e58e700 1 -- 172.21.15.31:0/1688787344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c58102f50 msgr2=0x7f7c58106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:02.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.015+0000 7f7c5e58e700 1 --2- 172.21.15.31:0/1688787344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c58102f50 0x7f7c58106fa0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c0099c0 tx=0x7f7c4c009cd0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.015+0000 7f7c5e58e700 1 -- 172.21.15.31:0/1688787344 shutdown_connections 2024-04-26T17:58:02.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.015+0000 7f7c5e58e700 1 --2- 172.21.15.31:0/1688787344 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c581025a0 0x7f7c58102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.015+0000 7f7c5e58e700 1 --2- 172.21.15.31:0/1688787344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c58107680 0x7f7c5810f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.015+0000 7f7c5e58e700 1 --2- 172.21.15.31:0/1688787344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c58102f50 0x7f7c58106fa0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.015+0000 7f7c5e58e700 1 -- 172.21.15.31:0/1688787344 >> 172.21.15.31:0/1688787344 conn(0x7f7c580793c0 msgr2=0x7f7c580797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:02.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.016+0000 7f7c5e58e700 1 -- 172.21.15.31:0/1688787344 shutdown_connections 2024-04-26T17:58:02.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.016+0000 7f7c5e58e700 1 -- 172.21.15.31:0/1688787344 wait complete. 2024-04-26T17:58:02.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.016+0000 7f7c5e58e700 1 Processor -- start 2024-04-26T17:58:02.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.016+0000 7f7c5e58e700 1 -- start start 2024-04-26T17:58:02.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c5e58e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c581025a0 0x7f7c5819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c5e58e700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c58102f50 0x7f7c5819fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c5e58e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c58107680 0x7f7c58199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c5e58e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7c58112570 con 0x7f7c581025a0 2024-04-26T17:58:02.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c5e58e700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7c581123f0 con 0x7f7c58102f50 2024-04-26T17:58:02.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c5e58e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7c581126f0 con 0x7f7c58107680 2024-04-26T17:58:02.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c57fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c581025a0 0x7f7c5819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c57fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c581025a0 0x7f7c5819f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41278/0 (socket says 172.21.15.31:41278) 2024-04-26T17:58:02.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c57fff700 1 -- 172.21.15.31:0/2806218800 learned_addr learned my addr 172.21.15.31:0/2806218800 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:02.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c5cb2b700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c58107680 0x7f7c58199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c577fe700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c58102f50 0x7f7c5819fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c57fff700 1 -- 172.21.15.31:0/2806218800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c58107680 msgr2=0x7f7c58199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:02.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c57fff700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c58107680 0x7f7c58199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c57fff700 1 -- 172.21.15.31:0/2806218800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c58102f50 msgr2=0x7f7c5819fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:02.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c57fff700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c58102f50 0x7f7c5819fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c57fff700 1 -- 172.21.15.31:0/2806218800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c5819a080 con 0x7f7c581025a0 2024-04-26T17:58:02.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c577fe700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c58102f50 0x7f7c5819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:58:02.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.017+0000 7f7c57fff700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c581025a0 0x7f7c5819f760 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800b530 tx=0x7f7c4800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:02.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.018+0000 7f7c557fa700 1 -- 172.21.15.31:0/2806218800 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48015020 con 0x7f7c581025a0 2024-04-26T17:58:02.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.018+0000 7f7c557fa700 1 -- 172.21.15.31:0/2806218800 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c48011ce0 con 0x7f7c581025a0 2024-04-26T17:58:02.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.018+0000 7f7c5e58e700 1 -- 172.21.15.31:0/2806218800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c5819a370 con 0x7f7c581025a0 2024-04-26T17:58:02.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.018+0000 7f7c557fa700 1 -- 172.21.15.31:0/2806218800 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48010e40 con 0x7f7c581025a0 2024-04-26T17:58:02.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.018+0000 7f7c5e58e700 1 -- 172.21.15.31:0/2806218800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c581a6630 con 0x7f7c581025a0 2024-04-26T17:58:02.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.020+0000 7f7c557fa700 1 -- 172.21.15.31:0/2806218800 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7c48018020 con 0x7f7c581025a0 2024-04-26T17:58:02.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.020+0000 7f7c557fa700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7c400a3db0 0x7f7c400a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.021+0000 7f7c577fe700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7c400a3db0 0x7f7c400a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.021+0000 7f7c557fa700 1 -- 172.21.15.31:0/2806218800 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7c480cff70 con 0x7f7c581025a0 2024-04-26T17:58:02.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.021+0000 7f7c577fe700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7c400a3db0 0x7f7c400a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c000c00 tx=0x7f7c4c005fd0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:02.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.021+0000 7f7c68e20b80 1 Processor -- start 2024-04-26T17:58:02.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.021+0000 7f7c68e20b80 1 -- start start 2024-04-26T17:58:02.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c68e20b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce212571a0 0x55ce21257580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c68e20b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce21262430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c68e20b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21262970 0x55ce21264d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c68e20b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ce211aa840 con 0x55ce21257ac0 2024-04-26T17:58:02.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c68e20b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ce211496e0 con 0x55ce212571a0 2024-04-26T17:58:02.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c68e20b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ce211a6800 con 0x55ce21262970 2024-04-26T17:58:02.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c577fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce21262430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c577fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce21262430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41284/0 (socket says 172.21.15.31:41284) 2024-04-26T17:58:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c577fe700 1 -- 172.21.15.31:0/1412565009 learned_addr learned my addr 172.21.15.31:0/1412565009 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c5cb2b700 1 --2- 172.21.15.31:0/1412565009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21262970 0x55ce21264d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c57fff700 1 --2- 172.21.15.31:0/1412565009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce212571a0 0x55ce21257580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c577fe700 1 -- 172.21.15.31:0/1412565009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21262970 msgr2=0x55ce21264d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c577fe700 1 --2- 172.21.15.31:0/1412565009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21262970 0x55ce21264d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c577fe700 1 -- 172.21.15.31:0/1412565009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce212571a0 msgr2=0x55ce21257580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c577fe700 1 --2- 172.21.15.31:0/1412565009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce212571a0 0x55ce21257580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c577fe700 1 -- 172.21.15.31:0/1412565009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce210f79d0 con 0x55ce21257ac0 2024-04-26T17:58:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c577fe700 1 --2- 172.21.15.31:0/1412565009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce21262430 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009750 tx=0x7f7c4c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0e47150884647a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c3e7fc700 1 -- 172.21.15.31:0/1412565009 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c003860 con 0x55ce21257ac0 2024-04-26T17:58:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.022+0000 7f7c3e7fc700 1 -- 172.21.15.31:0/1412565009 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c4c005030 con 0x55ce21257ac0 2024-04-26T17:58:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.023+0000 7f7c3e7fc700 1 -- 172.21.15.31:0/1412565009 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c00f740 con 0x55ce21257ac0 2024-04-26T17:58:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.023+0000 7f7c68e20b80 1 -- 172.21.15.31:0/1412565009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 msgr2=0x55ce21262430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:02.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.023+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/1412565009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce21262430 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009750 tx=0x7f7c4c01f000 comp rx=0 tx=0).stop 2024-04-26T17:58:02.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.023+0000 7f7c68e20b80 1 -- 172.21.15.31:0/1412565009 shutdown_connections 2024-04-26T17:58:02.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.023+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/1412565009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce212571a0 0x55ce21257580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.023+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/1412565009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21262970 0x55ce21264d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.023+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/1412565009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce21262430 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.023+0000 7f7c68e20b80 1 -- 172.21.15.31:0/1412565009 >> 172.21.15.31:0/1412565009 conn(0x55ce21184e40 msgr2=0x55ce21253690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:02.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.023+0000 7f7c68e20b80 1 -- 172.21.15.31:0/1412565009 shutdown_connections 2024-04-26T17:58:02.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.023+0000 7f7c68e20b80 1 -- 172.21.15.31:0/1412565009 wait complete. 2024-04-26T17:58:02.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.024+0000 7f7c68e20b80 1 -- start start 2024-04-26T17:58:02.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.024+0000 7f7c68e20b80 1 Processor -- start 2024-04-26T17:58:02.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.024+0000 7f7c68e20b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce212571a0 0x55ce21262170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.024+0000 7f7c68e20b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21257ac0 0x55ce2125f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.024+0000 7f7c68e20b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce21262970 0x55ce2125f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.024+0000 7f7c68e20b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ce212654e0 con 0x55ce212571a0 2024-04-26T17:58:02.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.024+0000 7f7c68e20b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ce21110a50 con 0x55ce21262970 2024-04-26T17:58:02.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.024+0000 7f7c68e20b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ce21265660 con 0x55ce21257ac0 2024-04-26T17:58:02.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.024+0000 7f7c68e20b80 10 client.0 _ll_register_callbacks cb 0x55ce2113e680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:02.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.024+0000 7f7c57fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce212571a0 0x55ce21262170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.024+0000 7f7c57fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce212571a0 0x55ce21262170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41298/0 (socket says 172.21.15.31:41298) 2024-04-26T17:58:02.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.024+0000 7f7c57fff700 1 -- 172.21.15.31:0/2605419675 learned_addr learned my addr 172.21.15.31:0/2605419675 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:02.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c577fe700 1 --2- 172.21.15.31:0/2605419675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21257ac0 0x55ce2125f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c5cb2b700 1 --2- 172.21.15.31:0/2605419675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce21262970 0x55ce2125f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c57fff700 1 -- 172.21.15.31:0/2605419675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21257ac0 msgr2=0x55ce2125f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:02.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c57fff700 1 --2- 172.21.15.31:0/2605419675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21257ac0 0x55ce2125f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c57fff700 1 -- 172.21.15.31:0/2605419675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce21262970 msgr2=0x55ce2125f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c57fff700 1 --2- 172.21.15.31:0/2605419675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce21262970 0x55ce2125f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c57fff700 1 -- 172.21.15.31:0/2605419675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce212600c0 con 0x55ce212571a0 2024-04-26T17:58:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c57fff700 1 --2- 172.21.15.31:0/2605419675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce212571a0 0x55ce21262170 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7c4809bfa0 tx=0x7f7c4809c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e23c664a4dd12996 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c377fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:58:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c377fe700 1 -- 172.21.15.31:0/2605419675 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c480b4020 con 0x55ce212571a0 2024-04-26T17:58:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c377fe700 1 -- 172.21.15.31:0/2605419675 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7c480bc040 con 0x55ce212571a0 2024-04-26T17:58:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c68e20b80 1 -- 172.21.15.31:0/2605419675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c4c005030 con 0x55ce212571a0 2024-04-26T17:58:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c377fe700 1 -- 172.21.15.31:0/2605419675 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7c480a4c80 con 0x55ce212571a0 2024-04-26T17:58:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c68e20b80 10 client.15480 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.025+0000 7f7c68e20b80 1 -- 172.21.15.31:0/2605419675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ce21270040 con 0x55ce212571a0 2024-04-26T17:58:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.026+0000 7f7c377fe700 1 -- 172.21.15.31:0/2605419675 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c4c005030 con 0x55ce212571a0 2024-04-26T17:58:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.026+0000 7f7c377fe700 1 -- 172.21.15.31:0/2605419675 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f7c48096b80 con 0x55ce212571a0 2024-04-26T17:58:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.026+0000 7f7c68e20b80 10 client.15480 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.026+0000 7f7c68e20b80 10 client.15480 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T17:58:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.026+0000 7f7c68e20b80 20 client.15480 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.026+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/2605419675 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ce212725d0 0x55ce212749e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.026+0000 7f7c68e20b80 4 client.15480 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:58:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.026+0000 7f7c68e20b80 1 -- 172.21.15.31:0/2605419675 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce211b8220 con 0x55ce212725d0 2024-04-26T17:58:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.026+0000 7f7c577fe700 1 --2- 172.21.15.31:0/2605419675 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ce212725d0 0x55ce212749e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.026+0000 7f7c577fe700 1 --2- 172.21.15.31:0/2605419675 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ce212725d0 0x55ce212749e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22f5344594a218c server_cookie=9945a2f370861da6 in_seq=0 out_seq=0 2024-04-26T17:58:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.026+0000 7f7c377fe700 10 client.15480 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.027+0000 7f7c377fe700 1 -- 172.21.15.31:0/2605419675 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce211b8220 con 0x55ce212725d0 2024-04-26T17:58:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.027+0000 7f7c377fe700 10 client.15480 handle_command_reply: tid=0 2024-04-26T17:58:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.027+0000 7f7c68e20b80 1 client.15480 shutdown 2024-04-26T17:58:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.027+0000 7f7c68e20b80 1 -- 172.21.15.31:0/2605419675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce212571a0 msgr2=0x55ce21262170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.027+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/2605419675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce212571a0 0x55ce21262170 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7c4809bfa0 tx=0x7f7c4809c800 comp rx=0 tx=0).stop 2024-04-26T17:58:02.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.027+0000 7f7c68e20b80 1 -- 172.21.15.31:0/2605419675 shutdown_connections 2024-04-26T17:58:02.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.027+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/2605419675 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ce212725d0 0x55ce212749e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.027+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/2605419675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce21262970 0x55ce2125f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.027+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/2605419675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce212571a0 0x55ce21262170 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.027+0000 7f7c377fe700 0 client.15480 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:02.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.027+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/2605419675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21257ac0 0x55ce2125f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.027+0000 7f7c68e20b80 1 -- 172.21.15.31:0/2605419675 >> 172.21.15.31:0/2605419675 conn(0x55ce21184e40 msgr2=0x55ce212532c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:02.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.028+0000 7f7c68e20b80 1 -- 172.21.15.31:0/2605419675 shutdown_connections 2024-04-26T17:58:02.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.028+0000 7f7c68e20b80 1 -- 172.21.15.31:0/2605419675 wait complete. 2024-04-26T17:58:02.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.028+0000 7f7c68e20b80 20 client.15480 trim_cache size 0 max 16384 2024-04-26T17:58:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.047+0000 7f7c68e20b80 1 Processor -- start 2024-04-26T17:58:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.047+0000 7f7c68e20b80 1 -- start start 2024-04-26T17:58:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.047+0000 7f7c68e20b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce212571a0 0x55ce2125aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.047+0000 7f7c68e20b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce21259800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.047+0000 7f7c68e20b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21262970 0x55ce2126e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.047+0000 7f7c68e20b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7c581b0460 con 0x55ce21257ac0 2024-04-26T17:58:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.047+0000 7f7c68e20b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7c581adc90 con 0x55ce212571a0 2024-04-26T17:58:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.047+0000 7f7c68e20b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7c581abae0 con 0x55ce21262970 2024-04-26T17:58:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c577fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce21259800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c577fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce21259800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41300/0 (socket says 172.21.15.31:41300) 2024-04-26T17:58:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c577fe700 1 -- 172.21.15.31:0/3634436773 learned_addr learned my addr 172.21.15.31:0/3634436773 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c57fff700 1 --2- 172.21.15.31:0/3634436773 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce212571a0 0x55ce2125aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c5cb2b700 1 --2- 172.21.15.31:0/3634436773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21262970 0x55ce2126e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c577fe700 1 -- 172.21.15.31:0/3634436773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21262970 msgr2=0x55ce2126e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c577fe700 1 --2- 172.21.15.31:0/3634436773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21262970 0x55ce2126e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c577fe700 1 -- 172.21.15.31:0/3634436773 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce212571a0 msgr2=0x55ce2125aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c577fe700 1 --2- 172.21.15.31:0/3634436773 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce212571a0 0x55ce2125aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c577fe700 1 -- 172.21.15.31:0/3634436773 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce21275060 con 0x55ce21257ac0 2024-04-26T17:58:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c5cb2b700 1 --2- 172.21.15.31:0/3634436773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21262970 0x55ce2126e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:58:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c577fe700 1 --2- 172.21.15.31:0/3634436773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce21259800 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009350 tx=0x7f7c4c019040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f6daab5742b745 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c3dffb700 1 -- 172.21.15.31:0/3634436773 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c0050f0 con 0x55ce21257ac0 2024-04-26T17:58:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c3dffb700 1 -- 172.21.15.31:0/3634436773 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c4c03a920 con 0x55ce21257ac0 2024-04-26T17:58:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.048+0000 7f7c3dffb700 1 -- 172.21.15.31:0/3634436773 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c00f4e0 con 0x55ce21257ac0 2024-04-26T17:58:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.049+0000 7f7c68e20b80 1 -- 172.21.15.31:0/3634436773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 msgr2=0x55ce21259800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.049+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/3634436773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce21259800 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009350 tx=0x7f7c4c019040 comp rx=0 tx=0).stop 2024-04-26T17:58:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.049+0000 7f7c68e20b80 1 -- 172.21.15.31:0/3634436773 shutdown_connections 2024-04-26T17:58:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.049+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/3634436773 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce212571a0 0x55ce2125aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.049+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/3634436773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce21262970 0x55ce2126e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.049+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/3634436773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce21259800 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.049+0000 7f7c68e20b80 1 -- 172.21.15.31:0/3634436773 >> 172.21.15.31:0/3634436773 conn(0x55ce21184e40 msgr2=0x55ce212590f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.049+0000 7f7c68e20b80 1 -- 172.21.15.31:0/3634436773 shutdown_connections 2024-04-26T17:58:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.049+0000 7f7c68e20b80 1 -- 172.21.15.31:0/3634436773 wait complete. 2024-04-26T17:58:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.050+0000 7f7c68e20b80 1 -- start start 2024-04-26T17:58:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.050+0000 7f7c68e20b80 1 Processor -- start 2024-04-26T17:58:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.050+0000 7f7c68e20b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce212571a0 0x55ce2126a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.050+0000 7f7c68e20b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce2126ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.050+0000 7f7c68e20b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce21262970 0x55ce212678c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.050+0000 7f7c68e20b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7c581ac150 con 0x55ce21257ac0 2024-04-26T17:58:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.050+0000 7f7c68e20b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7c581ad5c0 con 0x55ce21262970 2024-04-26T17:58:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.050+0000 7f7c68e20b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7c581ac820 con 0x55ce212571a0 2024-04-26T17:58:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.050+0000 7f7c68e20b80 10 client.0 _ll_register_callbacks cb 0x55ce2113e680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c577fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce2126ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c57fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce212571a0 0x55ce2126a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c577fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce2126ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41316/0 (socket says 172.21.15.31:41316) 2024-04-26T17:58:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c577fe700 1 -- 172.21.15.31:0/1177858917 learned_addr learned my addr 172.21.15.31:0/1177858917 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c5cb2b700 1 --2- 172.21.15.31:0/1177858917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce21262970 0x55ce212678c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c577fe700 1 -- 172.21.15.31:0/1177858917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce212571a0 msgr2=0x55ce2126a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c577fe700 1 --2- 172.21.15.31:0/1177858917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce212571a0 0x55ce2126a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c577fe700 1 -- 172.21.15.31:0/1177858917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce21262970 msgr2=0x55ce212678c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c577fe700 1 --2- 172.21.15.31:0/1177858917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce21262970 0x55ce212678c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c577fe700 1 -- 172.21.15.31:0/1177858917 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce21267f50 con 0x55ce21257ac0 2024-04-26T17:58:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c57fff700 1 --2- 172.21.15.31:0/1177858917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce212571a0 0x55ce2126a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:58:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c5cb2b700 1 --2- 172.21.15.31:0/1177858917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce21262970 0x55ce212678c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:58:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c577fe700 1 --2- 172.21.15.31:0/1177858917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce2126ae30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00bbf0 tx=0x7f7c4c005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08b398786b25e12 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c377fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:58:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c377fe700 1 -- 172.21.15.31:0/1177858917 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c005240 con 0x55ce21257ac0 2024-04-26T17:58:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c68e20b80 1 -- 172.21.15.31:0/1177858917 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ce211b22c0 con 0x55ce21257ac0 2024-04-26T17:58:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.051+0000 7f7c377fe700 1 -- 172.21.15.31:0/1177858917 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7c4c03a470 con 0x55ce21257ac0 2024-04-26T17:58:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.052+0000 7f7c377fe700 1 -- 172.21.15.31:0/1177858917 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7c4c03a670 con 0x55ce21257ac0 2024-04-26T17:58:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.052+0000 7f7c68e20b80 10 client.15486 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.052+0000 7f7c68e20b80 1 -- 172.21.15.31:0/1177858917 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ce212d94f0 con 0x55ce21257ac0 2024-04-26T17:58:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.052+0000 7f7c377fe700 1 -- 172.21.15.31:0/1177858917 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ce211b22c0 con 0x55ce21257ac0 2024-04-26T17:58:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.052+0000 7f7c377fe700 1 -- 172.21.15.31:0/1177858917 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f7c4c02fa70 con 0x55ce21257ac0 2024-04-26T17:58:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.052+0000 7f7c68e20b80 10 client.15486 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.052+0000 7f7c68e20b80 10 client.15486 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T17:58:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.052+0000 7f7c68e20b80 20 client.15486 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.052+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/1177858917 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ce212d9900 0x55ce212dbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.052+0000 7f7c68e20b80 4 client.15486 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "10d0f28a", "release": true, "await": true}] 2024-04-26T17:58:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.052+0000 7f7c68e20b80 1 -- 172.21.15.31:0/1177858917 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "quiesce db", "set_id": "10d0f28a", "release": true, "await": true}) -- 0x55ce2102a350 con 0x55ce212d9900 2024-04-26T17:58:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.053+0000 7f7c57fff700 1 --2- 172.21.15.31:0/1177858917 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ce212d9900 0x55ce212dbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.053+0000 7f7c377fe700 10 client.15486 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.053+0000 7f7c57fff700 1 --2- 172.21.15.31:0/1177858917 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ce212d9900 0x55ce212dbcf0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49b402726b4a9ecc server_cookie=47e6421f1c664a4f in_seq=0 out_seq=0 2024-04-26T17:58:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.084+0000 7f7c377fe700 1 -- 172.21.15.31:0/1177858917 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55ce2102a350 con 0x55ce212d9900 2024-04-26T17:58:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.084+0000 7f7c377fe700 10 client.15486 handle_command_reply: tid=0 2024-04-26T17:58:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.084+0000 7f7c68e20b80 1 client.15486 shutdown 2024-04-26T17:58:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.084+0000 7f7c68e20b80 1 -- 172.21.15.31:0/1177858917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 msgr2=0x55ce2126ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.084+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/1177858917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce2126ae30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00bbf0 tx=0x7f7c4c005730 comp rx=0 tx=0).stop 2024-04-26T17:58:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.084+0000 7f7c68e20b80 1 -- 172.21.15.31:0/1177858917 shutdown_connections 2024-04-26T17:58:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.084+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/1177858917 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ce212d9900 0x55ce212dbcf0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.084+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/1177858917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ce21262970 0x55ce212678c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.084+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/1177858917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce212571a0 0x55ce2126a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.084+0000 7f7c68e20b80 1 --2- 172.21.15.31:0/1177858917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ce21257ac0 0x55ce2126ae30 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.084+0000 7f7c68e20b80 1 -- 172.21.15.31:0/1177858917 >> 172.21.15.31:0/1177858917 conn(0x55ce21184e40 msgr2=0x55ce21259980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.084+0000 7f7c377fe700 0 client.15486 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.084+0000 7f7c68e20b80 1 -- 172.21.15.31:0/1177858917 shutdown_connections 2024-04-26T17:58:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.084+0000 7f7c68e20b80 1 -- 172.21.15.31:0/1177858917 wait complete. 2024-04-26T17:58:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.085+0000 7f7c68e20b80 20 client.15486 trim_cache size 0 max 16384 2024-04-26T17:58:02.086 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T17:58:02.086 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 29, 2024-04-26T17:58:02.086 INFO:teuthology.orchestra.run.smithi031.stdout: "leader": 24473, 2024-04-26T17:58:02.086 INFO:teuthology.orchestra.run.smithi031.stdout: "set_version": 4, 2024-04-26T17:58:02.086 INFO:teuthology.orchestra.run.smithi031.stdout: "sets": { 2024-04-26T17:58:02.086 INFO:teuthology.orchestra.run.smithi031.stdout: "10d0f28a": { 2024-04-26T17:58:02.086 INFO:teuthology.orchestra.run.smithi031.stdout: "version": 4, 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: "age_ref": 0.0, 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RELEASED", 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: "timeout": 60.0, 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: "expiration": 90.0, 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: "members": { 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: "file:/": { 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: "excluded": false, 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RELEASED", 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:02.087 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:02.088 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:02.088 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:02.088 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:02.088 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T17:58:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.086+0000 7f7c5e58e700 1 -- 172.21.15.31:0/2806218800 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7c400a3db0 msgr2=0x7f7c400a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.086+0000 7f7c5e58e700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7c400a3db0 0x7f7c400a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c000c00 tx=0x7f7c4c005fd0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.086+0000 7f7c5e58e700 1 -- 172.21.15.31:0/2806218800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c581025a0 msgr2=0x7f7c5819f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.086+0000 7f7c5e58e700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c581025a0 0x7f7c5819f760 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800b530 tx=0x7f7c4800b840 comp rx=0 tx=0).stop 2024-04-26T17:58:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.087+0000 7f7c5e58e700 1 -- 172.21.15.31:0/2806218800 shutdown_connections 2024-04-26T17:58:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.087+0000 7f7c5e58e700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7c400a3db0 0x7f7c400a6270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.087+0000 7f7c5e58e700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c58107680 0x7f7c58199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.087+0000 7f7c5e58e700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c581025a0 0x7f7c5819f760 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.087+0000 7f7c5e58e700 1 --2- 172.21.15.31:0/2806218800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7c58102f50 0x7f7c5819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.087+0000 7f7c5e58e700 1 -- 172.21.15.31:0/2806218800 >> 172.21.15.31:0/2806218800 conn(0x7f7c580793c0 msgr2=0x7f7c5806be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.087+0000 7f7c5e58e700 1 -- 172.21.15.31:0/2806218800 shutdown_connections 2024-04-26T17:58:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:02.087+0000 7f7c5e58e700 1 -- 172.21.15.31:0/2806218800 wait complete. 2024-04-26T17:58:02.939 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '10d0f28a', seconds elapsed: 43.3 2024-04-26T17:58:02.939 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-26T17:58:02.939 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:58:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 40.0 s, 492.1 tps, lat 1528.620 ms stddev 7231.587 2024-04-26T17:58:03.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:02 smithi186 ceph-mon[32797]: pgmap v49: 97 pgs: 97 active+clean; 1.1 GiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 35 op/s 2024-04-26T17:58:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:02 smithi186 ceph-mon[32797]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-26T17:58:03.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:02 smithi031 ceph-mon[29164]: pgmap v49: 97 pgs: 97 active+clean; 1.1 GiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 35 op/s 2024-04-26T17:58:03.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:02 smithi031 ceph-mon[29164]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-26T17:58:03.325 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:58:03.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:02 smithi097 ceph-mon[32419]: pgmap v49: 97 pgs: 97 active+clean; 1.1 GiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 35 op/s 2024-04-26T17:58:03.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:02 smithi097 ceph-mon[32419]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-26T17:58:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.359+0000 7f005c9f0640 1 -- 172.21.15.31:0/3543581185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0058107cd0 msgr2=0x7f005810f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.359+0000 7f005c9f0640 1 --2- 172.21.15.31:0/3543581185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0058107cd0 0x7f005810f850 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f0050009ca0 tx=0x7f005002a8c0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.359+0000 7f005c9f0640 1 -- 172.21.15.31:0/3543581185 shutdown_connections 2024-04-26T17:58:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.359+0000 7f005c9f0640 1 --2- 172.21.15.31:0/3543581185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0058107cd0 0x7f005810f850 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.359+0000 7f005c9f0640 1 --2- 172.21.15.31:0/3543581185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0058101a70 0x7f00581075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.359+0000 7f005c9f0640 1 --2- 172.21.15.31:0/3543581185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00581010c0 0x7f00581014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.359+0000 7f005c9f0640 1 -- 172.21.15.31:0/3543581185 >> 172.21.15.31:0/3543581185 conn(0x7f0058077dc0 msgr2=0x7f00580ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.359+0000 7f005c9f0640 1 -- 172.21.15.31:0/3543581185 shutdown_connections 2024-04-26T17:58:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.359+0000 7f005c9f0640 1 -- 172.21.15.31:0/3543581185 wait complete. 2024-04-26T17:58:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.360+0000 7f005c9f0640 1 Processor -- start 2024-04-26T17:58:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.360+0000 7f005c9f0640 1 -- start start 2024-04-26T17:58:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.360+0000 7f005c9f0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00581010c0 0x7f0058199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.360+0000 7f005c9f0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0058107cd0 0x7f00581995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.360+0000 7f005c9f0640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f005819d8f0 0x7f005819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.360+0000 7f005c9f0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0058112be0 con 0x7f00581010c0 2024-04-26T17:58:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.360+0000 7f005c9f0640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0058112a60 con 0x7f005819d8f0 2024-04-26T17:58:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.360+0000 7f005c9f0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0058112d60 con 0x7f0058107cd0 2024-04-26T17:58:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.361+0000 7f005659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00581010c0 0x7f0058199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.361+0000 7f0056d9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f005819d8f0 0x7f005819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.361+0000 7f0056d9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f005819d8f0 0x7f005819dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:40190/0 (socket says 172.21.15.31:40190) 2024-04-26T17:58:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.361+0000 7f0056d9d640 1 -- 172.21.15.31:0/2470719816 learned_addr learned my addr 172.21.15.31:0/2470719816 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.361+0000 7f004dd9b640 1 --2- 172.21.15.31:0/2470719816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0058107cd0 0x7f00581995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.361+0000 7f0056d9d640 1 -- 172.21.15.31:0/2470719816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0058107cd0 msgr2=0x7f00581995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.361+0000 7f0056d9d640 1 --2- 172.21.15.31:0/2470719816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0058107cd0 0x7f00581995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.361+0000 7f0056d9d640 1 -- 172.21.15.31:0/2470719816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00581010c0 msgr2=0x7f0058199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.361+0000 7f0056d9d640 1 --2- 172.21.15.31:0/2470719816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00581010c0 0x7f0058199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.361+0000 7f0056d9d640 1 -- 172.21.15.31:0/2470719816 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005819e3e0 con 0x7f005819d8f0 2024-04-26T17:58:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.361+0000 7f004dd9b640 1 --2- 172.21.15.31:0/2470719816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0058107cd0 0x7f00581995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:58:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.361+0000 7f0056d9d640 1 --2- 172.21.15.31:0/2470719816 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f005819d8f0 0x7f005819dda0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f005002f540 tx=0x7f0050031750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.362+0000 7f004ffff640 1 -- 172.21.15.31:0/2470719816 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050003970 con 0x7f005819d8f0 2024-04-26T17:58:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.362+0000 7f004ffff640 1 -- 172.21.15.31:0/2470719816 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0050004040 con 0x7f005819d8f0 2024-04-26T17:58:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.362+0000 7f005c9f0640 1 -- 172.21.15.31:0/2470719816 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f005819e670 con 0x7f005819d8f0 2024-04-26T17:58:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.362+0000 7f004ffff640 1 -- 172.21.15.31:0/2470719816 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005002f900 con 0x7f005819d8f0 2024-04-26T17:58:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.362+0000 7f005c9f0640 1 -- 172.21.15.31:0/2470719816 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00581a5e90 con 0x7f005819d8f0 2024-04-26T17:58:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.365+0000 7f005f7e3740 1 Processor -- start 2024-04-26T17:58:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.365+0000 7f004ffff640 1 -- 172.21.15.31:0/2470719816 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0050003b10 con 0x7f005819d8f0 2024-04-26T17:58:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.365+0000 7f005f7e3740 1 -- start start 2024-04-26T17:58:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f005f7e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043a91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f005f7e3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043a9720 0x5576043b7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f005f7e3740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043b84f0 0x5576043ba8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f005f7e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55760430f460 con 0x5576043a8e00 2024-04-26T17:58:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f005f7e3740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5576042c9590 con 0x5576043b84f0 2024-04-26T17:58:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f005f7e3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55760430b420 con 0x5576043a9720 2024-04-26T17:58:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f004dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043a91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f004dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043a91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41354/0 (socket says 172.21.15.31:41354) 2024-04-26T17:58:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f004dd9b640 1 -- 172.21.15.31:0/2732521294 learned_addr learned my addr 172.21.15.31:0/2732521294 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f004ffff640 1 --2- 172.21.15.31:0/2470719816 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f00440a3f70 0x7f00440a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f004dd9b640 1 --2- 172.21.15.31:0/2732521294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043b84f0 0x5576043ba8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f005659c640 1 --2- 172.21.15.31:0/2732521294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043a9720 0x5576043b7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f0056d9d640 1 --2- 172.21.15.31:0/2470719816 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f00440a3f70 0x7f00440a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f004ffff640 1 -- 172.21.15.31:0/2470719816 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f00500f1b90 con 0x7f005819d8f0 2024-04-26T17:58:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f004dd9b640 1 -- 172.21.15.31:0/2732521294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043a9720 msgr2=0x5576043b7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f004dd9b640 1 --2- 172.21.15.31:0/2732521294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043a9720 0x5576043b7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f004dd9b640 1 -- 172.21.15.31:0/2732521294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043b84f0 msgr2=0x5576043ba8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f004dd9b640 1 --2- 172.21.15.31:0/2732521294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043b84f0 0x5576043ba8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.366+0000 7f004dd9b640 1 -- 172.21.15.31:0/2732521294 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576042d60d0 con 0x5576043a8e00 2024-04-26T17:58:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.367+0000 7f004dd9b640 1 --2- 172.21.15.31:0/2732521294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043a91e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f00400136f0 tx=0x7f0040009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4031e0bd946d37b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.367+0000 7f0056d9d640 1 --2- 172.21.15.31:0/2470719816 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f00440a3f70 0x7f00440a6430 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00500ef4f0 tx=0x7f005004ae60 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.367+0000 7f004cd99640 1 -- 172.21.15.31:0/2732521294 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00400130d0 con 0x5576043a8e00 2024-04-26T17:58:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.367+0000 7f004cd99640 1 -- 172.21.15.31:0/2732521294 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0040013c70 con 0x5576043a8e00 2024-04-26T17:58:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.367+0000 7f004cd99640 1 -- 172.21.15.31:0/2732521294 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00400055c0 con 0x5576043a8e00 2024-04-26T17:58:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.368+0000 7f005f7e3740 1 -- 172.21.15.31:0/2732521294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 msgr2=0x5576043a91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.368+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2732521294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043a91e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f00400136f0 tx=0x7f0040009530 comp rx=0 tx=0).stop 2024-04-26T17:58:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.368+0000 7f005f7e3740 1 -- 172.21.15.31:0/2732521294 shutdown_connections 2024-04-26T17:58:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.368+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2732521294 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043b84f0 0x5576043ba8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.368+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2732521294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043a9720 0x5576043b7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.368+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2732521294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043a91e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.368+0000 7f005f7e3740 1 -- 172.21.15.31:0/2732521294 >> 172.21.15.31:0/2732521294 conn(0x5576043a5410 msgr2=0x5576043a57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.369+0000 7f005f7e3740 1 -- 172.21.15.31:0/2732521294 shutdown_connections 2024-04-26T17:58:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.369+0000 7f005f7e3740 1 -- 172.21.15.31:0/2732521294 wait complete. 2024-04-26T17:58:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.369+0000 7f005f7e3740 1 -- start start 2024-04-26T17:58:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.369+0000 7f005f7e3740 1 Processor -- start 2024-04-26T17:58:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005f7e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005f7e3740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043a9720 0x5576043b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005f7e3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043b84f0 0x5576043b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005f7e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5576043bb0c0 con 0x5576043a8e00 2024-04-26T17:58:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005f7e3740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x557604296b60 con 0x5576043a9720 2024-04-26T17:58:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005f7e3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5576043bb240 con 0x5576043b84f0 2024-04-26T17:58:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005f7e3740 10 client.0 _ll_register_callbacks cb 0x5576042bb2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005659c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043a9720 0x5576043b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f004dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005659c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043a9720 0x5576043b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:40202/0 (socket says 172.21.15.31:40202) 2024-04-26T17:58:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005659c640 1 -- 172.21.15.31:0/2532279663 learned_addr learned my addr 172.21.15.31:0/2532279663 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f004dd9b640 1 --2- 172.21.15.31:0/2532279663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043b84f0 0x5576043b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005659c640 1 -- 172.21.15.31:0/2532279663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043b84f0 msgr2=0x5576043b5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005659c640 1 --2- 172.21.15.31:0/2532279663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043b84f0 0x5576043b5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005659c640 1 -- 172.21.15.31:0/2532279663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 msgr2=0x5576043b53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005659c640 1 --2- 172.21.15.31:0/2532279663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043b53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005659c640 1 -- 172.21.15.31:0/2532279663 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557604021210 con 0x5576043a9720 2024-04-26T17:58:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f004dd9b640 1 --2- 172.21.15.31:0/2532279663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043b53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:58:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f005659c640 1 --2- 172.21.15.31:0/2532279663 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043a9720 0x5576043b58f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0048012770 tx=0x7f0048012c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b97292a43c2b5c77 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.370+0000 7f003dffb640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T17:58:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f003dffb640 1 -- 172.21.15.31:0/2532279663 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004800f920 con 0x5576043a9720 2024-04-26T17:58:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f003dffb640 1 -- 172.21.15.31:0/2532279663 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f004800fac0 con 0x5576043a9720 2024-04-26T17:58:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f005f7e3740 1 -- 172.21.15.31:0/2532279663 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0040013c70 con 0x5576043a9720 2024-04-26T17:58:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f003dffb640 1 -- 172.21.15.31:0/2532279663 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f004800fd00 con 0x5576043a9720 2024-04-26T17:58:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f005f7e3740 10 client.25208 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f005f7e3740 1 -- 172.21.15.31:0/2532279663 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5576040ed620 con 0x5576043a9720 2024-04-26T17:58:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f003dffb640 1 -- 172.21.15.31:0/2532279663 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0040013c70 con 0x5576043a9720 2024-04-26T17:58:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f003dffb640 1 -- 172.21.15.31:0/2532279663 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f004801c420 con 0x5576043a9720 2024-04-26T17:58:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f005f7e3740 10 client.25208 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f005f7e3740 10 client.25208 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:58:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f005f7e3740 20 client.25208 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2532279663 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5576043c86a0 0x5576043caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f005f7e3740 4 client.25208 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:58:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f005f7e3740 1 -- 172.21.15.31:0/2532279663 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55760431d070 con 0x5576043c86a0 2024-04-26T17:58:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.371+0000 7f004dd9b640 1 --2- 172.21.15.31:0/2532279663 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5576043c86a0 0x5576043caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.373+0000 7f004dd9b640 1 --2- 172.21.15.31:0/2532279663 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5576043c86a0 0x5576043caaf0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24e601d4dd6cee12 server_cookie=eed26c1636264777 in_seq=0 out_seq=0 2024-04-26T17:58:04.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f003dffb640 10 client.25208 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:04.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f003dffb640 1 -- 172.21.15.31:0/2532279663 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55760431d070 con 0x5576043c86a0 2024-04-26T17:58:04.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f003dffb640 10 client.25208 handle_command_reply: tid=0 2024-04-26T17:58:04.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f005f7e3740 1 client.25208 shutdown 2024-04-26T17:58:04.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f005f7e3740 1 -- 172.21.15.31:0/2532279663 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043a9720 msgr2=0x5576043b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:04.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2532279663 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043a9720 0x5576043b58f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0048012770 tx=0x7f0048012c30 comp rx=0 tx=0).stop 2024-04-26T17:58:04.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f005f7e3740 1 -- 172.21.15.31:0/2532279663 shutdown_connections 2024-04-26T17:58:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2532279663 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5576043c86a0 0x5576043caaf0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2532279663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043b84f0 0x5576043b5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f003dffb640 0 client.25208 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2532279663 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043a9720 0x5576043b58f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2532279663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043b53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f005f7e3740 1 -- 172.21.15.31:0/2532279663 >> 172.21.15.31:0/2532279663 conn(0x5576043a5410 msgr2=0x5576043ab280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f005f7e3740 1 -- 172.21.15.31:0/2532279663 shutdown_connections 2024-04-26T17:58:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.374+0000 7f005f7e3740 1 -- 172.21.15.31:0/2532279663 wait complete. 2024-04-26T17:58:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.375+0000 7f005f7e3740 20 client.25208 trim_cache size 0 max 16384 2024-04-26T17:58:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.397+0000 7f005f7e3740 1 Processor -- start 2024-04-26T17:58:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.397+0000 7f005f7e3740 1 -- start start 2024-04-26T17:58:04.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.397+0000 7f005f7e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.398+0000 7f005f7e3740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043a9720 0x5576043a4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.398+0000 7f005f7e3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043b84f0 0x5576043d03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.398+0000 7f005f7e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f00581aea40 con 0x5576043a8e00 2024-04-26T17:58:04.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.398+0000 7f005f7e3740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f00581ac2b0 con 0x5576043a9720 2024-04-26T17:58:04.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.398+0000 7f005f7e3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f00581aa120 con 0x5576043b84f0 2024-04-26T17:58:04.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.399+0000 7f005659c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043a9720 0x5576043a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.399+0000 7f005659c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043a9720 0x5576043a4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:40206/0 (socket says 172.21.15.31:40206) 2024-04-26T17:58:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.399+0000 7f005659c640 1 -- 172.21.15.31:0/908680006 learned_addr learned my addr 172.21.15.31:0/908680006 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.399+0000 7f004dd9b640 1 --2- 172.21.15.31:0/908680006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043b84f0 0x5576043d03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.399+0000 7f004dd9b640 1 --2- 172.21.15.31:0/908680006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.399+0000 7f005659c640 1 -- 172.21.15.31:0/908680006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043b84f0 msgr2=0x5576043d03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.399+0000 7f005659c640 1 --2- 172.21.15.31:0/908680006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043b84f0 0x5576043d03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.399+0000 7f005659c640 1 -- 172.21.15.31:0/908680006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 msgr2=0x5576043c3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.399+0000 7f005659c640 1 --2- 172.21.15.31:0/908680006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043c3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.399+0000 7f005659c640 1 -- 172.21.15.31:0/908680006 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576043a5240 con 0x5576043a9720 2024-04-26T17:58:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.399+0000 7f004dd9b640 1 --2- 172.21.15.31:0/908680006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043b84f0 0x5576043d03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:58:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.399+0000 7f005659c640 1 --2- 172.21.15.31:0/908680006 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043a9720 0x5576043a4430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0048012420 tx=0x7f0048012450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38f40f0f5a4206b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:04.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.400+0000 7f003ffff640 1 -- 172.21.15.31:0/908680006 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0048029020 con 0x5576043a9720 2024-04-26T17:58:04.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.400+0000 7f003ffff640 1 -- 172.21.15.31:0/908680006 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00480054e0 con 0x5576043a9720 2024-04-26T17:58:04.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.400+0000 7f003ffff640 1 -- 172.21.15.31:0/908680006 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004800fb60 con 0x5576043a9720 2024-04-26T17:58:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.400+0000 7f005f7e3740 1 -- 172.21.15.31:0/908680006 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043a9720 msgr2=0x5576043a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.400+0000 7f005f7e3740 1 --2- 172.21.15.31:0/908680006 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043a9720 0x5576043a4430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0048012420 tx=0x7f0048012450 comp rx=0 tx=0).stop 2024-04-26T17:58:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.401+0000 7f005f7e3740 1 -- 172.21.15.31:0/908680006 shutdown_connections 2024-04-26T17:58:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.401+0000 7f005f7e3740 1 --2- 172.21.15.31:0/908680006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043b84f0 0x5576043d03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.401+0000 7f005f7e3740 1 --2- 172.21.15.31:0/908680006 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043a9720 0x5576043a4430 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.401+0000 7f005f7e3740 1 --2- 172.21.15.31:0/908680006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043c3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.401+0000 7f005f7e3740 1 -- 172.21.15.31:0/908680006 >> 172.21.15.31:0/908680006 conn(0x5576043a5410 msgr2=0x5576043a3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.401+0000 7f005f7e3740 1 -- 172.21.15.31:0/908680006 shutdown_connections 2024-04-26T17:58:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.401+0000 7f005f7e3740 1 -- 172.21.15.31:0/908680006 wait complete. 2024-04-26T17:58:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.401+0000 7f005f7e3740 1 -- start start 2024-04-26T17:58:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.401+0000 7f005f7e3740 1 Processor -- start 2024-04-26T17:58:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005f7e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043c0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005f7e3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043a9720 0x5576043c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005f7e3740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043b84f0 0x5576043bd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005f7e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f00581aa770 con 0x5576043a8e00 2024-04-26T17:58:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005f7e3740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f00581abbe0 con 0x5576043b84f0 2024-04-26T17:58:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005f7e3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f00581aae40 con 0x5576043a9720 2024-04-26T17:58:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005f7e3740 10 client.0 _ll_register_callbacks cb 0x7f0048009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043a9720 0x5576043c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043a9720 0x5576043c0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:49864/0 (socket says 172.21.15.31:49864) 2024-04-26T17:58:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005659c640 1 -- 172.21.15.31:0/2565392736 learned_addr learned my addr 172.21.15.31:0/2565392736 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005659c640 1 -- 172.21.15.31:0/2565392736 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043b84f0 msgr2=0x5576043bd4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T17:58:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005659c640 1 --2- 172.21.15.31:0/2565392736 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043b84f0 0x5576043bd4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005659c640 1 -- 172.21.15.31:0/2565392736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 msgr2=0x5576043c0500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T17:58:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005659c640 1 --2- 172.21.15.31:0/2565392736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043c0500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.402+0000 7f005659c640 1 -- 172.21.15.31:0/2565392736 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576043afc70 con 0x5576043a9720 2024-04-26T17:58:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.403+0000 7f005659c640 1 --2- 172.21.15.31:0/2565392736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043a9720 0x5576043c0a40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f004801c9b0 tx=0x7f004801ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85b41f89d23858e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.403+0000 7f003dffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T17:58:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.403+0000 7f003dffb640 1 -- 172.21.15.31:0/2565392736 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0048032070 con 0x5576043a9720 2024-04-26T17:58:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.403+0000 7f005f7e3740 1 -- 172.21.15.31:0/2565392736 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00480054e0 con 0x5576043a9720 2024-04-26T17:58:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.403+0000 7f003dffb640 1 -- 172.21.15.31:0/2565392736 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f004801c630 con 0x5576043a9720 2024-04-26T17:58:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.403+0000 7f003dffb640 1 -- 172.21.15.31:0/2565392736 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f004803b050 con 0x5576043a9720 2024-04-26T17:58:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.404+0000 7f005f7e3740 10 client.25156 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.404+0000 7f005f7e3740 1 -- 172.21.15.31:0/2565392736 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5576044339a0 con 0x5576043a9720 2024-04-26T17:58:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.404+0000 7f003dffb640 1 -- 172.21.15.31:0/2565392736 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00480054e0 con 0x5576043a9720 2024-04-26T17:58:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.404+0000 7f003dffb640 1 -- 172.21.15.31:0/2565392736 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f0048035020 con 0x5576043a9720 2024-04-26T17:58:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.404+0000 7f005f7e3740 10 client.25156 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.404+0000 7f005f7e3740 10 client.25156 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:58:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.404+0000 7f005f7e3740 20 client.25156 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.404+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2565392736 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x557604434090 0x5576044364b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.404+0000 7f005f7e3740 4 client.25156 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:58:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.404+0000 7f005f7e3740 1 -- 172.21.15.31:0/2565392736 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557604316ee0 con 0x557604434090 2024-04-26T17:58:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.404+0000 7f004dd9b640 1 --2- 172.21.15.31:0/2565392736 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x557604434090 0x5576044364b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.405+0000 7f004dd9b640 1 --2- 172.21.15.31:0/2565392736 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x557604434090 0x5576044364b0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ba292d50e22c4c0 server_cookie=b0a29ad1c43cf914 in_seq=0 out_seq=0 2024-04-26T17:58:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.405+0000 7f003dffb640 10 client.25156 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.424+0000 7f003dffb640 1 -- 172.21.15.31:0/2565392736 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557604316ee0 con 0x557604434090 2024-04-26T17:58:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.424+0000 7f003dffb640 10 client.25156 handle_command_reply: tid=0 2024-04-26T17:58:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.424+0000 7f005f7e3740 1 client.25156 shutdown 2024-04-26T17:58:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.425+0000 7f005f7e3740 1 -- 172.21.15.31:0/2565392736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043a9720 msgr2=0x5576043c0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.425+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2565392736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043a9720 0x5576043c0a40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f004801c9b0 tx=0x7f004801ccb0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.425+0000 7f005f7e3740 1 -- 172.21.15.31:0/2565392736 shutdown_connections 2024-04-26T17:58:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.425+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2565392736 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x557604434090 0x5576044364b0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.425+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2565392736 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5576043b84f0 0x5576043bd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.425+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2565392736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576043a9720 0x5576043c0a40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.425+0000 7f003dffb640 0 client.25156 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.425+0000 7f005f7e3740 1 --2- 172.21.15.31:0/2565392736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576043a8e00 0x5576043c0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.425+0000 7f005f7e3740 1 -- 172.21.15.31:0/2565392736 >> 172.21.15.31:0/2565392736 conn(0x5576043a5410 msgr2=0x5576043a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.425+0000 7f005f7e3740 1 -- 172.21.15.31:0/2565392736 shutdown_connections 2024-04-26T17:58:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.425+0000 7f005f7e3740 1 -- 172.21.15.31:0/2565392736 wait complete. 2024-04-26T17:58:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.425+0000 7f005f7e3740 20 client.25156 trim_cache size 0 max 16384 2024-04-26T17:58:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:58:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:58:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:58:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:58:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.426+0000 7f005c9f0640 1 -- 172.21.15.31:0/2470719816 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f00440a3f70 msgr2=0x7f00440a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.426+0000 7f005c9f0640 1 --2- 172.21.15.31:0/2470719816 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f00440a3f70 0x7f00440a6430 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00500ef4f0 tx=0x7f005004ae60 comp rx=0 tx=0).stop 2024-04-26T17:58:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.426+0000 7f005c9f0640 1 -- 172.21.15.31:0/2470719816 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f005819d8f0 msgr2=0x7f005819dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.426+0000 7f005c9f0640 1 --2- 172.21.15.31:0/2470719816 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f005819d8f0 0x7f005819dda0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f005002f540 tx=0x7f0050031750 comp rx=0 tx=0).stop 2024-04-26T17:58:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.426+0000 7f005c9f0640 1 -- 172.21.15.31:0/2470719816 shutdown_connections 2024-04-26T17:58:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.426+0000 7f005c9f0640 1 --2- 172.21.15.31:0/2470719816 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f005819d8f0 0x7f005819dda0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.426+0000 7f005c9f0640 1 --2- 172.21.15.31:0/2470719816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0058107cd0 0x7f00581995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.426+0000 7f005c9f0640 1 --2- 172.21.15.31:0/2470719816 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f00440a3f70 0x7f00440a6430 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.426+0000 7f005c9f0640 1 --2- 172.21.15.31:0/2470719816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00581010c0 0x7f0058199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.426+0000 7f005c9f0640 1 -- 172.21.15.31:0/2470719816 >> 172.21.15.31:0/2470719816 conn(0x7f0058077dc0 msgr2=0x7f0058105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.426+0000 7f005c9f0640 1 -- 172.21.15.31:0/2470719816 shutdown_connections 2024-04-26T17:58:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:04.426+0000 7f005c9f0640 1 -- 172.21.15.31:0/2470719816 wait complete. 2024-04-26T17:58:05.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:04 smithi031 ceph-mon[29164]: pgmap v50: 97 pgs: 97 active+clean; 1.1 GiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 MiB/s wr, 32 op/s 2024-04-26T17:58:05.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:04 smithi186 ceph-mon[32797]: pgmap v50: 97 pgs: 97 active+clean; 1.1 GiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 MiB/s wr, 32 op/s 2024-04-26T17:58:05.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.291+0000 7f7553490700 1 -- 172.21.15.31:0/2998899069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f754c108390 msgr2=0x7f754c108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:05.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.291+0000 7f7553490700 1 --2- 172.21.15.31:0/2998899069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f754c108390 0x7f754c108770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7548009990 tx=0x7f7548009ca0 comp rx=0 tx=0).stop 2024-04-26T17:58:05.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.291+0000 7f7553490700 1 -- 172.21.15.31:0/2998899069 shutdown_connections 2024-04-26T17:58:05.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.291+0000 7f7553490700 1 --2- 172.21.15.31:0/2998899069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f754c108390 0x7f754c108770 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:05.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.291+0000 7f7553490700 1 --2- 172.21.15.31:0/2998899069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f754c102ce0 0x7f754c10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:05.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.291+0000 7f7553490700 1 --2- 172.21.15.31:0/2998899069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f754c102320 0x7f754c1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:05.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.291+0000 7f7553490700 1 -- 172.21.15.31:0/2998899069 >> 172.21.15.31:0/2998899069 conn(0x7f754c0792d0 msgr2=0x7f754c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:05.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.291+0000 7f7553490700 1 -- 172.21.15.31:0/2998899069 shutdown_connections 2024-04-26T17:58:05.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.292+0000 7f7553490700 1 -- 172.21.15.31:0/2998899069 wait complete. 2024-04-26T17:58:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.293+0000 7f7553490700 1 Processor -- start 2024-04-26T17:58:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.293+0000 7f7553490700 1 -- start start 2024-04-26T17:58:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.293+0000 7f7553490700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f754c102320 0x7f754c1a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7553490700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f754c102ce0 0x7f754c1a1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7553490700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f754c108390 0x7f754c19be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7553490700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f754c112560 con 0x7f754c102ce0 2024-04-26T17:58:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7553490700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1123e0 con 0x7f754c102320 2024-04-26T17:58:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7553490700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1126e0 con 0x7f754c108390 2024-04-26T17:58:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7550a2b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f754c102ce0 0x7f754c1a1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7550a2b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f754c102ce0 0x7f754c1a1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38986/0 (socket says 172.21.15.31:38986) 2024-04-26T17:58:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7550a2b700 1 -- 172.21.15.31:0/2041122312 learned_addr learned my addr 172.21.15.31:0/2041122312 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f755122c700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f754c102320 0x7f754c1a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7551a2d700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f754c108390 0x7f754c19be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7550a2b700 1 -- 172.21.15.31:0/2041122312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f754c108390 msgr2=0x7f754c19be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7550a2b700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f754c108390 0x7f754c19be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7550a2b700 1 -- 172.21.15.31:0/2041122312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f754c102320 msgr2=0x7f754c1a1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7550a2b700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f754c102320 0x7f754c1a1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7550a2b700 1 -- 172.21.15.31:0/2041122312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754c19c740 con 0x7f754c102ce0 2024-04-26T17:58:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f755122c700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f754c102320 0x7f754c1a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:58:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.294+0000 7f7550a2b700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f754c102ce0 0x7f754c1a1f60 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f753c00e9c0 tx=0x7f753c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.295+0000 7f75427fc700 1 -- 172.21.15.31:0/2041122312 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c00cbd0 con 0x7f754c102ce0 2024-04-26T17:58:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.295+0000 7f75427fc700 1 -- 172.21.15.31:0/2041122312 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f753c00ee60 con 0x7f754c102ce0 2024-04-26T17:58:05.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.295+0000 7f7553490700 1 -- 172.21.15.31:0/2041122312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f754c19c9d0 con 0x7f754c102ce0 2024-04-26T17:58:05.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.295+0000 7f75427fc700 1 -- 172.21.15.31:0/2041122312 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c0055c0 con 0x7f754c102ce0 2024-04-26T17:58:05.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.295+0000 7f7553490700 1 -- 172.21.15.31:0/2041122312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f754c1a89e0 con 0x7f754c102ce0 2024-04-26T17:58:05.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.296+0000 7f7553490700 1 -- 172.21.15.31:0/2041122312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f754c103a40 con 0x7f754c102ce0 2024-04-26T17:58:05.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.299+0000 7f75427fc700 1 -- 172.21.15.31:0/2041122312 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f753c004750 con 0x7f754c102ce0 2024-04-26T17:58:05.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.300+0000 7f75427fc700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75380a3e30 0x7f75380a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:05.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.300+0000 7f75427fc700 1 -- 172.21.15.31:0/2041122312 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f753c0d0c60 con 0x7f754c102ce0 2024-04-26T17:58:05.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.300+0000 7f755122c700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75380a3e30 0x7f75380a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:05.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.301+0000 7f755122c700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75380a3e30 0x7f75380a62f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7548000c00 tx=0x7f7548005960 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:05.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.303+0000 7f75427fc700 1 -- 172.21.15.31:0/2041122312 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f753c053ff0 con 0x7f754c102ce0 2024-04-26T17:58:05.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:04 smithi097 ceph-mon[32419]: pgmap v50: 97 pgs: 97 active+clean; 1.1 GiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 MiB/s wr, 32 op/s 2024-04-26T17:58:05.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.708+0000 7f7553490700 1 -- 172.21.15.31:0/2041122312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f754c1a8bb0 con 0x7f754c102ce0 2024-04-26T17:58:05.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.709+0000 7f75427fc700 1 -- 172.21.15.31:0/2041122312 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11875 (secure 0 0 0) 0x7f753c09d2f0 con 0x7f754c102ce0 2024-04-26T17:58:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:58:05.709 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T17:57:06.843229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24446,24457,24473]},"id":1}]} 2024-04-26T17:58:05.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.712+0000 7f7553490700 1 -- 172.21.15.31:0/2041122312 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75380a3e30 msgr2=0x7f75380a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:05.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.712+0000 7f7553490700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75380a3e30 0x7f75380a62f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7548000c00 tx=0x7f7548005960 comp rx=0 tx=0).stop 2024-04-26T17:58:05.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.712+0000 7f7553490700 1 -- 172.21.15.31:0/2041122312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f754c102ce0 msgr2=0x7f754c1a1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:05.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.712+0000 7f7553490700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f754c102ce0 0x7f754c1a1f60 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f753c00e9c0 tx=0x7f753c00ecd0 comp rx=0 tx=0).stop 2024-04-26T17:58:05.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.712+0000 7f7553490700 1 -- 172.21.15.31:0/2041122312 shutdown_connections 2024-04-26T17:58:05.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.712+0000 7f7553490700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75380a3e30 0x7f75380a62f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:05.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.712+0000 7f7553490700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f754c102320 0x7f754c1a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:05.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.712+0000 7f7553490700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f754c108390 0x7f754c19be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:05.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.712+0000 7f7553490700 1 --2- 172.21.15.31:0/2041122312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f754c102ce0 0x7f754c1a1f60 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:05.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.712+0000 7f7553490700 1 -- 172.21.15.31:0/2041122312 >> 172.21.15.31:0/2041122312 conn(0x7f754c0792d0 msgr2=0x7f754c100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:05.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.713+0000 7f7553490700 1 -- 172.21.15.31:0/2041122312 shutdown_connections 2024-04-26T17:58:05.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:05.713+0000 7f7553490700 1 -- 172.21.15.31:0/2041122312 wait complete. 2024-04-26T17:58:05.716 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-26T17:58:06.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:05 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2041122312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:58:06.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:05 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2041122312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:58:06.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:05 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2041122312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:58:06.882 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24473 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-26T17:58:06.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24473 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-26T17:58:07.264 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:58:07.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:06 smithi186 ceph-mon[32797]: pgmap v51: 97 pgs: 97 active+clean; 1.1 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 41 MiB/s wr, 240 op/s 2024-04-26T17:58:07.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:06 smithi031 ceph-mon[29164]: pgmap v51: 97 pgs: 97 active+clean; 1.1 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 41 MiB/s wr, 240 op/s 2024-04-26T17:58:07.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:06 smithi097 ceph-mon[32419]: pgmap v51: 97 pgs: 97 active+clean; 1.1 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 41 MiB/s wr, 240 op/s 2024-04-26T17:58:08.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 45.0 s, 3262.0 tps, lat 30.909 ms stddev 29.708 2024-04-26T17:58:09.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.229+0000 7f8ba4314700 1 -- 172.21.15.31:0/852996293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b9c10a960 msgr2=0x7f8b9c10ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:09.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.229+0000 7f8ba4314700 1 --2- 172.21.15.31:0/852996293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b9c10a960 0x7f8b9c10ade0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400b0d0 tx=0x7f8b9400b3e0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.231+0000 7f8ba4314700 1 -- 172.21.15.31:0/852996293 shutdown_connections 2024-04-26T17:58:09.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.231+0000 7f8ba4314700 1 --2- 172.21.15.31:0/852996293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8b9c11cc00 0x7f8b9c11eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.231+0000 7f8ba4314700 1 --2- 172.21.15.31:0/852996293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b9c111c70 0x7f8b9c10a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.231+0000 7f8ba4314700 1 --2- 172.21.15.31:0/852996293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b9c10a960 0x7f8b9c10ade0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.231+0000 7f8ba4314700 1 -- 172.21.15.31:0/852996293 >> 172.21.15.31:0/852996293 conn(0x7f8b9c06e590 msgr2=0x7f8b9c06e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:09.231 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:08 smithi031 ceph-mon[29164]: pgmap v52: 97 pgs: 97 active+clean; 1.1 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 24 MiB/s wr, 230 op/s 2024-04-26T17:58:09.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.231+0000 7f8ba4314700 1 -- 172.21.15.31:0/852996293 shutdown_connections 2024-04-26T17:58:09.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.231+0000 7f8ba4314700 1 -- 172.21.15.31:0/852996293 wait complete. 2024-04-26T17:58:09.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.232+0000 7f8ba4314700 1 Processor -- start 2024-04-26T17:58:09.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.232+0000 7f8ba4314700 1 -- start start 2024-04-26T17:58:09.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.232+0000 7f8ba4314700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b9c10a960 0x7f8b9c1ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.232+0000 7f8ba4314700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b9c111c70 0x7f8b9c1aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.232+0000 7f8ba4314700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8b9c11cc00 0x7f8b9c1a86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.232+0000 7f8ba4314700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c1223e0 con 0x7f8b9c111c70 2024-04-26T17:58:09.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.232+0000 7f8ba4314700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c122260 con 0x7f8b9c11cc00 2024-04-26T17:58:09.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.232+0000 7f8ba4314700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c122560 con 0x7f8b9c10a960 2024-04-26T17:58:09.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.232+0000 7f8ba18af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b9c111c70 0x7f8b9c1aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.232+0000 7f8ba20b0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b9c10a960 0x7f8b9c1ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.232+0000 7f8ba28b1700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8b9c11cc00 0x7f8b9c1a86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.232+0000 7f8ba20b0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b9c10a960 0x7f8b9c1ae5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:49376/0 (socket says 172.21.15.31:49376) 2024-04-26T17:58:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.232+0000 7f8ba20b0700 1 -- 172.21.15.31:0/2031797742 learned_addr learned my addr 172.21.15.31:0/2031797742 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.233+0000 7f8ba20b0700 1 -- 172.21.15.31:0/2031797742 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8b9c11cc00 msgr2=0x7f8b9c1a86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.233+0000 7f8ba20b0700 1 --2- 172.21.15.31:0/2031797742 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8b9c11cc00 0x7f8b9c1a86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.233+0000 7f8ba20b0700 1 -- 172.21.15.31:0/2031797742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b9c111c70 msgr2=0x7f8b9c1aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.233+0000 7f8ba20b0700 1 --2- 172.21.15.31:0/2031797742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b9c111c70 0x7f8b9c1aeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.233+0000 7f8ba20b0700 1 -- 172.21.15.31:0/2031797742 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b9c1a8fa0 con 0x7f8b9c10a960 2024-04-26T17:58:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.233+0000 7f8ba18af700 1 --2- 172.21.15.31:0/2031797742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b9c111c70 0x7f8b9c1aeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:58:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.233+0000 7f8ba20b0700 1 --2- 172.21.15.31:0/2031797742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b9c10a960 0x7f8b9c1ae5f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00d950 tx=0x7f8b8c00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:09.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.233+0000 7f8b937fe700 1 -- 172.21.15.31:0/2031797742 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c016070 con 0x7f8b9c10a960 2024-04-26T17:58:09.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.233+0000 7f8b937fe700 1 -- 172.21.15.31:0/2031797742 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b8c00fd40 con 0x7f8b9c10a960 2024-04-26T17:58:09.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.234+0000 7f8b937fe700 1 -- 172.21.15.31:0/2031797742 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c00e3b0 con 0x7f8b9c10a960 2024-04-26T17:58:09.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.234+0000 7f8ba4314700 1 -- 172.21.15.31:0/2031797742 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b9c1a9290 con 0x7f8b9c10a960 2024-04-26T17:58:09.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.234+0000 7f8ba4314700 1 -- 172.21.15.31:0/2031797742 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b9c1b5570 con 0x7f8b9c10a960 2024-04-26T17:58:09.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.239+0000 7f8baeba6b80 1 Processor -- start 2024-04-26T17:58:09.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.241+0000 7f8baeba6b80 1 -- start start 2024-04-26T17:58:09.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.242+0000 7f8baeba6b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec3694bf30 0x55ec3694c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.242+0000 7f8baeba6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694c850 0x55ec369462e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.242+0000 7f8baeba6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec36946900 0x55ec36946d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.242+0000 7f8baeba6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ec368989c0 con 0x55ec3694c850 2024-04-26T17:58:09.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.242+0000 7f8baeba6b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ec367feac0 con 0x55ec3694bf30 2024-04-26T17:58:09.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.242+0000 7f8baeba6b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ec36894980 con 0x55ec36946900 2024-04-26T17:58:09.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.242+0000 7f8ba18af700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec36946900 0x55ec36946d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.242+0000 7f8ba18af700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec36946900 0x55ec36946d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:49390/0 (socket says 172.21.15.31:49390) 2024-04-26T17:58:09.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.242+0000 7f8ba18af700 1 -- 172.21.15.31:0/3661944845 learned_addr learned my addr 172.21.15.31:0/3661944845 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:09.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.242+0000 7f8ba18af700 1 -- 172.21.15.31:0/3661944845 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec3694bf30 msgr2=0x55ec3694c310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T17:58:09.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.242+0000 7f8ba18af700 1 --2- 172.21.15.31:0/3661944845 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec3694bf30 0x55ec3694c310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.242+0000 7f8ba18af700 1 -- 172.21.15.31:0/3661944845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694c850 msgr2=0x55ec369462e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T17:58:09.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.242+0000 7f8ba18af700 1 --2- 172.21.15.31:0/3661944845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694c850 0x55ec369462e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.242+0000 7f8ba18af700 1 -- 172.21.15.31:0/3661944845 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec367e5a60 con 0x55ec36946900 2024-04-26T17:58:09.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.245+0000 7f8ba18af700 1 --2- 172.21.15.31:0/3661944845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec36946900 0x55ec36946d60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400b0d0 tx=0x7f8b9400be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d721ac4c6afa056 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:09.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.245+0000 7f8b937fe700 1 -- 172.21.15.31:0/2031797742 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8b8c019070 con 0x7f8b9c10a960 2024-04-26T17:58:09.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.246+0000 7f8b90d70700 1 -- 172.21.15.31:0/3661944845 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94003a00 con 0x55ec36946900 2024-04-26T17:58:09.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.246+0000 7f8b90d70700 1 -- 172.21.15.31:0/3661944845 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b94003ba0 con 0x55ec36946900 2024-04-26T17:58:09.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.246+0000 7f8b937fe700 1 --2- 172.21.15.31:0/2031797742 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f8b880a4030 0x7f8b880a64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.246+0000 7f8b937fe700 1 -- 172.21.15.31:0/2031797742 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f8b8c0cfc00 con 0x7f8b9c10a960 2024-04-26T17:58:09.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.246+0000 7f8ba28b1700 1 --2- 172.21.15.31:0/2031797742 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f8b880a4030 0x7f8b880a64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.247+0000 7f8baeba6b80 1 -- 172.21.15.31:0/3661944845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec36946900 msgr2=0x55ec36946d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:09.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.247+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/3661944845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec36946900 0x55ec36946d60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400b0d0 tx=0x7f8b9400be90 comp rx=0 tx=0).stop 2024-04-26T17:58:09.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.248+0000 7f8ba28b1700 1 --2- 172.21.15.31:0/2031797742 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f8b880a4030 0x7f8b880a64f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009bc0 tx=0x7f8b98009260 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:09.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.248+0000 7f8baeba6b80 1 -- 172.21.15.31:0/3661944845 shutdown_connections 2024-04-26T17:58:09.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.248+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/3661944845 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec3694bf30 0x55ec3694c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.248+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/3661944845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec36946900 0x55ec36946d60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.248+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/3661944845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694c850 0x55ec369462e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.248+0000 7f8baeba6b80 1 -- 172.21.15.31:0/3661944845 >> 172.21.15.31:0/3661944845 conn(0x55ec36874750 msgr2=0x55ec36941090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:09.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.250+0000 7f8baeba6b80 1 -- 172.21.15.31:0/3661944845 shutdown_connections 2024-04-26T17:58:09.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.250+0000 7f8baeba6b80 1 -- 172.21.15.31:0/3661944845 wait complete. 2024-04-26T17:58:09.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.251+0000 7f8baeba6b80 1 -- start start 2024-04-26T17:58:09.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.251+0000 7f8baeba6b80 1 Processor -- start 2024-04-26T17:58:09.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.251+0000 7f8baeba6b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec36946900 0x55ec369524b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.251+0000 7f8baeba6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694bf30 0x55ec3694f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.252+0000 7f8baeba6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec3694c850 0x55ec3694fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.252+0000 7f8baeba6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ec369532b0 con 0x55ec3694bf30 2024-04-26T17:58:09.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.252+0000 7f8baeba6b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ec36852c10 con 0x55ec36946900 2024-04-26T17:58:09.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.252+0000 7f8baeba6b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ec36953430 con 0x55ec3694c850 2024-04-26T17:58:09.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.252+0000 7f8baeba6b80 10 client.0 _ll_register_callbacks cb 0x55ec36830ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:09.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.253+0000 7f8ba18af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694bf30 0x55ec3694f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.253+0000 7f8ba18af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694bf30 0x55ec3694f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39022/0 (socket says 172.21.15.31:39022) 2024-04-26T17:58:09.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.253+0000 7f8ba18af700 1 -- 172.21.15.31:0/2480359649 learned_addr learned my addr 172.21.15.31:0/2480359649 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:09.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.253+0000 7f8ba20b0700 1 --2- 172.21.15.31:0/2480359649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec36946900 0x55ec369524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.254+0000 7f8ba18af700 1 -- 172.21.15.31:0/2480359649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec3694c850 msgr2=0x55ec3694fd90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T17:58:09.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.254+0000 7f8ba18af700 1 --2- 172.21.15.31:0/2480359649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec3694c850 0x55ec3694fd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.254+0000 7f8ba18af700 1 -- 172.21.15.31:0/2480359649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec36946900 msgr2=0x55ec369524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:09.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.254+0000 7f8ba18af700 1 --2- 172.21.15.31:0/2480359649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec36946900 0x55ec369524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.254+0000 7f8ba18af700 1 -- 172.21.15.31:0/2480359649 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec36856610 con 0x55ec3694bf30 2024-04-26T17:58:09.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.255+0000 7f8ba18af700 1 --2- 172.21.15.31:0/2480359649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694bf30 0x55ec3694f850 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009e60 tx=0x7f8b94007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38e0fa1591392ad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:09.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.257+0000 7f8b85ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:58:09.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.257+0000 7f8b85ffb700 1 -- 172.21.15.31:0/2480359649 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9402d020 con 0x55ec3694bf30 2024-04-26T17:58:09.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.257+0000 7f8b85ffb700 1 -- 172.21.15.31:0/2480359649 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f8b940076a0 con 0x55ec3694bf30 2024-04-26T17:58:09.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.257+0000 7f8baeba6b80 1 -- 172.21.15.31:0/2480359649 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b94003ba0 con 0x55ec3694bf30 2024-04-26T17:58:09.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.257+0000 7f8b85ffb700 1 -- 172.21.15.31:0/2480359649 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8b94038050 con 0x55ec3694bf30 2024-04-26T17:58:09.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.257+0000 7f8baeba6b80 10 client.15516 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:09.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.257+0000 7f8baeba6b80 1 -- 172.21.15.31:0/2480359649 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ec3695da80 con 0x55ec3694bf30 2024-04-26T17:58:09.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.258+0000 7f8b85ffb700 1 -- 172.21.15.31:0/2480359649 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b94003ba0 con 0x55ec3694bf30 2024-04-26T17:58:09.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.258+0000 7f8b85ffb700 1 -- 172.21.15.31:0/2480359649 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f8b94011070 con 0x55ec3694bf30 2024-04-26T17:58:09.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.258+0000 7f8baeba6b80 10 client.15516 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:09.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.258+0000 7f8baeba6b80 10 client.15516 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T17:58:09.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.258+0000 7f8baeba6b80 20 client.15516 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:09.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.258+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/2480359649 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec3695ff50 0x55ec36962340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.258+0000 7f8baeba6b80 4 client.15516 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:58:09.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.258+0000 7f8baeba6b80 1 -- 172.21.15.31:0/2480359649 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec368a63a0 con 0x55ec3695ff50 2024-04-26T17:58:09.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.258+0000 7f8ba20b0700 1 --2- 172.21.15.31:0/2480359649 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec3695ff50 0x55ec36962340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.259+0000 7f8b85ffb700 10 client.15516 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:09.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.259+0000 7f8ba20b0700 1 --2- 172.21.15.31:0/2480359649 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec3695ff50 0x55ec36962340 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a1c8232a2de182e server_cookie=2e4e728587481de3 in_seq=0 out_seq=0 2024-04-26T17:58:09.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8b85ffb700 1 -- 172.21.15.31:0/2480359649 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ec368a63a0 con 0x55ec3695ff50 2024-04-26T17:58:09.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8b85ffb700 10 client.15516 handle_command_reply: tid=0 2024-04-26T17:58:09.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8baeba6b80 1 client.15516 shutdown 2024-04-26T17:58:09.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8baeba6b80 1 -- 172.21.15.31:0/2480359649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694bf30 msgr2=0x55ec3694f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:09.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/2480359649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694bf30 0x55ec3694f850 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009e60 tx=0x7f8b94007d90 comp rx=0 tx=0).stop 2024-04-26T17:58:09.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8baeba6b80 1 -- 172.21.15.31:0/2480359649 shutdown_connections 2024-04-26T17:58:09.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/2480359649 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec3695ff50 0x55ec36962340 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8b85ffb700 0 client.15516 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:09.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/2480359649 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec36946900 0x55ec369524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/2480359649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec3694c850 0x55ec3694fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/2480359649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694bf30 0x55ec3694f850 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8baeba6b80 1 -- 172.21.15.31:0/2480359649 >> 172.21.15.31:0/2480359649 conn(0x55ec36874750 msgr2=0x55ec369452d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:09.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8baeba6b80 1 -- 172.21.15.31:0/2480359649 shutdown_connections 2024-04-26T17:58:09.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8baeba6b80 1 -- 172.21.15.31:0/2480359649 wait complete. 2024-04-26T17:58:09.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.260+0000 7f8baeba6b80 20 client.15516 trim_cache size 0 max 16384 2024-04-26T17:58:09.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:08 smithi186 ceph-mon[32797]: pgmap v52: 97 pgs: 97 active+clean; 1.1 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 24 MiB/s wr, 230 op/s 2024-04-26T17:58:09.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.291+0000 7f8baeba6b80 1 Processor -- start 2024-04-26T17:58:09.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8baeba6b80 1 -- start start 2024-04-26T17:58:09.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8baeba6b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec36946900 0x55ec3694d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8baeba6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694bf30 0x55ec369462e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8baeba6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec3694c850 0x55ec369410d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8ba18af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694bf30 0x55ec369462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8ba20b0700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec36946900 0x55ec3694d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8baeba6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c1bf290 con 0x55ec3694bf30 2024-04-26T17:58:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8baeba6b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c1bcac0 con 0x55ec36946900 2024-04-26T17:58:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8baeba6b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c1ba910 con 0x55ec3694c850 2024-04-26T17:58:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8ba20b0700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec36946900 0x55ec3694d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:55002/0 (socket says 172.21.15.31:55002) 2024-04-26T17:58:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8ba18af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694bf30 0x55ec369462e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39032/0 (socket says 172.21.15.31:39032) 2024-04-26T17:58:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8ba20b0700 1 -- 172.21.15.31:0/2643539242 learned_addr learned my addr 172.21.15.31:0/2643539242 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8ba28b1700 1 --2- 172.21.15.31:0/2643539242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec3694c850 0x55ec369410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8ba20b0700 1 -- 172.21.15.31:0/2643539242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec3694c850 msgr2=0x55ec369410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8ba20b0700 1 --2- 172.21.15.31:0/2643539242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec3694c850 0x55ec369410d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8ba20b0700 1 -- 172.21.15.31:0/2643539242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694bf30 msgr2=0x55ec369462e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8ba20b0700 1 --2- 172.21.15.31:0/2643539242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694bf30 0x55ec369462e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.292+0000 7f8ba20b0700 1 -- 172.21.15.31:0/2643539242 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec3693d880 con 0x55ec36946900 2024-04-26T17:58:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.293+0000 7f8ba20b0700 1 --2- 172.21.15.31:0/2643539242 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec36946900 0x55ec3694d4d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00e010 tx=0x7f8b8c010a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=221235adaabbf2e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.293+0000 7f8b87fff700 1 -- 172.21.15.31:0/2643539242 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c0b4070 con 0x55ec36946900 2024-04-26T17:58:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.293+0000 7f8b87fff700 1 -- 172.21.15.31:0/2643539242 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b8c099c30 con 0x55ec36946900 2024-04-26T17:58:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.293+0000 7f8b87fff700 1 -- 172.21.15.31:0/2643539242 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c014070 con 0x55ec36946900 2024-04-26T17:58:09.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.294+0000 7f8baeba6b80 1 -- 172.21.15.31:0/2643539242 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec36946900 msgr2=0x55ec3694d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:09.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.294+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/2643539242 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec36946900 0x55ec3694d4d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00e010 tx=0x7f8b8c010a40 comp rx=0 tx=0).stop 2024-04-26T17:58:09.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.295+0000 7f8baeba6b80 1 -- 172.21.15.31:0/2643539242 shutdown_connections 2024-04-26T17:58:09.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.295+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/2643539242 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec36946900 0x55ec3694d4d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.295+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/2643539242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec3694c850 0x55ec369410d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.295+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/2643539242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec3694bf30 0x55ec369462e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.295+0000 7f8baeba6b80 1 -- 172.21.15.31:0/2643539242 >> 172.21.15.31:0/2643539242 conn(0x55ec36874750 msgr2=0x55ec3693af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:09.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.295+0000 7f8baeba6b80 1 -- 172.21.15.31:0/2643539242 shutdown_connections 2024-04-26T17:58:09.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.295+0000 7f8baeba6b80 1 -- 172.21.15.31:0/2643539242 wait complete. 2024-04-26T17:58:09.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.296+0000 7f8baeba6b80 1 -- start start 2024-04-26T17:58:09.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.296+0000 7f8baeba6b80 1 Processor -- start 2024-04-26T17:58:09.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.296+0000 7f8baeba6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec36946900 0x55ec369654a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.296+0000 7f8baeba6b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec3694bf30 0x55ec36963850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.296+0000 7f8baeba6b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec3694c850 0x55ec36963d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.296+0000 7f8baeba6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c1baf80 con 0x55ec36946900 2024-04-26T17:58:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.296+0000 7f8baeba6b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c1bc3f0 con 0x55ec3694c850 2024-04-26T17:58:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.296+0000 7f8baeba6b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c1bb650 con 0x55ec3694bf30 2024-04-26T17:58:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.296+0000 7f8baeba6b80 10 client.0 _ll_register_callbacks cb 0x55ec36830ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.296+0000 7f8ba28b1700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec3694c850 0x55ec36963d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.296+0000 7f8ba28b1700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec3694c850 0x55ec36963d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:55018/0 (socket says 172.21.15.31:55018) 2024-04-26T17:58:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.296+0000 7f8ba28b1700 1 -- 172.21.15.31:0/232249075 learned_addr learned my addr 172.21.15.31:0/232249075 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.296+0000 7f8ba20b0700 1 --2- 172.21.15.31:0/232249075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec36946900 0x55ec369654a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.297+0000 7f8ba18af700 1 --2- 172.21.15.31:0/232249075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec3694bf30 0x55ec36963850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.297+0000 7f8ba28b1700 1 -- 172.21.15.31:0/232249075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec3694bf30 msgr2=0x55ec36963850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.297+0000 7f8ba28b1700 1 --2- 172.21.15.31:0/232249075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec3694bf30 0x55ec36963850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.297+0000 7f8ba28b1700 1 -- 172.21.15.31:0/232249075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec36946900 msgr2=0x55ec369654a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.297+0000 7f8ba28b1700 1 --2- 172.21.15.31:0/232249075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec36946900 0x55ec369654a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.297+0000 7f8ba28b1700 1 -- 172.21.15.31:0/232249075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec36964380 con 0x55ec3694c850 2024-04-26T17:58:09.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.297+0000 7f8ba28b1700 1 --2- 172.21.15.31:0/232249075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec3694c850 0x55ec36963d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8b98011b40 tx=0x7f8b980047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fc56c05955a691b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:09.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.297+0000 7f8b85ffb700 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T17:58:09.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.298+0000 7f8b85ffb700 1 -- 172.21.15.31:0/232249075 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b98003c90 con 0x55ec3694c850 2024-04-26T17:58:09.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.298+0000 7f8b85ffb700 1 -- 172.21.15.31:0/232249075 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f8b9801e420 con 0x55ec3694c850 2024-04-26T17:58:09.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.298+0000 7f8baeba6b80 1 -- 172.21.15.31:0/232249075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b8c099c30 con 0x55ec3694c850 2024-04-26T17:58:09.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.299+0000 7f8b85ffb700 1 -- 172.21.15.31:0/232249075 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8b9802c030 con 0x55ec3694c850 2024-04-26T17:58:09.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.299+0000 7f8baeba6b80 10 client.25232 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:09.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.299+0000 7f8baeba6b80 1 -- 172.21.15.31:0/232249075 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ec369c6f40 con 0x55ec3694c850 2024-04-26T17:58:09.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.301+0000 7f8b85ffb700 1 -- 172.21.15.31:0/232249075 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b8c099c30 con 0x55ec3694c850 2024-04-26T17:58:09.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.301+0000 7f8b85ffb700 1 -- 172.21.15.31:0/232249075 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f8b98031670 con 0x55ec3694c850 2024-04-26T17:58:09.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.301+0000 7f8baeba6b80 10 client.25232 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:09.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.301+0000 7f8baeba6b80 10 client.25232 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T17:58:09.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.301+0000 7f8baeba6b80 20 client.25232 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:09.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.301+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/232249075 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec369c77c0 0x55ec369c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:09.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.301+0000 7f8baeba6b80 4 client.25232 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-26T17:58:09.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.301+0000 7f8baeba6b80 1 -- 172.21.15.31:0/232249075 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x55ec368a0440 con 0x55ec369c77c0 2024-04-26T17:58:09.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.301+0000 7f8ba20b0700 1 --2- 172.21.15.31:0/232249075 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec369c77c0 0x55ec369c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:09.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.302+0000 7f8ba20b0700 1 --2- 172.21.15.31:0/232249075 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec369c77c0 0x55ec369c9be0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3d92e4c7f7b3cdf server_cookie=26943a9600d45c3b in_seq=0 out_seq=0 2024-04-26T17:58:09.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.302+0000 7f8b85ffb700 10 client.25232 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:09.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:08 smithi097 ceph-mon[32419]: pgmap v52: 97 pgs: 97 active+clean; 1.1 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 24 MiB/s wr, 230 op/s 2024-04-26T17:58:09.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.494+0000 7f8b85ffb700 1 -- 172.21.15.31:0/232249075 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55ec368a0440 con 0x55ec369c77c0 2024-04-26T17:58:09.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.494+0000 7f8b85ffb700 10 client.25232 handle_command_reply: tid=0 2024-04-26T17:58:09.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.494+0000 7f8baeba6b80 1 client.25232 shutdown 2024-04-26T17:58:09.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.494+0000 7f8baeba6b80 1 -- 172.21.15.31:0/232249075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec3694c850 msgr2=0x55ec36963d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:09.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.494+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/232249075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec3694c850 0x55ec36963d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8b98011b40 tx=0x7f8b980047c0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.494+0000 7f8baeba6b80 1 -- 172.21.15.31:0/232249075 shutdown_connections 2024-04-26T17:58:09.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.494+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/232249075 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec369c77c0 0x55ec369c9be0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.494+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/232249075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec3694c850 0x55ec36963d90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.494+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/232249075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec36946900 0x55ec369654a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.494+0000 7f8baeba6b80 1 --2- 172.21.15.31:0/232249075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec3694bf30 0x55ec36963850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.494+0000 7f8b85ffb700 0 client.25232 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:09.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.494+0000 7f8baeba6b80 1 -- 172.21.15.31:0/232249075 >> 172.21.15.31:0/232249075 conn(0x55ec36874750 msgr2=0x55ec36950e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:09.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.494+0000 7f8baeba6b80 1 -- 172.21.15.31:0/232249075 shutdown_connections 2024-04-26T17:58:09.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.495+0000 7f8baeba6b80 1 -- 172.21.15.31:0/232249075 wait complete. 2024-04-26T17:58:09.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.495+0000 7f8baeba6b80 20 client.25232 trim_cache size 0 max 16384 2024-04-26T17:58:09.496 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T17:58:09.496 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 29, 2024-04-26T17:58:09.497 INFO:teuthology.orchestra.run.smithi031.stdout: "leader": 24473, 2024-04-26T17:58:09.497 INFO:teuthology.orchestra.run.smithi031.stdout: "set_version": 6, 2024-04-26T17:58:09.497 INFO:teuthology.orchestra.run.smithi031.stdout: "sets": { 2024-04-26T17:58:09.497 INFO:teuthology.orchestra.run.smithi031.stdout: "502c2818": { 2024-04-26T17:58:09.497 INFO:teuthology.orchestra.run.smithi031.stdout: "version": 6, 2024-04-26T17:58:09.497 INFO:teuthology.orchestra.run.smithi031.stdout: "age_ref": 0.0, 2024-04-26T17:58:09.497 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T17:58:09.497 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "QUIESCED", 2024-04-26T17:58:09.497 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T17:58:09.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T17:58:09.497 INFO:teuthology.orchestra.run.smithi031.stdout: "timeout": 60.0, 2024-04-26T17:58:09.497 INFO:teuthology.orchestra.run.smithi031.stdout: "expiration": 90.0, 2024-04-26T17:58:09.497 INFO:teuthology.orchestra.run.smithi031.stdout: "members": { 2024-04-26T17:58:09.497 INFO:teuthology.orchestra.run.smithi031.stdout: "file:/": { 2024-04-26T17:58:09.498 INFO:teuthology.orchestra.run.smithi031.stdout: "excluded": false, 2024-04-26T17:58:09.498 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T17:58:09.498 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "QUIESCED", 2024-04-26T17:58:09.498 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T17:58:09.498 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:09.498 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:09.498 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:09.498 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:09.498 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:09.498 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T17:58:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.495+0000 7f8ba4314700 1 -- 172.21.15.31:0/2031797742 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f8b880a4030 msgr2=0x7f8b880a64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.495+0000 7f8ba4314700 1 --2- 172.21.15.31:0/2031797742 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f8b880a4030 0x7f8b880a64f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009bc0 tx=0x7f8b98009260 comp rx=0 tx=0).stop 2024-04-26T17:58:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.496+0000 7f8ba4314700 1 -- 172.21.15.31:0/2031797742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b9c10a960 msgr2=0x7f8b9c1ae5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.496+0000 7f8ba4314700 1 --2- 172.21.15.31:0/2031797742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b9c10a960 0x7f8b9c1ae5f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00d950 tx=0x7f8b8c00dc60 comp rx=0 tx=0).stop 2024-04-26T17:58:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.496+0000 7f8ba4314700 1 -- 172.21.15.31:0/2031797742 shutdown_connections 2024-04-26T17:58:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.496+0000 7f8ba4314700 1 --2- 172.21.15.31:0/2031797742 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f8b880a4030 0x7f8b880a64f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.496+0000 7f8ba4314700 1 --2- 172.21.15.31:0/2031797742 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8b9c11cc00 0x7f8b9c1a86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.496+0000 7f8ba4314700 1 --2- 172.21.15.31:0/2031797742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b9c10a960 0x7f8b9c1ae5f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.496+0000 7f8ba4314700 1 --2- 172.21.15.31:0/2031797742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b9c111c70 0x7f8b9c1aeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.496+0000 7f8ba4314700 1 -- 172.21.15.31:0/2031797742 >> 172.21.15.31:0/2031797742 conn(0x7f8b9c06e590 msgr2=0x7f8b9c11abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:09.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.496+0000 7f8ba4314700 1 -- 172.21.15.31:0/2031797742 shutdown_connections 2024-04-26T17:58:09.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:09.496+0000 7f8ba4314700 1 -- 172.21.15.31:0/2031797742 wait complete. 2024-04-26T17:58:10.382 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '502c2818', quiesce took 7.4 seconds. Will release after: 30.0 2024-04-26T17:58:11.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:10 smithi186 ceph-mon[32797]: pgmap v53: 97 pgs: 97 active+clean; 1.1 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 24 MiB/s wr, 230 op/s 2024-04-26T17:58:11.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:10 smithi031 ceph-mon[29164]: pgmap v53: 97 pgs: 97 active+clean; 1.1 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 24 MiB/s wr, 230 op/s 2024-04-26T17:58:11.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:10 smithi097 ceph-mon[32419]: pgmap v53: 97 pgs: 97 active+clean; 1.1 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 24 MiB/s wr, 230 op/s 2024-04-26T17:58:13.098 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 50.0 s, 808.4 tps, lat 30.025 ms stddev 32.389 2024-04-26T17:58:13.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:12 smithi186 ceph-mon[32797]: pgmap v54: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 28 MiB/s wr, 451 op/s 2024-04-26T17:58:13.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:12 smithi031 ceph-mon[29164]: pgmap v54: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 28 MiB/s wr, 451 op/s 2024-04-26T17:58:13.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:12 smithi097 ceph-mon[32419]: pgmap v54: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 28 MiB/s wr, 451 op/s 2024-04-26T17:58:15.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:15 smithi186 ceph-mon[32797]: pgmap v55: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 28 MiB/s wr, 429 op/s 2024-04-26T17:58:15.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:15 smithi031 ceph-mon[29164]: pgmap v55: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 28 MiB/s wr, 429 op/s 2024-04-26T17:58:15.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:15 smithi097 ceph-mon[32419]: pgmap v55: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 28 MiB/s wr, 429 op/s 2024-04-26T17:58:17.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:17 smithi186 ceph-mon[32797]: pgmap v56: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 28 MiB/s wr, 432 op/s 2024-04-26T17:58:17.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:17 smithi031 ceph-mon[29164]: pgmap v56: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 28 MiB/s wr, 432 op/s 2024-04-26T17:58:17.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:17 smithi097 ceph-mon[32419]: pgmap v56: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 28 MiB/s wr, 432 op/s 2024-04-26T17:58:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.531+0000 7f719aa23640 1 -- 172.21.15.31:0/2369263372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7194107c60 msgr2=0x7f719410f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.531+0000 7f719aa23640 1 --2- 172.21.15.31:0/2369263372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7194107c60 0x7f719410f830 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f718c009ca0 tx=0x7f718c02a8c0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.531+0000 7f719aa23640 1 -- 172.21.15.31:0/2369263372 shutdown_connections 2024-04-26T17:58:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.531+0000 7f719aa23640 1 --2- 172.21.15.31:0/2369263372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7194107c60 0x7f719410f830 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.531+0000 7f719aa23640 1 --2- 172.21.15.31:0/2369263372 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7194101a50 0x7f7194107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.531+0000 7f719aa23640 1 --2- 172.21.15.31:0/2369263372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71941010a0 0x7f7194101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.531+0000 7f719aa23640 1 -- 172.21.15.31:0/2369263372 >> 172.21.15.31:0/2369263372 conn(0x7f7194077dc0 msgr2=0x7f71940ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:17.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.531+0000 7f719aa23640 1 -- 172.21.15.31:0/2369263372 shutdown_connections 2024-04-26T17:58:17.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.531+0000 7f719aa23640 1 -- 172.21.15.31:0/2369263372 wait complete. 2024-04-26T17:58:17.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.532+0000 7f719aa23640 1 Processor -- start 2024-04-26T17:58:17.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.532+0000 7f719aa23640 1 -- start start 2024-04-26T17:58:17.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.532+0000 7f719aa23640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71941010a0 0x7f719419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.532+0000 7f719aa23640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7194107c60 0x7f719419b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.532+0000 7f719aa23640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f719419fb70 0x7f71941a0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.532+0000 7f719aa23640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7194112bc0 con 0x7f719419fb70 2024-04-26T17:58:17.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.532+0000 7f719aa23640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7194112a40 con 0x7f71941010a0 2024-04-26T17:58:17.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.532+0000 7f719aa23640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7194112d40 con 0x7f7194107c60 2024-04-26T17:58:17.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.532+0000 7f7198fc0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f719419fb70 0x7f71941a0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.532+0000 7f7198fc0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f719419fb70 0x7f71941a0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45834/0 (socket says 172.21.15.31:45834) 2024-04-26T17:58:17.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.532+0000 7f7198fc0640 1 -- 172.21.15.31:0/2787723482 learned_addr learned my addr 172.21.15.31:0/2787723482 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:17.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.533+0000 7f7193fff640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71941010a0 0x7f719419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.533+0000 7f71937fe640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7194107c60 0x7f719419b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.533+0000 7f7198fc0640 1 -- 172.21.15.31:0/2787723482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7194107c60 msgr2=0x7f719419b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:17.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.533+0000 7f7198fc0640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7194107c60 0x7f719419b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.533+0000 7f7198fc0640 1 -- 172.21.15.31:0/2787723482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71941010a0 msgr2=0x7f719419b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:17.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.533+0000 7f7198fc0640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71941010a0 0x7f719419b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.533+0000 7f7198fc0640 1 -- 172.21.15.31:0/2787723482 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71941a0660 con 0x7f719419fb70 2024-04-26T17:58:17.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.533+0000 7f7193fff640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71941010a0 0x7f719419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:58:17.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.533+0000 7f7198fc0640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f719419fb70 0x7f71941a0020 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f718c005cc0 tx=0x7f718c02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:17.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.533+0000 7f71917fa640 1 -- 172.21.15.31:0/2787723482 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c003bf0 con 0x7f719419fb70 2024-04-26T17:58:17.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.533+0000 7f719aa23640 1 -- 172.21.15.31:0/2787723482 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71941a08f0 con 0x7f719419fb70 2024-04-26T17:58:17.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.533+0000 7f71917fa640 1 -- 172.21.15.31:0/2787723482 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f718c003d90 con 0x7f719419fb70 2024-04-26T17:58:17.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.533+0000 7f71917fa640 1 -- 172.21.15.31:0/2787723482 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c02fbc0 con 0x7f719419fb70 2024-04-26T17:58:17.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.533+0000 7f719aa23640 1 -- 172.21.15.31:0/2787723482 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71941a8140 con 0x7f719419fb70 2024-04-26T17:58:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.535+0000 7f71917fa640 1 -- 172.21.15.31:0/2787723482 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f718c02fd60 con 0x7f719419fb70 2024-04-26T17:58:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.536+0000 7f719d81d740 1 Processor -- start 2024-04-26T17:58:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.536+0000 7f719d81d740 1 -- start start 2024-04-26T17:58:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.536+0000 7f71917fa640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f717c0a3dc0 0x7f717c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.536+0000 7f719d81d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b158ff00 0x5563b15902e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.536+0000 7f719d81d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b1590820 0x5563b159aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.536+0000 7f719d81d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b159b0e0 0x5563b159d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.536+0000 7f719d81d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5563b14f6460 con 0x5563b159b0e0 2024-04-26T17:58:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.536+0000 7f719d81d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5563b14b0590 con 0x5563b158ff00 2024-04-26T17:58:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.536+0000 7f719d81d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5563b14f2420 con 0x5563b1590820 2024-04-26T17:58:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.536+0000 7f71937fe640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f717c0a3dc0 0x7f717c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.536+0000 7f71917fa640 1 -- 172.21.15.31:0/2787723482 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f718c0f1ef0 con 0x7f719419fb70 2024-04-26T17:58:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.536+0000 7f7198fc0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b159b0e0 0x5563b159d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.537+0000 7f7198fc0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b159b0e0 0x5563b159d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45848/0 (socket says 172.21.15.31:45848) 2024-04-26T17:58:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.537+0000 7f7198fc0640 1 -- 172.21.15.31:0/2448178537 learned_addr learned my addr 172.21.15.31:0/2448178537 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.537+0000 7f7193fff640 1 --2- 172.21.15.31:0/2448178537 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b158ff00 0x5563b15902e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.537+0000 7f71937fe640 1 --2- 172.21.15.31:0/2448178537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b1590820 0x5563b159aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.537+0000 7f7198fc0640 1 -- 172.21.15.31:0/2448178537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b1590820 msgr2=0x5563b159aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.537+0000 7f7198fc0640 1 --2- 172.21.15.31:0/2448178537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b1590820 0x5563b159aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.537+0000 7f7198fc0640 1 -- 172.21.15.31:0/2448178537 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b158ff00 msgr2=0x5563b15902e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.537+0000 7f7198fc0640 1 --2- 172.21.15.31:0/2448178537 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b158ff00 0x5563b15902e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.537+0000 7f7198fc0640 1 -- 172.21.15.31:0/2448178537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563b14bd0d0 con 0x5563b159b0e0 2024-04-26T17:58:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.537+0000 7f71937fe640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f717c0a3dc0 0x7f717c0a6280 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7188005fd0 tx=0x7f71880138e0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.537+0000 7f7198fc0640 1 --2- 172.21.15.31:0/2448178537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b159b0e0 0x5563b159d4d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f718c0ba960 tx=0x7f718c0bdb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f34cf650b6752b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.537+0000 7f717a7fc640 1 -- 172.21.15.31:0/2448178537 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c03be80 con 0x5563b159b0e0 2024-04-26T17:58:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.537+0000 7f717a7fc640 1 -- 172.21.15.31:0/2448178537 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f718c036070 con 0x5563b159b0e0 2024-04-26T17:58:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.537+0000 7f717a7fc640 1 -- 172.21.15.31:0/2448178537 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c0c2340 con 0x5563b159b0e0 2024-04-26T17:58:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.538+0000 7f719d81d740 1 -- 172.21.15.31:0/2448178537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b159b0e0 msgr2=0x5563b159d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.538+0000 7f719d81d740 1 --2- 172.21.15.31:0/2448178537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b159b0e0 0x5563b159d4d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f718c0ba960 tx=0x7f718c0bdb90 comp rx=0 tx=0).stop 2024-04-26T17:58:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.538+0000 7f719d81d740 1 -- 172.21.15.31:0/2448178537 shutdown_connections 2024-04-26T17:58:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.538+0000 7f719d81d740 1 --2- 172.21.15.31:0/2448178537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b159b0e0 0x5563b159d4d0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.538+0000 7f719d81d740 1 --2- 172.21.15.31:0/2448178537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b1590820 0x5563b159aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.538+0000 7f719d81d740 1 --2- 172.21.15.31:0/2448178537 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b158ff00 0x5563b15902e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.538+0000 7f719d81d740 1 -- 172.21.15.31:0/2448178537 >> 172.21.15.31:0/2448178537 conn(0x5563b158c410 msgr2=0x5563b158c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.538+0000 7f719d81d740 1 -- 172.21.15.31:0/2448178537 shutdown_connections 2024-04-26T17:58:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.538+0000 7f719d81d740 1 -- 172.21.15.31:0/2448178537 wait complete. 2024-04-26T17:58:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.538+0000 7f719d81d740 1 -- start start 2024-04-26T17:58:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f719d81d740 1 Processor -- start 2024-04-26T17:58:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f719d81d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b158ff00 0x5563b1598040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f719d81d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b1590820 0x5563b1598580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f719d81d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 0x5563b1598ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f719d81d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5563b14f6460 con 0x5563b158ff00 2024-04-26T17:58:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f719d81d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5563b147db60 con 0x5563b1590820 2024-04-26T17:58:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f719d81d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5563b159dad0 con 0x5563b159b0e0 2024-04-26T17:58:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f719d81d740 10 client.0 _ll_register_callbacks cb 0x5563b14a22b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f7193fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b158ff00 0x5563b1598040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f7193fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b158ff00 0x5563b1598040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45856/0 (socket says 172.21.15.31:45856) 2024-04-26T17:58:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f7193fff640 1 -- 172.21.15.31:0/4142907279 learned_addr learned my addr 172.21.15.31:0/4142907279 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f71937fe640 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b1590820 0x5563b1598580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f7198fc0640 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 0x5563b1598ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f7193fff640 1 -- 172.21.15.31:0/4142907279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 msgr2=0x5563b1598ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f7193fff640 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 0x5563b1598ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f7193fff640 1 -- 172.21.15.31:0/4142907279 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b1590820 msgr2=0x5563b1598580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f7193fff640 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b1590820 0x5563b1598580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f7193fff640 1 -- 172.21.15.31:0/4142907279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563b1208210 con 0x5563b158ff00 2024-04-26T17:58:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f71937fe640 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b1590820 0x5563b1598580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:58:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.539+0000 7f7198fc0640 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 0x5563b1598ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:58:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f7193fff640 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b158ff00 0x5563b1598040 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7184012480 tx=0x7f7184012940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c253be990918cd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f71737fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:58:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f71737fe640 1 -- 172.21.15.31:0/4142907279 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71840035f0 con 0x5563b158ff00 2024-04-26T17:58:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f71737fe640 1 -- 172.21.15.31:0/4142907279 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7184003790 con 0x5563b158ff00 2024-04-26T17:58:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f719d81d740 1 -- 172.21.15.31:0/4142907279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f718c036070 con 0x5563b158ff00 2024-04-26T17:58:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f71737fe640 1 -- 172.21.15.31:0/4142907279 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7184003990 con 0x5563b158ff00 2024-04-26T17:58:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f719d81d740 10 client.15534 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f719d81d740 1 -- 172.21.15.31:0/4142907279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5563b12d4620 con 0x5563b158ff00 2024-04-26T17:58:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f71737fe640 1 -- 172.21.15.31:0/4142907279 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f718c036070 con 0x5563b158ff00 2024-04-26T17:58:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f71737fe640 1 -- 172.21.15.31:0/4142907279 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f7184019070 con 0x5563b158ff00 2024-04-26T17:58:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f719d81d740 10 client.15534 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f719d81d740 10 client.15534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:58:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f719d81d740 20 client.15534 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f719d81d740 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5563b15aaf40 0x5563b15ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f719d81d740 4 client.15534 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:58:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.540+0000 7f719d81d740 1 -- 172.21.15.31:0/4142907279 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563b1504070 con 0x5563b15aaf40 2024-04-26T17:58:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.541+0000 7f71937fe640 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5563b15aaf40 0x5563b15ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.541+0000 7f71937fe640 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5563b15aaf40 0x5563b15ad3b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89c513ad2c0b8fde server_cookie=6a1aa5f1cea04163 in_seq=0 out_seq=0 2024-04-26T17:58:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.541+0000 7f71737fe640 10 client.15534 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.542+0000 7f71737fe640 1 -- 172.21.15.31:0/4142907279 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5563b1504070 con 0x5563b15aaf40 2024-04-26T17:58:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.542+0000 7f71737fe640 10 client.15534 handle_command_reply: tid=0 2024-04-26T17:58:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.542+0000 7f719d81d740 1 client.15534 shutdown 2024-04-26T17:58:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.542+0000 7f719d81d740 1 -- 172.21.15.31:0/4142907279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b158ff00 msgr2=0x5563b1598040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.542+0000 7f719d81d740 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b158ff00 0x5563b1598040 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7184012480 tx=0x7f7184012940 comp rx=0 tx=0).stop 2024-04-26T17:58:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.542+0000 7f719d81d740 1 -- 172.21.15.31:0/4142907279 shutdown_connections 2024-04-26T17:58:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.542+0000 7f719d81d740 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5563b15aaf40 0x5563b15ad3b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.542+0000 7f719d81d740 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 0x5563b1598ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.542+0000 7f719d81d740 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b1590820 0x5563b1598580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.542+0000 7f71737fe640 0 client.15534 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.542+0000 7f719d81d740 1 --2- 172.21.15.31:0/4142907279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b158ff00 0x5563b1598040 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.542+0000 7f719d81d740 1 -- 172.21.15.31:0/4142907279 >> 172.21.15.31:0/4142907279 conn(0x5563b158c410 msgr2=0x5563b1592190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.542+0000 7f719d81d740 1 -- 172.21.15.31:0/4142907279 shutdown_connections 2024-04-26T17:58:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.542+0000 7f719d81d740 1 -- 172.21.15.31:0/4142907279 wait complete. 2024-04-26T17:58:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.543+0000 7f719d81d740 20 client.15534 trim_cache size 0 max 16384 2024-04-26T17:58:17.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.557+0000 7f719d81d740 1 Processor -- start 2024-04-26T17:58:17.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.557+0000 7f719d81d740 1 -- start start 2024-04-26T17:58:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f719d81d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b158ff00 0x5563b158bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f719d81d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b1590820 0x5563b15928c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f719d81d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 0x5563b15b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f719d81d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f71941b2f10 con 0x5563b1590820 2024-04-26T17:58:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f719d81d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f71941b0780 con 0x5563b158ff00 2024-04-26T17:58:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f719d81d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f71941ae5d0 con 0x5563b159b0e0 2024-04-26T17:58:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f71937fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b1590820 0x5563b15928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f71937fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b1590820 0x5563b15928c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45872/0 (socket says 172.21.15.31:45872) 2024-04-26T17:58:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f71937fe640 1 -- 172.21.15.31:0/1924335248 learned_addr learned my addr 172.21.15.31:0/1924335248 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f7193fff640 1 --2- 172.21.15.31:0/1924335248 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b158ff00 0x5563b158bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f71937fe640 1 -- 172.21.15.31:0/1924335248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 msgr2=0x5563b15b2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f7198fc0640 1 --2- 172.21.15.31:0/1924335248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 0x5563b15b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f71937fe640 1 --2- 172.21.15.31:0/1924335248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 0x5563b15b2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f71937fe640 1 -- 172.21.15.31:0/1924335248 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b158ff00 msgr2=0x5563b158bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f71937fe640 1 --2- 172.21.15.31:0/1924335248 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b158ff00 0x5563b158bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f71937fe640 1 -- 172.21.15.31:0/1924335248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563b15965b0 con 0x5563b1590820 2024-04-26T17:58:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f7198fc0640 1 --2- 172.21.15.31:0/1924335248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 0x5563b15b2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:58:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f71937fe640 1 --2- 172.21.15.31:0/1924335248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b1590820 0x5563b15928c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7188005d80 tx=0x7f718801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17092be9da1fc8ec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.558+0000 7f7179ffb640 1 -- 172.21.15.31:0/1924335248 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7188028b50 con 0x5563b1590820 2024-04-26T17:58:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.559+0000 7f7179ffb640 1 -- 172.21.15.31:0/1924335248 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71880288c0 con 0x5563b1590820 2024-04-26T17:58:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.559+0000 7f7179ffb640 1 -- 172.21.15.31:0/1924335248 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7188005280 con 0x5563b1590820 2024-04-26T17:58:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.559+0000 7f719d81d740 1 -- 172.21.15.31:0/1924335248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b1590820 msgr2=0x5563b15928c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.559+0000 7f719d81d740 1 --2- 172.21.15.31:0/1924335248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b1590820 0x5563b15928c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7188005d80 tx=0x7f718801d000 comp rx=0 tx=0).stop 2024-04-26T17:58:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.559+0000 7f719d81d740 1 -- 172.21.15.31:0/1924335248 shutdown_connections 2024-04-26T17:58:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.559+0000 7f719d81d740 1 --2- 172.21.15.31:0/1924335248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 0x5563b15b2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.559+0000 7f719d81d740 1 --2- 172.21.15.31:0/1924335248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b1590820 0x5563b15928c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.559+0000 7f719d81d740 1 --2- 172.21.15.31:0/1924335248 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b158ff00 0x5563b158bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.559+0000 7f719d81d740 1 -- 172.21.15.31:0/1924335248 >> 172.21.15.31:0/1924335248 conn(0x5563b158c410 msgr2=0x5563b1599870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.559+0000 7f719d81d740 1 -- 172.21.15.31:0/1924335248 shutdown_connections 2024-04-26T17:58:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.559+0000 7f719d81d740 1 -- 172.21.15.31:0/1924335248 wait complete. 2024-04-26T17:58:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.559+0000 7f719d81d740 1 -- start start 2024-04-26T17:58:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f719d81d740 1 Processor -- start 2024-04-26T17:58:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f719d81d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b158ff00 0x5563b15a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f719d81d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b1590820 0x5563b15a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f719d81d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 0x5563b159fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f719d81d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f71941aec40 con 0x5563b158ff00 2024-04-26T17:58:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f719d81d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f71941b00b0 con 0x5563b1590820 2024-04-26T17:58:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f719d81d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f71941af310 con 0x5563b159b0e0 2024-04-26T17:58:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f719d81d740 10 client.0 _ll_register_callbacks cb 0x7f718400b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f7193fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b158ff00 0x5563b15a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f7193fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b158ff00 0x5563b15a2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45882/0 (socket says 172.21.15.31:45882) 2024-04-26T17:58:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f7193fff640 1 -- 172.21.15.31:0/1690202605 learned_addr learned my addr 172.21.15.31:0/1690202605 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f7198fc0640 1 --2- 172.21.15.31:0/1690202605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 0x5563b159fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f71937fe640 1 --2- 172.21.15.31:0/1690202605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b1590820 0x5563b15a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f7193fff640 1 -- 172.21.15.31:0/1690202605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 msgr2=0x5563b159fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f7193fff640 1 --2- 172.21.15.31:0/1690202605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 0x5563b159fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f7193fff640 1 -- 172.21.15.31:0/1690202605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b1590820 msgr2=0x5563b15a31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f7193fff640 1 --2- 172.21.15.31:0/1690202605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b1590820 0x5563b15a31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.560+0000 7f7193fff640 1 -- 172.21.15.31:0/1690202605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563b15a01d0 con 0x5563b158ff00 2024-04-26T17:58:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f7193fff640 1 --2- 172.21.15.31:0/1690202605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b158ff00 0x5563b15a2c90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f718400b410 tx=0x7f7184005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a32354133c3ebb05 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f71737fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:58:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f71737fe640 1 -- 172.21.15.31:0/1690202605 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7184028040 con 0x5563b158ff00 2024-04-26T17:58:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f71737fe640 1 -- 172.21.15.31:0/1690202605 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7184003b30 con 0x5563b158ff00 2024-04-26T17:58:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f719d81d740 1 -- 172.21.15.31:0/1690202605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71880288c0 con 0x5563b158ff00 2024-04-26T17:58:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f71737fe640 1 -- 172.21.15.31:0/1690202605 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7184003d30 con 0x5563b158ff00 2024-04-26T17:58:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f719d81d740 10 client.15540 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f719d81d740 1 -- 172.21.15.31:0/1690202605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5563b1611d10 con 0x5563b158ff00 2024-04-26T17:58:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f71737fe640 1 -- 172.21.15.31:0/1690202605 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71880288c0 con 0x5563b158ff00 2024-04-26T17:58:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f71737fe640 1 -- 172.21.15.31:0/1690202605 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f718402b070 con 0x5563b158ff00 2024-04-26T17:58:17.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f719d81d740 10 client.15540 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:17.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f719d81d740 10 client.15540 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:58:17.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f719d81d740 20 client.15540 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:17.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f719d81d740 1 --2- 172.21.15.31:0/1690202605 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5563b1612540 0x5563b1614960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:17.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f719d81d740 4 client.15540 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:58:17.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.561+0000 7f719d81d740 1 -- 172.21.15.31:0/1690202605 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5563b14fdee0 con 0x5563b1612540 2024-04-26T17:58:17.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.562+0000 7f71937fe640 1 --2- 172.21.15.31:0/1690202605 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5563b1612540 0x5563b1614960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:17.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.562+0000 7f71937fe640 1 --2- 172.21.15.31:0/1690202605 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5563b1612540 0x5563b1614960 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=965136ea0296035f server_cookie=94b391a2df71ab3 in_seq=0 out_seq=0 2024-04-26T17:58:17.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.562+0000 7f71737fe640 10 client.15540 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f71737fe640 1 -- 172.21.15.31:0/1690202605 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5563b14fdee0 con 0x5563b1612540 2024-04-26T17:58:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f71737fe640 10 client.15540 handle_command_reply: tid=0 2024-04-26T17:58:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f719d81d740 1 client.15540 shutdown 2024-04-26T17:58:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f719d81d740 1 -- 172.21.15.31:0/1690202605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b158ff00 msgr2=0x5563b15a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f719d81d740 1 --2- 172.21.15.31:0/1690202605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b158ff00 0x5563b15a2c90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f718400b410 tx=0x7f7184005710 comp rx=0 tx=0).stop 2024-04-26T17:58:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f719d81d740 1 -- 172.21.15.31:0/1690202605 shutdown_connections 2024-04-26T17:58:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f719d81d740 1 --2- 172.21.15.31:0/1690202605 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5563b1612540 0x5563b1614960 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f719d81d740 1 --2- 172.21.15.31:0/1690202605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563b159b0e0 0x5563b159fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f719d81d740 1 --2- 172.21.15.31:0/1690202605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5563b1590820 0x5563b15a31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f719d81d740 1 --2- 172.21.15.31:0/1690202605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563b158ff00 0x5563b15a2c90 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f71737fe640 0 client.15540 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f719d81d740 1 -- 172.21.15.31:0/1690202605 >> 172.21.15.31:0/1690202605 conn(0x5563b158c410 msgr2=0x5563b159a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f719d81d740 1 -- 172.21.15.31:0/1690202605 shutdown_connections 2024-04-26T17:58:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f719d81d740 1 -- 172.21.15.31:0/1690202605 wait complete. 2024-04-26T17:58:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.574+0000 7f719d81d740 20 client.15540 trim_cache size 0 max 16384 2024-04-26T17:58:17.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:58:17.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:58:17.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:58:17.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:58:17.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.575+0000 7f719aa23640 1 -- 172.21.15.31:0/2787723482 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f717c0a3dc0 msgr2=0x7f717c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:17.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.575+0000 7f719aa23640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f717c0a3dc0 0x7f717c0a6280 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7188005fd0 tx=0x7f71880138e0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.575+0000 7f719aa23640 1 -- 172.21.15.31:0/2787723482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f719419fb70 msgr2=0x7f71941a0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:17.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.575+0000 7f719aa23640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f719419fb70 0x7f71941a0020 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f718c005cc0 tx=0x7f718c02f380 comp rx=0 tx=0).stop 2024-04-26T17:58:17.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.575+0000 7f719aa23640 1 -- 172.21.15.31:0/2787723482 shutdown_connections 2024-04-26T17:58:17.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.575+0000 7f719aa23640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f717c0a3dc0 0x7f717c0a6280 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.575+0000 7f719aa23640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f719419fb70 0x7f71941a0020 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.575+0000 7f719aa23640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7194107c60 0x7f719419b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.575+0000 7f719aa23640 1 --2- 172.21.15.31:0/2787723482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71941010a0 0x7f719419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.575+0000 7f719aa23640 1 -- 172.21.15.31:0/2787723482 >> 172.21.15.31:0/2787723482 conn(0x7f7194077dc0 msgr2=0x7f71941073b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.575+0000 7f719aa23640 1 -- 172.21.15.31:0/2787723482 shutdown_connections 2024-04-26T17:58:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:17.575+0000 7f719aa23640 1 -- 172.21.15.31:0/2787723482 wait complete. 2024-04-26T17:58:18.095 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 55.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T17:58:19.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:19 smithi186 ceph-mon[32797]: pgmap v57: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.5 MiB/s wr, 224 op/s 2024-04-26T17:58:19.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:19 smithi031 ceph-mon[29164]: pgmap v57: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.5 MiB/s wr, 224 op/s 2024-04-26T17:58:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:19 smithi097 ceph-mon[32419]: pgmap v57: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.5 MiB/s wr, 224 op/s 2024-04-26T17:58:21.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:21 smithi186 ceph-mon[32797]: pgmap v58: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.5 MiB/s wr, 224 op/s 2024-04-26T17:58:21.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:21 smithi031 ceph-mon[29164]: pgmap v58: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.5 MiB/s wr, 224 op/s 2024-04-26T17:58:21.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:21 smithi097 ceph-mon[32419]: pgmap v58: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.5 MiB/s wr, 224 op/s 2024-04-26T17:58:23.097 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 60.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T17:58:23.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:23 smithi186 ceph-mon[32797]: pgmap v59: 97 pgs: 97 active+clean; 1.2 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 MiB/s wr, 1.19k op/s 2024-04-26T17:58:23.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:23 smithi031 ceph-mon[29164]: pgmap v59: 97 pgs: 97 active+clean; 1.2 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 MiB/s wr, 1.19k op/s 2024-04-26T17:58:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:23 smithi097 ceph-mon[32419]: pgmap v59: 97 pgs: 97 active+clean; 1.2 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 MiB/s wr, 1.19k op/s 2024-04-26T17:58:25.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:25 smithi031 ceph-mon[29164]: pgmap v60: 97 pgs: 97 active+clean; 1.2 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 12 MiB/s wr, 965 op/s 2024-04-26T17:58:25.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:25 smithi097 ceph-mon[32419]: pgmap v60: 97 pgs: 97 active+clean; 1.2 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 12 MiB/s wr, 965 op/s 2024-04-26T17:58:25.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:25 smithi186 ceph-mon[32797]: pgmap v60: 97 pgs: 97 active+clean; 1.2 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 12 MiB/s wr, 965 op/s 2024-04-26T17:58:27.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:27 smithi031 ceph-mon[29164]: pgmap v61: 97 pgs: 97 active+clean; 1.2 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 990 op/s 2024-04-26T17:58:27.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:27 smithi097 ceph-mon[32419]: pgmap v61: 97 pgs: 97 active+clean; 1.2 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 990 op/s 2024-04-26T17:58:27.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:27 smithi186 ceph-mon[32797]: pgmap v61: 97 pgs: 97 active+clean; 1.2 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 990 op/s 2024-04-26T17:58:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 65.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T17:58:29.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:29 smithi031 ceph-mon[29164]: pgmap v62: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-26T17:58:29.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:29 smithi097 ceph-mon[32419]: pgmap v62: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-26T17:58:29.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:29 smithi186 ceph-mon[32797]: pgmap v62: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-26T17:58:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.674+0000 7fa4f79f2640 1 -- 172.21.15.31:0/954435959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f01032f0 msgr2=0x7fa4f010f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.674+0000 7fa4f79f2640 1 --2- 172.21.15.31:0/954435959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f01032f0 0x7fa4f010f800 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8009cd0 tx=0x7fa4e802a8e0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.675+0000 7fa4f79f2640 1 -- 172.21.15.31:0/954435959 shutdown_connections 2024-04-26T17:58:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.675+0000 7fa4f79f2640 1 --2- 172.21.15.31:0/954435959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f01032f0 0x7fa4f010f800 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.675+0000 7fa4f79f2640 1 --2- 172.21.15.31:0/954435959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4f0102930 0x7fa4f0102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.675+0000 7fa4f79f2640 1 --2- 172.21.15.31:0/954435959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4f0108940 0x7fa4f0108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.675+0000 7fa4f79f2640 1 -- 172.21.15.31:0/954435959 >> 172.21.15.31:0/954435959 conn(0x7fa4f00fe400 msgr2=0x7fa4f0100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.675+0000 7fa4f79f2640 1 -- 172.21.15.31:0/954435959 shutdown_connections 2024-04-26T17:58:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.675+0000 7fa4f79f2640 1 -- 172.21.15.31:0/954435959 wait complete. 2024-04-26T17:58:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.675+0000 7fa4f79f2640 1 Processor -- start 2024-04-26T17:58:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.675+0000 7fa4f79f2640 1 -- start start 2024-04-26T17:58:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.675+0000 7fa4f79f2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4f0102930 0x7fa4f0199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f79f2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f01032f0 0x7fa4f01995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f79f2640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4f0108940 0x7fa4f019d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f79f2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f0112bd0 con 0x7fa4f01032f0 2024-04-26T17:58:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f79f2640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f0112a50 con 0x7fa4f0108940 2024-04-26T17:58:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f79f2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f0112d50 con 0x7fa4f0102930 2024-04-26T17:58:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f4f8d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f01032f0 0x7fa4f01995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f4f8d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f01032f0 0x7fa4f01995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39564/0 (socket says 172.21.15.31:39564) 2024-04-26T17:58:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f4f8d640 1 -- 172.21.15.31:0/3660435465 learned_addr learned my addr 172.21.15.31:0/3660435465 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f578e640 1 --2- 172.21.15.31:0/3660435465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4f0102930 0x7fa4f0199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f5f8f640 1 --2- 172.21.15.31:0/3660435465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4f0108940 0x7fa4f019d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f4f8d640 1 -- 172.21.15.31:0/3660435465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4f0102930 msgr2=0x7fa4f0199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f4f8d640 1 --2- 172.21.15.31:0/3660435465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4f0102930 0x7fa4f0199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f4f8d640 1 -- 172.21.15.31:0/3660435465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4f0108940 msgr2=0x7fa4f019d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f4f8d640 1 --2- 172.21.15.31:0/3660435465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4f0108940 0x7fa4f019d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f4f8d640 1 -- 172.21.15.31:0/3660435465 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f019e080 con 0x7fa4f01032f0 2024-04-26T17:58:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f4f8d640 1 --2- 172.21.15.31:0/3660435465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f01032f0 0x7fa4f01995d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa4e000d950 tx=0x7fa4e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4e67fc640 1 -- 172.21.15.31:0/3660435465 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0017030 con 0x7fa4f01032f0 2024-04-26T17:58:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4e67fc640 1 -- 172.21.15.31:0/3660435465 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4e0004020 con 0x7fa4f01032f0 2024-04-26T17:58:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4f79f2640 1 -- 172.21.15.31:0/3660435465 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4f019e370 con 0x7fa4f01032f0 2024-04-26T17:58:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.676+0000 7fa4e67fc640 1 -- 172.21.15.31:0/3660435465 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e00047a0 con 0x7fa4f01032f0 2024-04-26T17:58:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.677+0000 7fa4f79f2640 1 -- 172.21.15.31:0/3660435465 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4f01a5c00 con 0x7fa4f01032f0 2024-04-26T17:58:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.679+0000 7fa4e67fc640 1 -- 172.21.15.31:0/3660435465 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa4e0009a30 con 0x7fa4f01032f0 2024-04-26T17:58:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.679+0000 7fa4fa7df740 1 Processor -- start 2024-04-26T17:58:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4fa7df740 1 -- start start 2024-04-26T17:58:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4fa7df740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc7f00 0x560567fc82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4fa7df740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fd2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4fa7df740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 0x560567fd54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4fa7df740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560567f2e460 con 0x560567fc8820 2024-04-26T17:58:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4fa7df740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x560567ee8590 con 0x560567fc7f00 2024-04-26T17:58:30.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4fa7df740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560567f2a420 con 0x560567fd30e0 2024-04-26T17:58:30.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4e67fc640 1 --2- 172.21.15.31:0/3660435465 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa4dc0a3dc0 0x7fa4dc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4f578e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fd2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4f578e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fd2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39570/0 (socket says 172.21.15.31:39570) 2024-04-26T17:58:30.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4f578e640 1 -- 172.21.15.31:0/2576517093 learned_addr learned my addr 172.21.15.31:0/2576517093 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:30.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4f5f8f640 1 --2- 172.21.15.31:0/2576517093 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc7f00 0x560567fc82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4f4f8d640 1 --2- 172.21.15.31:0/2576517093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 0x560567fd54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4e67fc640 1 -- 172.21.15.31:0/3660435465 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fa4e00d0120 con 0x7fa4f01032f0 2024-04-26T17:58:30.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4f5f8f640 1 --2- 172.21.15.31:0/3660435465 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa4dc0a3dc0 0x7fa4dc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4f578e640 1 -- 172.21.15.31:0/2576517093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 msgr2=0x560567fd54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:30.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4f578e640 1 --2- 172.21.15.31:0/2576517093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 0x560567fd54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4f578e640 1 -- 172.21.15.31:0/2576517093 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc7f00 msgr2=0x560567fc82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:30.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4f578e640 1 --2- 172.21.15.31:0/2576517093 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc7f00 0x560567fc82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.680+0000 7fa4f578e640 1 -- 172.21.15.31:0/2576517093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560567ef50d0 con 0x560567fc8820 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.681+0000 7fa4f578e640 1 --2- 172.21.15.31:0/2576517093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fd2ba0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00bc50 tx=0x7fa4ec0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d77a9974074b1c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.681+0000 7fa4f5f8f640 1 --2- 172.21.15.31:0/3660435465 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa4dc0a3dc0 0x7fa4dc0a6280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8009430 tx=0x7fa4e8005da0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.681+0000 7fa4db7fe640 1 -- 172.21.15.31:0/2576517093 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec01c480 con 0x560567fc8820 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.681+0000 7fa4db7fe640 1 -- 172.21.15.31:0/2576517093 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4ec01cad0 con 0x560567fc8820 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.681+0000 7fa4db7fe640 1 -- 172.21.15.31:0/2576517093 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec011b00 con 0x560567fc8820 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.682+0000 7fa4fa7df740 1 -- 172.21.15.31:0/2576517093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 msgr2=0x560567fd2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.682+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/2576517093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fd2ba0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00bc50 tx=0x7fa4ec0107a0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.682+0000 7fa4fa7df740 1 -- 172.21.15.31:0/2576517093 shutdown_connections 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.682+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/2576517093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 0x560567fd54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.682+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/2576517093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fd2ba0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.682+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/2576517093 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc7f00 0x560567fc82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.682+0000 7fa4fa7df740 1 -- 172.21.15.31:0/2576517093 >> 172.21.15.31:0/2576517093 conn(0x560567fc4410 msgr2=0x560567fc47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.682+0000 7fa4fa7df740 1 -- 172.21.15.31:0/2576517093 shutdown_connections 2024-04-26T17:58:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.682+0000 7fa4fa7df740 1 -- 172.21.15.31:0/2576517093 wait complete. 2024-04-26T17:58:30.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.682+0000 7fa4fa7df740 1 -- start start 2024-04-26T17:58:30.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.682+0000 7fa4fa7df740 1 Processor -- start 2024-04-26T17:58:30.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4fa7df740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc7f00 0x560567fd0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4fa7df740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc8820 0x560567fd05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4fa7df740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 0x560567fd0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4fa7df740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560567fd5d10 con 0x560567fc7f00 2024-04-26T17:58:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4fa7df740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x560567eb5b60 con 0x560567fc8820 2024-04-26T17:58:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4fa7df740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560567fd5e90 con 0x560567fd30e0 2024-04-26T17:58:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4fa7df740 10 client.0 _ll_register_callbacks cb 0x560567eda2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4f578e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc7f00 0x560567fd0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4f578e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc7f00 0x560567fd0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39572/0 (socket says 172.21.15.31:39572) 2024-04-26T17:58:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4f578e640 1 -- 172.21.15.31:0/1998146406 learned_addr learned my addr 172.21.15.31:0/1998146406 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4f5f8f640 1 --2- 172.21.15.31:0/1998146406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 0x560567fd0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4f4f8d640 1 --2- 172.21.15.31:0/1998146406 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc8820 0x560567fd05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4f578e640 1 -- 172.21.15.31:0/1998146406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 msgr2=0x560567fd0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4f578e640 1 --2- 172.21.15.31:0/1998146406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 0x560567fd0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4f578e640 1 -- 172.21.15.31:0/1998146406 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc8820 msgr2=0x560567fd05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4f578e640 1 --2- 172.21.15.31:0/1998146406 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc8820 0x560567fd05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4f578e640 1 -- 172.21.15.31:0/1998146406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560567c40210 con 0x560567fc7f00 2024-04-26T17:58:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4f578e640 1 --2- 172.21.15.31:0/1998146406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc7f00 0x560567fd0070 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00c010 tx=0x7fa4ec00b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb014c00388598e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:58:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4d8ff9640 1 -- 172.21.15.31:0/1998146406 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec02b060 con 0x560567fc7f00 2024-04-26T17:58:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4d8ff9640 1 -- 172.21.15.31:0/1998146406 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fa4ec01c690 con 0x560567fc7f00 2024-04-26T17:58:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.683+0000 7fa4fa7df740 1 -- 172.21.15.31:0/1998146406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4ec01cad0 con 0x560567fc7f00 2024-04-26T17:58:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.684+0000 7fa4d8ff9640 1 -- 172.21.15.31:0/1998146406 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa4ec01c890 con 0x560567fc7f00 2024-04-26T17:58:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.684+0000 7fa4fa7df740 10 client.15552 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.684+0000 7fa4fa7df740 1 -- 172.21.15.31:0/1998146406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560567d0c620 con 0x560567fc7f00 2024-04-26T17:58:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.684+0000 7fa4d8ff9640 1 -- 172.21.15.31:0/1998146406 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4ec01cad0 con 0x560567fc7f00 2024-04-26T17:58:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.684+0000 7fa4d8ff9640 1 -- 172.21.15.31:0/1998146406 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fa4ec030840 con 0x560567fc7f00 2024-04-26T17:58:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.684+0000 7fa4fa7df740 10 client.15552 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.684+0000 7fa4fa7df740 10 client.15552 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:58:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.684+0000 7fa4fa7df740 20 client.15552 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.684+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/1998146406 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560567fe3160 0x560567fe5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.684+0000 7fa4fa7df740 4 client.15552 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:58:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.684+0000 7fa4fa7df740 1 -- 172.21.15.31:0/1998146406 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560567f3c070 con 0x560567fe3160 2024-04-26T17:58:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.684+0000 7fa4f4f8d640 1 --2- 172.21.15.31:0/1998146406 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560567fe3160 0x560567fe5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.685+0000 7fa4f4f8d640 1 --2- 172.21.15.31:0/1998146406 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560567fe3160 0x560567fe5550 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d65b55a4b347df0 server_cookie=f418765b93a6d577 in_seq=0 out_seq=0 2024-04-26T17:58:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.685+0000 7fa4d8ff9640 10 client.15552 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.685+0000 7fa4d8ff9640 1 -- 172.21.15.31:0/1998146406 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560567f3c070 con 0x560567fe3160 2024-04-26T17:58:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.685+0000 7fa4d8ff9640 10 client.15552 handle_command_reply: tid=0 2024-04-26T17:58:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.685+0000 7fa4fa7df740 1 client.15552 shutdown 2024-04-26T17:58:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.686+0000 7fa4fa7df740 1 -- 172.21.15.31:0/1998146406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc7f00 msgr2=0x560567fd0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.686+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/1998146406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc7f00 0x560567fd0070 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00c010 tx=0x7fa4ec00b7e0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.686+0000 7fa4fa7df740 1 -- 172.21.15.31:0/1998146406 shutdown_connections 2024-04-26T17:58:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.686+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/1998146406 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560567fe3160 0x560567fe5550 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.686+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/1998146406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 0x560567fd0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.686+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/1998146406 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc8820 0x560567fd05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.686+0000 7fa4d8ff9640 0 client.15552 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.686+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/1998146406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc7f00 0x560567fd0070 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.686+0000 7fa4fa7df740 1 -- 172.21.15.31:0/1998146406 >> 172.21.15.31:0/1998146406 conn(0x560567fc4410 msgr2=0x560567fca190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.686+0000 7fa4fa7df740 1 -- 172.21.15.31:0/1998146406 shutdown_connections 2024-04-26T17:58:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.686+0000 7fa4fa7df740 1 -- 172.21.15.31:0/1998146406 wait complete. 2024-04-26T17:58:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.686+0000 7fa4fa7df740 20 client.15552 trim_cache size 0 max 16384 2024-04-26T17:58:30.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.701+0000 7fa4fa7df740 1 Processor -- start 2024-04-26T17:58:30.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.701+0000 7fa4fa7df740 1 -- start start 2024-04-26T17:58:30.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.701+0000 7fa4fa7df740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc7f00 0x560567fd2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.701+0000 7fa4fa7df740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fcf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.701+0000 7fa4fa7df740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 0x560567fca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.701+0000 7fa4fa7df740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f01aeaf0 con 0x560567fc8820 2024-04-26T17:58:30.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.701+0000 7fa4fa7df740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f01ac360 con 0x560567fc7f00 2024-04-26T17:58:30.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.701+0000 7fa4fa7df740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f01aa1b0 con 0x560567fd30e0 2024-04-26T17:58:30.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4f4f8d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fcf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4f578e640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc7f00 0x560567fd2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4f4f8d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fcf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39582/0 (socket says 172.21.15.31:39582) 2024-04-26T17:58:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4f4f8d640 1 -- 172.21.15.31:0/3885302711 learned_addr learned my addr 172.21.15.31:0/3885302711 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4f5f8f640 1 --2- 172.21.15.31:0/3885302711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 0x560567fca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4f4f8d640 1 -- 172.21.15.31:0/3885302711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 msgr2=0x560567fca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4f4f8d640 1 --2- 172.21.15.31:0/3885302711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 0x560567fca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4f4f8d640 1 -- 172.21.15.31:0/3885302711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc7f00 msgr2=0x560567fd2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4f4f8d640 1 --2- 172.21.15.31:0/3885302711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc7f00 0x560567fd2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4f4f8d640 1 -- 172.21.15.31:0/3885302711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560567fd54d0 con 0x560567fc8820 2024-04-26T17:58:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4f578e640 1 --2- 172.21.15.31:0/3885302711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc7f00 0x560567fd2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:58:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4f4f8d640 1 --2- 172.21.15.31:0/3885302711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fcf950 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0005fd0 tx=0x7fa4e00207f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237c23dcc83cdaef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4daffd640 1 -- 172.21.15.31:0/3885302711 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e009c850 con 0x560567fc8820 2024-04-26T17:58:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4daffd640 1 -- 172.21.15.31:0/3885302711 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4e009c150 con 0x560567fc8820 2024-04-26T17:58:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.702+0000 7fa4daffd640 1 -- 172.21.15.31:0/3885302711 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e00cdb80 con 0x560567fc8820 2024-04-26T17:58:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.703+0000 7fa4fa7df740 1 -- 172.21.15.31:0/3885302711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 msgr2=0x560567fcf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.703+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/3885302711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fcf950 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0005fd0 tx=0x7fa4e00207f0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.703+0000 7fa4fa7df740 1 -- 172.21.15.31:0/3885302711 shutdown_connections 2024-04-26T17:58:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.703+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/3885302711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fd30e0 0x560567fca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.703+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/3885302711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fcf950 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.703+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/3885302711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fc7f00 0x560567fd2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.703+0000 7fa4fa7df740 1 -- 172.21.15.31:0/3885302711 >> 172.21.15.31:0/3885302711 conn(0x560567fc4410 msgr2=0x560567fc35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.703+0000 7fa4fa7df740 1 -- 172.21.15.31:0/3885302711 shutdown_connections 2024-04-26T17:58:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.703+0000 7fa4fa7df740 1 -- 172.21.15.31:0/3885302711 wait complete. 2024-04-26T17:58:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.703+0000 7fa4fa7df740 1 -- start start 2024-04-26T17:58:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4fa7df740 1 Processor -- start 2024-04-26T17:58:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4fa7df740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fc7f00 0x560567fdb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4fa7df740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fd6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4fa7df740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fd30e0 0x560567fd6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4fa7df740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f01aa820 con 0x560567fc8820 2024-04-26T17:58:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4fa7df740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f01abc90 con 0x560567fd30e0 2024-04-26T17:58:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4fa7df740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f01aaef0 con 0x560567fc7f00 2024-04-26T17:58:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4fa7df740 10 client.0 _ll_register_callbacks cb 0x7fa4e80099a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4f4f8d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fd6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4f4f8d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fd6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39586/0 (socket says 172.21.15.31:39586) 2024-04-26T17:58:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4f4f8d640 1 -- 172.21.15.31:0/2186113962 learned_addr learned my addr 172.21.15.31:0/2186113962 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4f5f8f640 1 --2- 172.21.15.31:0/2186113962 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fd30e0 0x560567fd6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4f578e640 1 --2- 172.21.15.31:0/2186113962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fc7f00 0x560567fdb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4f4f8d640 1 -- 172.21.15.31:0/2186113962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fc7f00 msgr2=0x560567fdb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4f4f8d640 1 --2- 172.21.15.31:0/2186113962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fc7f00 0x560567fdb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4f4f8d640 1 -- 172.21.15.31:0/2186113962 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fd30e0 msgr2=0x560567fd6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.704+0000 7fa4f4f8d640 1 --2- 172.21.15.31:0/2186113962 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fd30e0 0x560567fd6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4f4f8d640 1 -- 172.21.15.31:0/2186113962 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560567fd7330 con 0x560567fc8820 2024-04-26T17:58:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4f4f8d640 1 --2- 172.21.15.31:0/2186113962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fd6700 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00d00f0 tx=0x7fa4e00a0500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87cb25cc3512e98 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:58:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4d8ff9640 1 -- 172.21.15.31:0/2186113962 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e009c850 con 0x560567fc8820 2024-04-26T17:58:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4d8ff9640 1 -- 172.21.15.31:0/2186113962 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fa4e00041d0 con 0x560567fc8820 2024-04-26T17:58:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4fa7df740 1 -- 172.21.15.31:0/2186113962 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4e009c150 con 0x560567fc8820 2024-04-26T17:58:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4d8ff9640 1 -- 172.21.15.31:0/2186113962 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa4e00cdb80 con 0x560567fc8820 2024-04-26T17:58:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4fa7df740 10 client.15558 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4fa7df740 1 -- 172.21.15.31:0/2186113962 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5605680492a0 con 0x560567fc8820 2024-04-26T17:58:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4d8ff9640 1 -- 172.21.15.31:0/2186113962 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4e009c150 con 0x560567fc8820 2024-04-26T17:58:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4d8ff9640 1 -- 172.21.15.31:0/2186113962 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fa4e0017c10 con 0x560567fc8820 2024-04-26T17:58:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4fa7df740 10 client.15558 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4fa7df740 10 client.15558 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:58:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4fa7df740 20 client.15558 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/2186113962 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560568049900 0x56056804bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4fa7df740 4 client.15558 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:58:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.705+0000 7fa4fa7df740 1 -- 172.21.15.31:0/2186113962 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560567f35ee0 con 0x560568049900 2024-04-26T17:58:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.706+0000 7fa4f578e640 1 --2- 172.21.15.31:0/2186113962 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560568049900 0x56056804bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.706+0000 7fa4f578e640 1 --2- 172.21.15.31:0/2186113962 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560568049900 0x56056804bcf0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12270478b691489a server_cookie=5fa84273c0a5fa40 in_seq=0 out_seq=0 2024-04-26T17:58:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.706+0000 7fa4d8ff9640 10 client.15558 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.713+0000 7fa4d8ff9640 1 -- 172.21.15.31:0/2186113962 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560567f35ee0 con 0x560568049900 2024-04-26T17:58:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.713+0000 7fa4d8ff9640 10 client.15558 handle_command_reply: tid=0 2024-04-26T17:58:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.713+0000 7fa4fa7df740 1 client.15558 shutdown 2024-04-26T17:58:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4fa7df740 1 -- 172.21.15.31:0/2186113962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 msgr2=0x560567fd6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/2186113962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fd6700 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00d00f0 tx=0x7fa4e00a0500 comp rx=0 tx=0).stop 2024-04-26T17:58:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4fa7df740 1 -- 172.21.15.31:0/2186113962 shutdown_connections 2024-04-26T17:58:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/2186113962 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560568049900 0x56056804bcf0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/2186113962 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560567fd30e0 0x560567fd6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4d8ff9640 0 client.15558 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/2186113962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560567fc8820 0x560567fd6700 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4fa7df740 1 --2- 172.21.15.31:0/2186113962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560567fc7f00 0x560567fdb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4fa7df740 1 -- 172.21.15.31:0/2186113962 >> 172.21.15.31:0/2186113962 conn(0x560567fc4410 msgr2=0x560567fcfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4fa7df740 1 -- 172.21.15.31:0/2186113962 shutdown_connections 2024-04-26T17:58:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4fa7df740 1 -- 172.21.15.31:0/2186113962 wait complete. 2024-04-26T17:58:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4fa7df740 20 client.15558 trim_cache size 0 max 16384 2024-04-26T17:58:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:58:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:58:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:58:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:58:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4f79f2640 1 -- 172.21.15.31:0/3660435465 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa4dc0a3dc0 msgr2=0x7fa4dc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4f79f2640 1 --2- 172.21.15.31:0/3660435465 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa4dc0a3dc0 0x7fa4dc0a6280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8009430 tx=0x7fa4e8005da0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4f79f2640 1 -- 172.21.15.31:0/3660435465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f01032f0 msgr2=0x7fa4f01995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4f79f2640 1 --2- 172.21.15.31:0/3660435465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f01032f0 0x7fa4f01995d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa4e000d950 tx=0x7fa4e000de10 comp rx=0 tx=0).stop 2024-04-26T17:58:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4f79f2640 1 -- 172.21.15.31:0/3660435465 shutdown_connections 2024-04-26T17:58:30.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4f79f2640 1 --2- 172.21.15.31:0/3660435465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa4f0108940 0x7fa4f019d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4f79f2640 1 --2- 172.21.15.31:0/3660435465 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa4dc0a3dc0 0x7fa4dc0a6280 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4f79f2640 1 --2- 172.21.15.31:0/3660435465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f01032f0 0x7fa4f01995d0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4f79f2640 1 --2- 172.21.15.31:0/3660435465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4f0102930 0x7fa4f0199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:30.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.714+0000 7fa4f79f2640 1 -- 172.21.15.31:0/3660435465 >> 172.21.15.31:0/3660435465 conn(0x7fa4f00fe400 msgr2=0x7fa4f00fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:30.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.715+0000 7fa4f79f2640 1 -- 172.21.15.31:0/3660435465 shutdown_connections 2024-04-26T17:58:30.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:30.715+0000 7fa4f79f2640 1 -- 172.21.15.31:0/3660435465 wait complete. 2024-04-26T17:58:31.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:31 smithi031 ceph-mon[29164]: pgmap v63: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-26T17:58:31.319 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:31 smithi186 ceph-mon[32797]: pgmap v63: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-26T17:58:31.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:31 smithi097 ceph-mon[32419]: pgmap v63: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-26T17:58:33.099 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 70.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T17:58:33.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:33 smithi031 ceph-mon[29164]: pgmap v64: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-26T17:58:33.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:33 smithi097 ceph-mon[32419]: pgmap v64: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-26T17:58:33.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:33 smithi186 ceph-mon[32797]: pgmap v64: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-26T17:58:35.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:35 smithi031 ceph-mon[29164]: pgmap v65: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 631 KiB/s wr, 59 op/s 2024-04-26T17:58:35.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:35 smithi097 ceph-mon[32419]: pgmap v65: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 631 KiB/s wr, 59 op/s 2024-04-26T17:58:35.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:35 smithi186 ceph-mon[32797]: pgmap v65: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 631 KiB/s wr, 59 op/s 2024-04-26T17:58:36.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:36 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:58:36.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:36 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:58:36.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:36 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:58:37.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:37 smithi031 ceph-mon[29164]: pgmap v66: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 640 KiB/s wr, 63 op/s 2024-04-26T17:58:37.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:37 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:58:37.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:37 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:58:37.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:37 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:58:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:37 smithi097 ceph-mon[32419]: pgmap v66: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 640 KiB/s wr, 63 op/s 2024-04-26T17:58:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:37 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:58:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:37 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:58:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:37 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:58:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:37 smithi186 ceph-mon[32797]: pgmap v66: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 640 KiB/s wr, 63 op/s 2024-04-26T17:58:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:37 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:58:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:37 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:58:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:37 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:58:38.095 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 75.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T17:58:39.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:39 smithi031 ceph-mon[29164]: pgmap v67: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 370 KiB/s wr, 41 op/s 2024-04-26T17:58:39.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:39 smithi097 ceph-mon[32419]: pgmap v67: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 370 KiB/s wr, 41 op/s 2024-04-26T17:58:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:39 smithi186 ceph-mon[32797]: pgmap v67: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 370 KiB/s wr, 41 op/s 2024-04-26T17:58:40.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T17:58:40.635 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:58:41.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:41 smithi031 ceph-mon[29164]: pgmap v68: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 20 KiB/s wr, 10 op/s 2024-04-26T17:58:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:41 smithi097 ceph-mon[32419]: pgmap v68: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 20 KiB/s wr, 10 op/s 2024-04-26T17:58:41.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:41 smithi186 ceph-mon[32797]: pgmap v68: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 20 KiB/s wr, 10 op/s 2024-04-26T17:58:42.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.467+0000 7fa04b752700 1 -- 172.21.15.31:0/399279303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0441094c0 msgr2=0x7fa0441098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.467+0000 7fa04b752700 1 --2- 172.21.15.31:0/399279303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0441094c0 0x7fa0441098a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa040009990 tx=0x7fa040009ca0 comp rx=0 tx=0).stop 2024-04-26T17:58:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.467+0000 7fa04b752700 1 -- 172.21.15.31:0/399279303 shutdown_connections 2024-04-26T17:58:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.467+0000 7fa04b752700 1 --2- 172.21.15.31:0/399279303 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa044103e30 0x7fa044110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.467+0000 7fa04b752700 1 --2- 172.21.15.31:0/399279303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0441094c0 0x7fa0441098a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.467+0000 7fa04b752700 1 --2- 172.21.15.31:0/399279303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa044103470 0x7fa0441038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.467+0000 7fa04b752700 1 -- 172.21.15.31:0/399279303 >> 172.21.15.31:0/399279303 conn(0x7fa04407a580 msgr2=0x7fa04407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.468+0000 7fa04b752700 1 -- 172.21.15.31:0/399279303 shutdown_connections 2024-04-26T17:58:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.468+0000 7fa04b752700 1 -- 172.21.15.31:0/399279303 wait complete. 2024-04-26T17:58:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.468+0000 7fa04b752700 1 Processor -- start 2024-04-26T17:58:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.468+0000 7fa04b752700 1 -- start start 2024-04-26T17:58:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa04b752700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa044103470 0x7fa04419f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa04b752700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa044103e30 0x7fa04419fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa04b752700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0441094c0 0x7fa044199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa04b752700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa0441135f0 con 0x7fa044103470 2024-04-26T17:58:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa04b752700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa044113470 con 0x7fa0441094c0 2024-04-26T17:58:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa04b752700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa044113770 con 0x7fa044103e30 2024-04-26T17:58:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa0494ee700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa044103470 0x7fa04419f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa0494ee700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa044103470 0x7fa04419f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48914/0 (socket says 172.21.15.31:48914) 2024-04-26T17:58:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa0494ee700 1 -- 172.21.15.31:0/2768519529 learned_addr learned my addr 172.21.15.31:0/2768519529 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa048ced700 1 --2- 172.21.15.31:0/2768519529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa044103e30 0x7fa04419fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa049cef700 1 --2- 172.21.15.31:0/2768519529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0441094c0 0x7fa044199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa0494ee700 1 -- 172.21.15.31:0/2768519529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa044103e30 msgr2=0x7fa04419fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa0494ee700 1 --2- 172.21.15.31:0/2768519529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa044103e30 0x7fa04419fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa0494ee700 1 -- 172.21.15.31:0/2768519529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0441094c0 msgr2=0x7fa044199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa0494ee700 1 --2- 172.21.15.31:0/2768519529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0441094c0 0x7fa044199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa0494ee700 1 -- 172.21.15.31:0/2768519529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa04419a170 con 0x7fa044103470 2024-04-26T17:58:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa0494ee700 1 --2- 172.21.15.31:0/2768519529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa044103470 0x7fa04419f850 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa040000c00 tx=0x7fa0400047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.469+0000 7fa03a7fc700 1 -- 172.21.15.31:0/2768519529 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040027070 con 0x7fa044103470 2024-04-26T17:58:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.470+0000 7fa03a7fc700 1 -- 172.21.15.31:0/2768519529 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa040003f20 con 0x7fa044103470 2024-04-26T17:58:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.470+0000 7fa04b752700 1 -- 172.21.15.31:0/2768519529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa04419a400 con 0x7fa044103470 2024-04-26T17:58:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.470+0000 7fa03a7fc700 1 -- 172.21.15.31:0/2768519529 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040022ca0 con 0x7fa044103470 2024-04-26T17:58:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.470+0000 7fa04b752700 1 -- 172.21.15.31:0/2768519529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0441a6720 con 0x7fa044103470 2024-04-26T17:58:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.471+0000 7fa04b752700 1 -- 172.21.15.31:0/2768519529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa04410b930 con 0x7fa044103470 2024-04-26T17:58:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.472+0000 7fa03a7fc700 1 -- 172.21.15.31:0/2768519529 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa04002a070 con 0x7fa044103470 2024-04-26T17:58:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.472+0000 7fa03a7fc700 1 --2- 172.21.15.31:0/2768519529 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa0300a3db0 0x7fa0300a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.472+0000 7fa048ced700 1 --2- 172.21.15.31:0/2768519529 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa0300a3db0 0x7fa0300a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.472+0000 7fa03a7fc700 1 -- 172.21.15.31:0/2768519529 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fa0400d19d0 con 0x7fa044103470 2024-04-26T17:58:42.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.473+0000 7fa048ced700 1 --2- 172.21.15.31:0/2768519529 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa0300a3db0 0x7fa0300a6270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa04407ad30 tx=0x7fa03400a300 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:42.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.476+0000 7fa03a7fc700 1 -- 172.21.15.31:0/2768519529 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa040031070 con 0x7fa044103470 2024-04-26T17:58:42.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.737+0000 7fa04b752700 1 -- 172.21.15.31:0/2768519529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa04406a110 con 0x7fa044103470 2024-04-26T17:58:42.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.738+0000 7fa03a7fc700 1 -- 172.21.15.31:0/2768519529 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11875 (secure 0 0 0) 0x7fa040013070 con 0x7fa044103470 2024-04-26T17:58:42.738 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T17:58:42.739 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T17:57:06.843229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24446,24457,24473]},"id":1}]} 2024-04-26T17:58:42.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.740+0000 7fa04b752700 1 -- 172.21.15.31:0/2768519529 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa0300a3db0 msgr2=0x7fa0300a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:42.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.740+0000 7fa04b752700 1 --2- 172.21.15.31:0/2768519529 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa0300a3db0 0x7fa0300a6270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa04407ad30 tx=0x7fa03400a300 comp rx=0 tx=0).stop 2024-04-26T17:58:42.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.740+0000 7fa04b752700 1 -- 172.21.15.31:0/2768519529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa044103470 msgr2=0x7fa04419f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:42.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.740+0000 7fa04b752700 1 --2- 172.21.15.31:0/2768519529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa044103470 0x7fa04419f850 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa040000c00 tx=0x7fa0400047b0 comp rx=0 tx=0).stop 2024-04-26T17:58:42.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.740+0000 7fa04b752700 1 -- 172.21.15.31:0/2768519529 shutdown_connections 2024-04-26T17:58:42.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.740+0000 7fa04b752700 1 --2- 172.21.15.31:0/2768519529 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa0300a3db0 0x7fa0300a6270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:42.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.740+0000 7fa04b752700 1 --2- 172.21.15.31:0/2768519529 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0441094c0 0x7fa044199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:42.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.740+0000 7fa04b752700 1 --2- 172.21.15.31:0/2768519529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa044103470 0x7fa04419f850 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:42.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.740+0000 7fa04b752700 1 --2- 172.21.15.31:0/2768519529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa044103e30 0x7fa04419fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:42.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.740+0000 7fa04b752700 1 -- 172.21.15.31:0/2768519529 >> 172.21.15.31:0/2768519529 conn(0x7fa04407a580 msgr2=0x7fa044102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:42.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.740+0000 7fa04b752700 1 -- 172.21.15.31:0/2768519529 shutdown_connections 2024-04-26T17:58:42.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:42.740+0000 7fa04b752700 1 -- 172.21.15.31:0/2768519529 wait complete. 2024-04-26T17:58:42.742 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-26T17:58:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 80.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T17:58:43.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:43 smithi031 ceph-mon[29164]: pgmap v69: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 36 KiB/s wr, 15 op/s 2024-04-26T17:58:43.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:43 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2768519529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:58:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:43 smithi097 ceph-mon[32419]: pgmap v69: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 36 KiB/s wr, 15 op/s 2024-04-26T17:58:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:43 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2768519529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:58:43.474 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24473 quiesce db --set-id 502c2818 --release --await' 2024-04-26T17:58:43.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24473 quiesce db --set-id 502c2818 --release --await 2024-04-26T17:58:43.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:43 smithi186 ceph-mon[32797]: pgmap v69: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 36 KiB/s wr, 15 op/s 2024-04-26T17:58:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:43 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2768519529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T17:58:43.732 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T17:58:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.809+0000 7f76fd39e640 1 -- 172.21.15.31:0/3879963702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f8101620 msgr2=0x7f76f8101aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.809+0000 7f76fd39e640 1 --2- 172.21.15.31:0/3879963702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f8101620 0x7f76f8101aa0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009930 tx=0x7f76ec02a420 comp rx=0 tx=0).stop 2024-04-26T17:58:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.809+0000 7f76fd39e640 1 -- 172.21.15.31:0/3879963702 shutdown_connections 2024-04-26T17:58:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.809+0000 7f76fd39e640 1 --2- 172.21.15.31:0/3879963702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76f8101fe0 0x7f76f810e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.809+0000 7f76fd39e640 1 --2- 172.21.15.31:0/3879963702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f8101620 0x7f76f8101aa0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.809+0000 7f76fd39e640 1 --2- 172.21.15.31:0/3879963702 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76f8107630 0x7f76f8107a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.809+0000 7f76fd39e640 1 -- 172.21.15.31:0/3879963702 >> 172.21.15.31:0/3879963702 conn(0x7f76f80fd150 msgr2=0x7f76f80ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.809+0000 7f76fd39e640 1 -- 172.21.15.31:0/3879963702 shutdown_connections 2024-04-26T17:58:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.809+0000 7f76fd39e640 1 -- 172.21.15.31:0/3879963702 wait complete. 2024-04-26T17:58:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.809+0000 7f76fd39e640 1 Processor -- start 2024-04-26T17:58:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76fd39e640 1 -- start start 2024-04-26T17:58:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76fd39e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76f8101620 0x7f76f81990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76fd39e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f8101fe0 0x7f76f8199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76fd39e640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76f8107630 0x7f76f819d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76fd39e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f76f8111970 con 0x7f76f8101fe0 2024-04-26T17:58:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76fd39e640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81117f0 con 0x7f76f8107630 2024-04-26T17:58:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76fd39e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f76f8111af0 con 0x7f76f8101620 2024-04-26T17:58:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76f77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f8101fe0 0x7f76f8199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76f77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f8101fe0 0x7f76f8199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48932/0 (socket says 172.21.15.31:48932) 2024-04-26T17:58:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76f77fe640 1 -- 172.21.15.31:0/3957995636 learned_addr learned my addr 172.21.15.31:0/3957995636 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76fcb9d640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76f8107630 0x7f76f819d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76f7fff640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76f8101620 0x7f76f81990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76f77fe640 1 -- 172.21.15.31:0/3957995636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76f8101620 msgr2=0x7f76f81990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76f77fe640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76f8101620 0x7f76f81990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76f77fe640 1 -- 172.21.15.31:0/3957995636 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76f8107630 msgr2=0x7f76f819d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76f77fe640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76f8107630 0x7f76f819d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76f77fe640 1 -- 172.21.15.31:0/3957995636 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f819e0b0 con 0x7f76f8101fe0 2024-04-26T17:58:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.810+0000 7f76f7fff640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76f8101620 0x7f76f81990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:58:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.811+0000 7f76f77fe640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f8101fe0 0x7f76f8199600 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f76ec005940 tx=0x7f76ec004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.811+0000 7f76f57fa640 1 -- 172.21.15.31:0/3957995636 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec03f030 con 0x7f76f8101fe0 2024-04-26T17:58:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.811+0000 7f76f57fa640 1 -- 172.21.15.31:0/3957995636 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76ec003aa0 con 0x7f76f8101fe0 2024-04-26T17:58:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.811+0000 7f76fd39e640 1 -- 172.21.15.31:0/3957995636 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f819e340 con 0x7f76f8101fe0 2024-04-26T17:58:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.811+0000 7f76f57fa640 1 -- 172.21.15.31:0/3957995636 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec035070 con 0x7f76f8101fe0 2024-04-26T17:58:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.811+0000 7f76fd39e640 1 -- 172.21.15.31:0/3957995636 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f81a5c30 con 0x7f76f8101fe0 2024-04-26T17:58:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.813+0000 7f76f57fa640 1 -- 172.21.15.31:0/3957995636 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f76ec042030 con 0x7f76f8101fe0 2024-04-26T17:58:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.813+0000 7f77013f0740 1 Processor -- start 2024-04-26T17:58:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.813+0000 7f77013f0740 1 -- start start 2024-04-26T17:58:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f77013f0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219030dfc0 0x56219030e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f77013f0740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030e8e0 0x56219031d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f77013f0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219031d6b0 0x56219031faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f77013f0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562190274460 con 0x56219030dfc0 2024-04-26T17:58:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f77013f0740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56219022e590 con 0x56219030e8e0 2024-04-26T17:58:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f77013f0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562190270420 con 0x56219031d6b0 2024-04-26T17:58:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76f57fa640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f76e00a3d90 0x7f76e00a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76fcb9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219030dfc0 0x56219030e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76fcb9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219030dfc0 0x56219030e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48940/0 (socket says 172.21.15.31:48940) 2024-04-26T17:58:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76fcb9d640 1 -- 172.21.15.31:0/3772026030 learned_addr learned my addr 172.21.15.31:0/3772026030 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76fcb9d640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f76e00a3d90 0x7f76e00a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76f77fe640 1 --2- 172.21.15.31:0/3772026030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219031d6b0 0x56219031faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76f7fff640 1 --2- 172.21.15.31:0/3772026030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030e8e0 0x56219031d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76f57fa640 1 -- 172.21.15.31:0/3957995636 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f76ec0f0ff0 con 0x7f76f8101fe0 2024-04-26T17:58:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76fcb9d640 1 -- 172.21.15.31:0/3772026030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219031d6b0 msgr2=0x56219031faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76fcb9d640 1 --2- 172.21.15.31:0/3772026030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219031d6b0 0x56219031faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76fcb9d640 1 -- 172.21.15.31:0/3772026030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030e8e0 msgr2=0x56219031d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76fcb9d640 1 --2- 172.21.15.31:0/3772026030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030e8e0 0x56219031d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76fcb9d640 1 -- 172.21.15.31:0/3772026030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56219023b0d0 con 0x56219030dfc0 2024-04-26T17:58:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76fcb9d640 1 --2- 172.21.15.31:0/3772026030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219030dfc0 0x56219030e3a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f76f000af30 tx=0x7f76f0016700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=533a3aaedae176b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.814+0000 7f76fcb9d640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f76e00a3d90 0x7f76e00a6250 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f76f0009fd0 tx=0x7f76f001a000 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.815+0000 7f76de7fc640 1 -- 172.21.15.31:0/3772026030 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0026070 con 0x56219030dfc0 2024-04-26T17:58:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.815+0000 7f76de7fc640 1 -- 172.21.15.31:0/3772026030 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76f000a590 con 0x56219030dfc0 2024-04-26T17:58:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.815+0000 7f76de7fc640 1 -- 172.21.15.31:0/3772026030 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0013040 con 0x56219030dfc0 2024-04-26T17:58:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.815+0000 7f77013f0740 1 -- 172.21.15.31:0/3772026030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219030dfc0 msgr2=0x56219030e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.815+0000 7f77013f0740 1 --2- 172.21.15.31:0/3772026030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219030dfc0 0x56219030e3a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f76f000af30 tx=0x7f76f0016700 comp rx=0 tx=0).stop 2024-04-26T17:58:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.815+0000 7f77013f0740 1 -- 172.21.15.31:0/3772026030 shutdown_connections 2024-04-26T17:58:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.815+0000 7f77013f0740 1 --2- 172.21.15.31:0/3772026030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219031d6b0 0x56219031faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.815+0000 7f77013f0740 1 --2- 172.21.15.31:0/3772026030 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030e8e0 0x56219031d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.815+0000 7f77013f0740 1 --2- 172.21.15.31:0/3772026030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219030dfc0 0x56219030e3a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.815+0000 7f77013f0740 1 -- 172.21.15.31:0/3772026030 >> 172.21.15.31:0/3772026030 conn(0x56219030a5d0 msgr2=0x56219030a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.815+0000 7f77013f0740 1 -- 172.21.15.31:0/3772026030 shutdown_connections 2024-04-26T17:58:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.815+0000 7f77013f0740 1 -- 172.21.15.31:0/3772026030 wait complete. 2024-04-26T17:58:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.816+0000 7f77013f0740 1 -- start start 2024-04-26T17:58:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.816+0000 7f77013f0740 1 Processor -- start 2024-04-26T17:58:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.816+0000 7f77013f0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030dfc0 0x56219031a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.816+0000 7f77013f0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219030e8e0 0x56219031ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.816+0000 7f77013f0740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219031d6b0 0x56219031b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.816+0000 7f77013f0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562190320280 con 0x56219030e8e0 2024-04-26T17:58:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.816+0000 7f77013f0740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5621901fbb60 con 0x56219031d6b0 2024-04-26T17:58:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.816+0000 7f77013f0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562190320400 con 0x56219030dfc0 2024-04-26T17:58:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.816+0000 7f77013f0740 10 client.0 _ll_register_callbacks cb 0x5621902202b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.816+0000 7f76f77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219030e8e0 0x56219031ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.816+0000 7f76f77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219030e8e0 0x56219031ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48948/0 (socket says 172.21.15.31:48948) 2024-04-26T17:58:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.816+0000 7f76f77fe640 1 -- 172.21.15.31:0/1463208377 learned_addr learned my addr 172.21.15.31:0/1463208377 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.816+0000 7f76f7fff640 1 --2- 172.21.15.31:0/1463208377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030dfc0 0x56219031a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.816+0000 7f76fcb9d640 1 --2- 172.21.15.31:0/1463208377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219031d6b0 0x56219031b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f76f77fe640 1 -- 172.21.15.31:0/1463208377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030dfc0 msgr2=0x56219031a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f76f77fe640 1 --2- 172.21.15.31:0/1463208377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030dfc0 0x56219031a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f76f77fe640 1 -- 172.21.15.31:0/1463208377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219031d6b0 msgr2=0x56219031b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f76f77fe640 1 --2- 172.21.15.31:0/1463208377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219031d6b0 0x56219031b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f76f77fe640 1 -- 172.21.15.31:0/1463208377 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56218ff86210 con 0x56219030e8e0 2024-04-26T17:58:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f76fcb9d640 1 --2- 172.21.15.31:0/1463208377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219031d6b0 0x56219031b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:58:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f76f77fe640 1 --2- 172.21.15.31:0/1463208377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219030e8e0 0x56219031ab70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f76ec0bcc80 tx=0x7f76ec0bce10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e005f75a4f5045f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f76d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:58:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f76d77fe640 1 -- 172.21.15.31:0/1463208377 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec0d5030 con 0x56219030e8e0 2024-04-26T17:58:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f76d77fe640 1 -- 172.21.15.31:0/1463208377 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f76ec003c50 con 0x56219030e8e0 2024-04-26T17:58:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f77013f0740 1 -- 172.21.15.31:0/1463208377 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76f000a590 con 0x56219030e8e0 2024-04-26T17:58:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f76d77fe640 1 -- 172.21.15.31:0/1463208377 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f76ec049040 con 0x56219030e8e0 2024-04-26T17:58:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f77013f0740 10 client.15576 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f77013f0740 1 -- 172.21.15.31:0/1463208377 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562190052620 con 0x56219030e8e0 2024-04-26T17:58:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f76d77fe640 1 -- 172.21.15.31:0/1463208377 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76f000a590 con 0x56219030e8e0 2024-04-26T17:58:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f76d77fe640 1 -- 172.21.15.31:0/1463208377 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f76ec0b95f0 con 0x56219030e8e0 2024-04-26T17:58:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f77013f0740 10 client.15576 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.817+0000 7f77013f0740 10 client.15576 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:58:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.818+0000 7f77013f0740 20 client.15576 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.818+0000 7f77013f0740 1 --2- 172.21.15.31:0/1463208377 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56219032d7d0 0x56219032fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.818+0000 7f77013f0740 4 client.15576 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:58:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.818+0000 7f77013f0740 1 -- 172.21.15.31:0/1463208377 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562190282070 con 0x56219032d7d0 2024-04-26T17:58:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.818+0000 7f76f7fff640 1 --2- 172.21.15.31:0/1463208377 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56219032d7d0 0x56219032fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.818+0000 7f76f7fff640 1 --2- 172.21.15.31:0/1463208377 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56219032d7d0 0x56219032fc10 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d49e95ca9c9201d server_cookie=ece913934067a6a3 in_seq=0 out_seq=0 2024-04-26T17:58:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.818+0000 7f76d77fe640 10 client.15576 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f76d77fe640 1 -- 172.21.15.31:0/1463208377 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562190282070 con 0x56219032d7d0 2024-04-26T17:58:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f76d77fe640 10 client.15576 handle_command_reply: tid=0 2024-04-26T17:58:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f77013f0740 1 client.15576 shutdown 2024-04-26T17:58:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f77013f0740 1 -- 172.21.15.31:0/1463208377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219030e8e0 msgr2=0x56219031ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f77013f0740 1 --2- 172.21.15.31:0/1463208377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219030e8e0 0x56219031ab70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f76ec0bcc80 tx=0x7f76ec0bce10 comp rx=0 tx=0).stop 2024-04-26T17:58:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f77013f0740 1 -- 172.21.15.31:0/1463208377 shutdown_connections 2024-04-26T17:58:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f77013f0740 1 --2- 172.21.15.31:0/1463208377 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56219032d7d0 0x56219032fc10 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f77013f0740 1 --2- 172.21.15.31:0/1463208377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219031d6b0 0x56219031b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f77013f0740 1 --2- 172.21.15.31:0/1463208377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219030e8e0 0x56219031ab70 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f77013f0740 1 --2- 172.21.15.31:0/1463208377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030dfc0 0x56219031a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f76d77fe640 0 client.15576 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f77013f0740 1 -- 172.21.15.31:0/1463208377 >> 172.21.15.31:0/1463208377 conn(0x56219030a5d0 msgr2=0x562190310440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:43.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f77013f0740 1 -- 172.21.15.31:0/1463208377 shutdown_connections 2024-04-26T17:58:43.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f77013f0740 1 -- 172.21.15.31:0/1463208377 wait complete. 2024-04-26T17:58:43.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.819+0000 7f77013f0740 20 client.15576 trim_cache size 0 max 16384 2024-04-26T17:58:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.834+0000 7f77013f0740 1 Processor -- start 2024-04-26T17:58:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.834+0000 7f77013f0740 1 -- start start 2024-04-26T17:58:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f77013f0740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030dfc0 0x562190329140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f77013f0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030e8e0 0x5621903095d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f77013f0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219031d6b0 0x562190335520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f77013f0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81ae9c0 con 0x56219031d6b0 2024-04-26T17:58:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f77013f0740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81ac230 con 0x56219030dfc0 2024-04-26T17:58:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f77013f0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81aa080 con 0x56219030e8e0 2024-04-26T17:58:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f76fcb9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219031d6b0 0x562190335520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f76fcb9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219031d6b0 0x562190335520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48956/0 (socket says 172.21.15.31:48956) 2024-04-26T17:58:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f76fcb9d640 1 -- 172.21.15.31:0/1257359038 learned_addr learned my addr 172.21.15.31:0/1257359038 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f76f77fe640 1 --2- 172.21.15.31:0/1257359038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030e8e0 0x5621903095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f76f7fff640 1 --2- 172.21.15.31:0/1257359038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030dfc0 0x562190329140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f76fcb9d640 1 -- 172.21.15.31:0/1257359038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030e8e0 msgr2=0x5621903095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f76fcb9d640 1 --2- 172.21.15.31:0/1257359038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030e8e0 0x5621903095d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f76fcb9d640 1 -- 172.21.15.31:0/1257359038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030dfc0 msgr2=0x562190329140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f76fcb9d640 1 --2- 172.21.15.31:0/1257359038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030dfc0 0x562190329140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f76fcb9d640 1 -- 172.21.15.31:0/1257359038 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562190330290 con 0x56219031d6b0 2024-04-26T17:58:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f76fcb9d640 1 --2- 172.21.15.31:0/1257359038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219031d6b0 0x562190335520 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f76f000bfd0 tx=0x7f76f0009d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e07c127464ce0e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f76ddffb640 1 -- 172.21.15.31:0/1257359038 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0026040 con 0x56219031d6b0 2024-04-26T17:58:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f76ddffb640 1 -- 172.21.15.31:0/1257359038 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76f0019d10 con 0x56219031d6b0 2024-04-26T17:58:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.835+0000 7f76ddffb640 1 -- 172.21.15.31:0/1257359038 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f002dd30 con 0x56219031d6b0 2024-04-26T17:58:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.836+0000 7f77013f0740 1 -- 172.21.15.31:0/1257359038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219031d6b0 msgr2=0x562190335520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.836+0000 7f77013f0740 1 --2- 172.21.15.31:0/1257359038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219031d6b0 0x562190335520 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f76f000bfd0 tx=0x7f76f0009d00 comp rx=0 tx=0).stop 2024-04-26T17:58:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.836+0000 7f77013f0740 1 -- 172.21.15.31:0/1257359038 shutdown_connections 2024-04-26T17:58:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.836+0000 7f77013f0740 1 --2- 172.21.15.31:0/1257359038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219031d6b0 0x562190335520 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.836+0000 7f77013f0740 1 --2- 172.21.15.31:0/1257359038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030e8e0 0x5621903095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.836+0000 7f77013f0740 1 --2- 172.21.15.31:0/1257359038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030dfc0 0x562190329140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.836+0000 7f77013f0740 1 -- 172.21.15.31:0/1257359038 >> 172.21.15.31:0/1257359038 conn(0x56219030a5d0 msgr2=0x562190308ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.836+0000 7f77013f0740 1 -- 172.21.15.31:0/1257359038 shutdown_connections 2024-04-26T17:58:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.836+0000 7f77013f0740 1 -- 172.21.15.31:0/1257359038 wait complete. 2024-04-26T17:58:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.836+0000 7f77013f0740 1 -- start start 2024-04-26T17:58:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f77013f0740 1 Processor -- start 2024-04-26T17:58:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f77013f0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030dfc0 0x562190325840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f77013f0740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030e8e0 0x562190322620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f77013f0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219031d6b0 0x562190322b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f77013f0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81aa6f0 con 0x56219031d6b0 2024-04-26T17:58:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f77013f0740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81abb60 con 0x56219030e8e0 2024-04-26T17:58:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f77013f0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81aadc0 con 0x56219030dfc0 2024-04-26T17:58:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f77013f0740 10 client.0 _ll_register_callbacks cb 0x7f76ec009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f76fcb9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219031d6b0 0x562190322b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f76fcb9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219031d6b0 0x562190322b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48964/0 (socket says 172.21.15.31:48964) 2024-04-26T17:58:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f76fcb9d640 1 -- 172.21.15.31:0/1967265354 learned_addr learned my addr 172.21.15.31:0/1967265354 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f76f7fff640 1 --2- 172.21.15.31:0/1967265354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030dfc0 0x562190325840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f76f77fe640 1 --2- 172.21.15.31:0/1967265354 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030e8e0 0x562190322620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f76fcb9d640 1 -- 172.21.15.31:0/1967265354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030dfc0 msgr2=0x562190325840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f76fcb9d640 1 --2- 172.21.15.31:0/1967265354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030dfc0 0x562190325840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f76fcb9d640 1 -- 172.21.15.31:0/1967265354 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030e8e0 msgr2=0x562190322620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f76fcb9d640 1 --2- 172.21.15.31:0/1967265354 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030e8e0 0x562190322620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f76fcb9d640 1 -- 172.21.15.31:0/1967265354 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56219031faa0 con 0x56219031d6b0 2024-04-26T17:58:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.837+0000 7f76f77fe640 1 --2- 172.21.15.31:0/1967265354 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030e8e0 0x562190322620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:58:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f76fcb9d640 1 --2- 172.21.15.31:0/1967265354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219031d6b0 0x562190322b60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f76f000ae50 tx=0x7f76f0007100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954736cb97a25dfd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f76d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:58:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f76d77fe640 1 -- 172.21.15.31:0/1967265354 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0026040 con 0x56219031d6b0 2024-04-26T17:58:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f77013f0740 1 -- 172.21.15.31:0/1967265354 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76f0019d10 con 0x56219031d6b0 2024-04-26T17:58:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f76d77fe640 1 -- 172.21.15.31:0/1967265354 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f76f000a880 con 0x56219031d6b0 2024-04-26T17:58:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f76d77fe640 1 -- 172.21.15.31:0/1967265354 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f76f0045050 con 0x56219031d6b0 2024-04-26T17:58:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f77013f0740 10 client.15582 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f77013f0740 1 -- 172.21.15.31:0/1967265354 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56219030ee20 con 0x56219031d6b0 2024-04-26T17:58:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f76d77fe640 1 -- 172.21.15.31:0/1967265354 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76f0019d10 con 0x56219031d6b0 2024-04-26T17:58:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f76d77fe640 1 -- 172.21.15.31:0/1967265354 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f76f0021550 con 0x56219031d6b0 2024-04-26T17:58:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f77013f0740 10 client.15582 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f77013f0740 10 client.15582 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:58:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f77013f0740 20 client.15582 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f77013f0740 1 --2- 172.21.15.31:0/1967265354 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x562190394c20 0x562190397010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f77013f0740 4 client.15582 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:58:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.838+0000 7f77013f0740 1 -- 172.21.15.31:0/1967265354 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56219027bee0 con 0x562190394c20 2024-04-26T17:58:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.839+0000 7f76f7fff640 1 --2- 172.21.15.31:0/1967265354 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x562190394c20 0x562190397010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.839+0000 7f76f7fff640 1 --2- 172.21.15.31:0/1967265354 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x562190394c20 0x562190397010 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=319c19713824e3d9 server_cookie=8fc25618803dcb96 in_seq=0 out_seq=0 2024-04-26T17:58:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.839+0000 7f76d77fe640 10 client.15582 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f76d77fe640 1 -- 172.21.15.31:0/1967265354 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56219027bee0 con 0x562190394c20 2024-04-26T17:58:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f76d77fe640 10 client.15582 handle_command_reply: tid=0 2024-04-26T17:58:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f77013f0740 1 client.15582 shutdown 2024-04-26T17:58:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f77013f0740 1 -- 172.21.15.31:0/1967265354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219031d6b0 msgr2=0x562190322b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f77013f0740 1 --2- 172.21.15.31:0/1967265354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219031d6b0 0x562190322b60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f76f000ae50 tx=0x7f76f0007100 comp rx=0 tx=0).stop 2024-04-26T17:58:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f77013f0740 1 -- 172.21.15.31:0/1967265354 shutdown_connections 2024-04-26T17:58:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f77013f0740 1 --2- 172.21.15.31:0/1967265354 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x562190394c20 0x562190397010 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f77013f0740 1 --2- 172.21.15.31:0/1967265354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56219031d6b0 0x562190322b60 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f77013f0740 1 --2- 172.21.15.31:0/1967265354 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56219030e8e0 0x562190322620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f77013f0740 1 --2- 172.21.15.31:0/1967265354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56219030dfc0 0x562190325840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f76d77fe640 0 client.15582 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f77013f0740 1 -- 172.21.15.31:0/1967265354 >> 172.21.15.31:0/1967265354 conn(0x56219030a5d0 msgr2=0x5621903103a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f77013f0740 1 -- 172.21.15.31:0/1967265354 shutdown_connections 2024-04-26T17:58:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f77013f0740 1 -- 172.21.15.31:0/1967265354 wait complete. 2024-04-26T17:58:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f77013f0740 20 client.15582 trim_cache size 0 max 16384 2024-04-26T17:58:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:58:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:58:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:58:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:58:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f76fd39e640 1 -- 172.21.15.31:0/3957995636 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f76e00a3d90 msgr2=0x7f76e00a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f76fd39e640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f76e00a3d90 0x7f76e00a6250 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f76f0009fd0 tx=0x7f76f001a000 comp rx=0 tx=0).stop 2024-04-26T17:58:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.848+0000 7f76fd39e640 1 -- 172.21.15.31:0/3957995636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f8101fe0 msgr2=0x7f76f8199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.849+0000 7f76fd39e640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f8101fe0 0x7f76f8199600 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f76ec005940 tx=0x7f76ec004270 comp rx=0 tx=0).stop 2024-04-26T17:58:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.849+0000 7f76fd39e640 1 -- 172.21.15.31:0/3957995636 shutdown_connections 2024-04-26T17:58:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.849+0000 7f76fd39e640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f76f8107630 0x7f76f819d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.849+0000 7f76fd39e640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f76e00a3d90 0x7f76e00a6250 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.849+0000 7f76fd39e640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76f8101fe0 0x7f76f8199600 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.849+0000 7f76fd39e640 1 --2- 172.21.15.31:0/3957995636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76f8101620 0x7f76f81990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.849+0000 7f76fd39e640 1 -- 172.21.15.31:0/3957995636 >> 172.21.15.31:0/3957995636 conn(0x7f76f80fd150 msgr2=0x7f76f806a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.849+0000 7f76fd39e640 1 -- 172.21.15.31:0/3957995636 shutdown_connections 2024-04-26T17:58:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:43.849+0000 7f76fd39e640 1 -- 172.21.15.31:0/3957995636 wait complete. 2024-04-26T17:58:45.155 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:45 smithi031 ceph-mon[29164]: pgmap v70: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 31 KiB/s wr, 11 op/s 2024-04-26T17:58:45.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:45 smithi097 ceph-mon[32419]: pgmap v70: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 31 KiB/s wr, 11 op/s 2024-04-26T17:58:45.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.415+0000 7f08778ad700 1 -- 172.21.15.31:0/4102288336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870107680 msgr2=0x7f087010f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:45.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.415+0000 7f08778ad700 1 --2- 172.21.15.31:0/4102288336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870107680 0x7f087010f250 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f086800b0d0 tx=0x7f086800b3e0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.415+0000 7f08778ad700 1 -- 172.21.15.31:0/4102288336 shutdown_connections 2024-04-26T17:58:45.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.415+0000 7f08778ad700 1 --2- 172.21.15.31:0/4102288336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870107680 0x7f087010f250 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.415+0000 7f08778ad700 1 --2- 172.21.15.31:0/4102288336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08701025a0 0x7f0870102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.415+0000 7f08778ad700 1 --2- 172.21.15.31:0/4102288336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0870102f50 0x7f0870106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.415+0000 7f08778ad700 1 -- 172.21.15.31:0/4102288336 >> 172.21.15.31:0/4102288336 conn(0x7f08700793c0 msgr2=0x7f08700797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:45.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.415+0000 7f08778ad700 1 -- 172.21.15.31:0/4102288336 shutdown_connections 2024-04-26T17:58:45.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.415+0000 7f08778ad700 1 -- 172.21.15.31:0/4102288336 wait complete. 2024-04-26T17:58:45.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.416+0000 7f08778ad700 1 Processor -- start 2024-04-26T17:58:45.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.416+0000 7f08778ad700 1 -- start start 2024-04-26T17:58:45.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.416+0000 7f08778ad700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08701025a0 0x7f087019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.416+0000 7f08778ad700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870102f50 0x7f087019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.416+0000 7f08778ad700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0870107680 0x7f0870199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.416+0000 7f08778ad700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0870112520 con 0x7f0870102f50 2024-04-26T17:58:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.416+0000 7f08778ad700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f08701123a0 con 0x7f0870107680 2024-04-26T17:58:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.416+0000 7f08778ad700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f08701126a0 con 0x7f08701025a0 2024-04-26T17:58:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f0874e48700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870102f50 0x7f087019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f0874e48700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870102f50 0x7f087019fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57552/0 (socket says 172.21.15.31:57552) 2024-04-26T17:58:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f0874e48700 1 -- 172.21.15.31:0/1226147601 learned_addr learned my addr 172.21.15.31:0/1226147601 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f0875649700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08701025a0 0x7f087019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f0875e4a700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0870107680 0x7f0870199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f0874e48700 1 -- 172.21.15.31:0/1226147601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08701025a0 msgr2=0x7f087019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f0874e48700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08701025a0 0x7f087019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f0874e48700 1 -- 172.21.15.31:0/1226147601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0870107680 msgr2=0x7f0870199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f0874e48700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0870107680 0x7f0870199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f0874e48700 1 -- 172.21.15.31:0/1226147601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f087019a0c0 con 0x7f0870102f50 2024-04-26T17:58:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f0875e4a700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0870107680 0x7f0870199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:58:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f0874e48700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870102f50 0x7f087019fc90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f086000d950 tx=0x7f086000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f08667fc700 1 -- 172.21.15.31:0/1226147601 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0860016070 con 0x7f0870102f50 2024-04-26T17:58:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f08667fc700 1 -- 172.21.15.31:0/1226147601 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0860004830 con 0x7f0870102f50 2024-04-26T17:58:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f08778ad700 1 -- 172.21.15.31:0/1226147601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f087019a3b0 con 0x7f0870102f50 2024-04-26T17:58:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f08667fc700 1 -- 172.21.15.31:0/1226147601 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0860005440 con 0x7f0870102f50 2024-04-26T17:58:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.417+0000 7f08778ad700 1 -- 172.21.15.31:0/1226147601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08701a6620 con 0x7f0870102f50 2024-04-26T17:58:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.419+0000 7f08667fc700 1 -- 172.21.15.31:0/1226147601 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0860004220 con 0x7f0870102f50 2024-04-26T17:58:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.420+0000 7f08667fc700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f085c0a3f00 0x7f085c0a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.420+0000 7f0875649700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f085c0a3f00 0x7f085c0a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.420+0000 7f08667fc700 1 -- 172.21.15.31:0/1226147601 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f08600d0420 con 0x7f0870102f50 2024-04-26T17:58:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.421+0000 7f0875649700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f085c0a3f00 0x7f085c0a63c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f086c005950 tx=0x7f086c00a300 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.421+0000 7f088213fb80 1 Processor -- start 2024-04-26T17:58:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.421+0000 7f088213fb80 1 -- start start 2024-04-26T17:58:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.421+0000 7f088213fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c421a0 0x557013c42580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.421+0000 7f088213fb80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c42ac0 0x557013c4d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.421+0000 7f088213fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c4d970 0x557013c4fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.421+0000 7f088213fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557013b95840 con 0x557013c421a0 2024-04-26T17:58:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.421+0000 7f088213fb80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x557013b346e0 con 0x557013c42ac0 2024-04-26T17:58:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.421+0000 7f088213fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557013b91800 con 0x557013c4d970 2024-04-26T17:58:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.421+0000 7f0875649700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c421a0 0x557013c42580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.421+0000 7f0875649700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c421a0 0x557013c42580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57560/0 (socket says 172.21.15.31:57560) 2024-04-26T17:58:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.421+0000 7f0875649700 1 -- 172.21.15.31:0/2167138387 learned_addr learned my addr 172.21.15.31:0/2167138387 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.422+0000 7f0874e48700 1 --2- 172.21.15.31:0/2167138387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c42ac0 0x557013c4d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.422+0000 7f0875649700 1 -- 172.21.15.31:0/2167138387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c4d970 msgr2=0x557013c4fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.422+0000 7f0875649700 1 --2- 172.21.15.31:0/2167138387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c4d970 0x557013c4fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.422+0000 7f0875649700 1 -- 172.21.15.31:0/2167138387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c42ac0 msgr2=0x557013c4d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.422+0000 7f0875649700 1 --2- 172.21.15.31:0/2167138387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c42ac0 0x557013c4d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.422+0000 7f0875649700 1 -- 172.21.15.31:0/2167138387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557013ae29d0 con 0x557013c421a0 2024-04-26T17:58:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.422+0000 7f0874e48700 1 --2- 172.21.15.31:0/2167138387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c42ac0 0x557013c4d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:58:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.422+0000 7f0875649700 1 --2- 172.21.15.31:0/2167138387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c421a0 0x557013c42580 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f086c00fda0 tx=0x7f086c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197f792a99da2197 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.422+0000 7f085b7fe700 1 -- 172.21.15.31:0/2167138387 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c016070 con 0x557013c421a0 2024-04-26T17:58:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.422+0000 7f085b7fe700 1 -- 172.21.15.31:0/2167138387 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f086c00fe80 con 0x557013c421a0 2024-04-26T17:58:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.422+0000 7f085b7fe700 1 -- 172.21.15.31:0/2167138387 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c01b420 con 0x557013c421a0 2024-04-26T17:58:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.423+0000 7f088213fb80 1 -- 172.21.15.31:0/2167138387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c421a0 msgr2=0x557013c42580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.423+0000 7f088213fb80 1 --2- 172.21.15.31:0/2167138387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c421a0 0x557013c42580 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f086c00fda0 tx=0x7f086c012000 comp rx=0 tx=0).stop 2024-04-26T17:58:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.423+0000 7f088213fb80 1 -- 172.21.15.31:0/2167138387 shutdown_connections 2024-04-26T17:58:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.423+0000 7f088213fb80 1 --2- 172.21.15.31:0/2167138387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c4d970 0x557013c4fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.423+0000 7f088213fb80 1 --2- 172.21.15.31:0/2167138387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c421a0 0x557013c42580 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.423+0000 7f088213fb80 1 --2- 172.21.15.31:0/2167138387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c42ac0 0x557013c4d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.423+0000 7f088213fb80 1 -- 172.21.15.31:0/2167138387 >> 172.21.15.31:0/2167138387 conn(0x557013b6fe40 msgr2=0x557013c3e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.423+0000 7f088213fb80 1 -- 172.21.15.31:0/2167138387 shutdown_connections 2024-04-26T17:58:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.423+0000 7f088213fb80 1 -- 172.21.15.31:0/2167138387 wait complete. 2024-04-26T17:58:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.423+0000 7f088213fb80 1 -- start start 2024-04-26T17:58:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.423+0000 7f088213fb80 1 Processor -- start 2024-04-26T17:58:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f088213fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c421a0 0x557013c4d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f088213fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c42ac0 0x557013c4a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f088213fb80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c4d970 0x557013c4a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f088213fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557013c50540 con 0x557013c421a0 2024-04-26T17:58:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f088213fb80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x557013afba50 con 0x557013c4d970 2024-04-26T17:58:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f088213fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557013c506c0 con 0x557013c42ac0 2024-04-26T17:58:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f088213fb80 10 client.0 _ll_register_callbacks cb 0x557013b29680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f0875649700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c421a0 0x557013c4d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f0875649700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c421a0 0x557013c4d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57570/0 (socket says 172.21.15.31:57570) 2024-04-26T17:58:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f0875649700 1 -- 172.21.15.31:0/1897395526 learned_addr learned my addr 172.21.15.31:0/1897395526 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f0875e4a700 1 --2- 172.21.15.31:0/1897395526 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c4d970 0x557013c4a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f0874e48700 1 --2- 172.21.15.31:0/1897395526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c42ac0 0x557013c4a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f0875649700 1 -- 172.21.15.31:0/1897395526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c42ac0 msgr2=0x557013c4a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f0875649700 1 --2- 172.21.15.31:0/1897395526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c42ac0 0x557013c4a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f0875649700 1 -- 172.21.15.31:0/1897395526 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c4d970 msgr2=0x557013c4a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f0875649700 1 --2- 172.21.15.31:0/1897395526 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c4d970 0x557013c4a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f0875649700 1 -- 172.21.15.31:0/1897395526 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557013c4b000 con 0x557013c421a0 2024-04-26T17:58:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.424+0000 7f0874e48700 1 --2- 172.21.15.31:0/1897395526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c42ac0 0x557013c4a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:58:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f0875649700 1 --2- 172.21.15.31:0/1897395526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c421a0 0x557013c4d100 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f086c00fd70 tx=0x7f086c00f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae830119fb715fb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f0858ff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:58:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f0858ff9700 1 -- 172.21.15.31:0/1897395526 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c010cd0 con 0x557013c421a0 2024-04-26T17:58:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f0858ff9700 1 -- 172.21.15.31:0/1897395526 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f086c01c420 con 0x557013c421a0 2024-04-26T17:58:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f088213fb80 1 -- 172.21.15.31:0/1897395526 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557013ba3220 con 0x557013c421a0 2024-04-26T17:58:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f0858ff9700 1 -- 172.21.15.31:0/1897395526 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f086c004870 con 0x557013c421a0 2024-04-26T17:58:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f088213fb80 10 client.15594 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f088213fb80 1 -- 172.21.15.31:0/1897395526 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557013c5b000 con 0x557013c421a0 2024-04-26T17:58:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f0858ff9700 1 -- 172.21.15.31:0/1897395526 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x557013ba3220 con 0x557013c421a0 2024-04-26T17:58:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f0858ff9700 1 -- 172.21.15.31:0/1897395526 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f086c005580 con 0x557013c421a0 2024-04-26T17:58:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f088213fb80 10 client.15594 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f088213fb80 10 client.15594 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T17:58:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f088213fb80 20 client.15594 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f088213fb80 1 --2- 172.21.15.31:0/1897395526 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x557013c5d5e0 0x557013c5f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f088213fb80 4 client.15594 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.425+0000 7f088213fb80 1 -- 172.21.15.31:0/1897395526 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557013b9d2c0 con 0x557013c5d5e0 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.426+0000 7f0874e48700 1 --2- 172.21.15.31:0/1897395526 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x557013c5d5e0 0x557013c5f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.426+0000 7f0874e48700 1 --2- 172.21.15.31:0/1897395526 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x557013c5d5e0 0x557013c5f9d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67eacabd2f9f0478 server_cookie=e839d842ba664b10 in_seq=0 out_seq=0 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.426+0000 7f0858ff9700 10 client.15594 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f0858ff9700 1 -- 172.21.15.31:0/1897395526 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557013b9d2c0 con 0x557013c5d5e0 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f0858ff9700 10 client.15594 handle_command_reply: tid=0 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f088213fb80 1 client.15594 shutdown 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f088213fb80 1 -- 172.21.15.31:0/1897395526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c421a0 msgr2=0x557013c4d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f088213fb80 1 --2- 172.21.15.31:0/1897395526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c421a0 0x557013c4d100 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f086c00fd70 tx=0x7f086c00f9e0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f088213fb80 1 -- 172.21.15.31:0/1897395526 shutdown_connections 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f088213fb80 1 --2- 172.21.15.31:0/1897395526 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x557013c5d5e0 0x557013c5f9d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f088213fb80 1 --2- 172.21.15.31:0/1897395526 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c4d970 0x557013c4a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f088213fb80 1 --2- 172.21.15.31:0/1897395526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c421a0 0x557013c4d100 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f088213fb80 1 --2- 172.21.15.31:0/1897395526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c42ac0 0x557013c4a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f088213fb80 1 -- 172.21.15.31:0/1897395526 >> 172.21.15.31:0/1897395526 conn(0x557013b6fe40 msgr2=0x557013c3e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f0858ff9700 0 client.15594 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f088213fb80 1 -- 172.21.15.31:0/1897395526 shutdown_connections 2024-04-26T17:58:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f088213fb80 1 -- 172.21.15.31:0/1897395526 wait complete. 2024-04-26T17:58:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.427+0000 7f088213fb80 20 client.15594 trim_cache size 0 max 16384 2024-04-26T17:58:45.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.447+0000 7f088213fb80 1 Processor -- start 2024-04-26T17:58:45.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.447+0000 7f088213fb80 1 -- start start 2024-04-26T17:58:45.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.447+0000 7f088213fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c421a0 0x557013c3e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.447+0000 7f088213fb80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c42ac0 0x557013c45930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.447+0000 7f088213fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c4d970 0x557013c59470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.447+0000 7f088213fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f08701b0510 con 0x557013c4d970 2024-04-26T17:58:45.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.447+0000 7f088213fb80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f08701add40 con 0x557013c42ac0 2024-04-26T17:58:45.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.447+0000 7f088213fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f08701abb90 con 0x557013c421a0 2024-04-26T17:58:45.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.447+0000 7f0875e4a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c4d970 0x557013c59470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.447+0000 7f0875e4a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c4d970 0x557013c59470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57586/0 (socket says 172.21.15.31:57586) 2024-04-26T17:58:45.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.447+0000 7f0875e4a700 1 -- 172.21.15.31:0/474620975 learned_addr learned my addr 172.21.15.31:0/474620975 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:45.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.448+0000 7f0875649700 1 --2- 172.21.15.31:0/474620975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c421a0 0x557013c3e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.448+0000 7f0874e48700 1 --2- 172.21.15.31:0/474620975 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c42ac0 0x557013c45930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.448+0000 7f0875e4a700 1 -- 172.21.15.31:0/474620975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c421a0 msgr2=0x557013c3e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:45.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.448+0000 7f0875e4a700 1 --2- 172.21.15.31:0/474620975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c421a0 0x557013c3e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.448+0000 7f0875e4a700 1 -- 172.21.15.31:0/474620975 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c42ac0 msgr2=0x557013c45930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:45.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.448+0000 7f0875e4a700 1 --2- 172.21.15.31:0/474620975 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c42ac0 0x557013c45930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.448+0000 7f0875e4a700 1 -- 172.21.15.31:0/474620975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557013c4d420 con 0x557013c4d970 2024-04-26T17:58:45.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.448+0000 7f0875649700 1 --2- 172.21.15.31:0/474620975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c421a0 0x557013c3e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:58:45.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.448+0000 7f0875e4a700 1 --2- 172.21.15.31:0/474620975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c4d970 0x557013c59470 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f086800b0d0 tx=0x7f0868009180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde0e4f4182e6e33 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:45.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.448+0000 7f085affd700 1 -- 172.21.15.31:0/474620975 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0868004a50 con 0x557013c4d970 2024-04-26T17:58:45.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.448+0000 7f085affd700 1 -- 172.21.15.31:0/474620975 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0868021070 con 0x557013c4d970 2024-04-26T17:58:45.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.448+0000 7f085affd700 1 -- 172.21.15.31:0/474620975 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08680266d0 con 0x557013c4d970 2024-04-26T17:58:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 -- 172.21.15.31:0/474620975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c4d970 msgr2=0x557013c59470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 --2- 172.21.15.31:0/474620975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c4d970 0x557013c59470 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f086800b0d0 tx=0x7f0868009180 comp rx=0 tx=0).stop 2024-04-26T17:58:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 -- 172.21.15.31:0/474620975 shutdown_connections 2024-04-26T17:58:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 --2- 172.21.15.31:0/474620975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c4d970 0x557013c59470 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 --2- 172.21.15.31:0/474620975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c421a0 0x557013c3e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 --2- 172.21.15.31:0/474620975 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c42ac0 0x557013c45930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 -- 172.21.15.31:0/474620975 >> 172.21.15.31:0/474620975 conn(0x557013b6fe40 msgr2=0x557013c45220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 -- 172.21.15.31:0/474620975 shutdown_connections 2024-04-26T17:58:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 -- 172.21.15.31:0/474620975 wait complete. 2024-04-26T17:58:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 -- start start 2024-04-26T17:58:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 Processor -- start 2024-04-26T17:58:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c421a0 0x557013c55cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c42ac0 0x557013c51040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c4d970 0x557013c51580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f08701ac200 con 0x557013c42ac0 2024-04-26T17:58:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f08701ad670 con 0x557013c4d970 2024-04-26T17:58:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.449+0000 7f088213fb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f08701ac8d0 con 0x557013c421a0 2024-04-26T17:58:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f088213fb80 10 client.0 _ll_register_callbacks cb 0x557013b29680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0874e48700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c42ac0 0x557013c51040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0874e48700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c42ac0 0x557013c51040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57594/0 (socket says 172.21.15.31:57594) 2024-04-26T17:58:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0874e48700 1 -- 172.21.15.31:0/2580768290 learned_addr learned my addr 172.21.15.31:0/2580768290 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0875e4a700 1 --2- 172.21.15.31:0/2580768290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c4d970 0x557013c51580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0875649700 1 --2- 172.21.15.31:0/2580768290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c421a0 0x557013c55cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0874e48700 1 -- 172.21.15.31:0/2580768290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c421a0 msgr2=0x557013c55cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:45.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0874e48700 1 --2- 172.21.15.31:0/2580768290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c421a0 0x557013c55cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0874e48700 1 -- 172.21.15.31:0/2580768290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c4d970 msgr2=0x557013c51580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0874e48700 1 --2- 172.21.15.31:0/2580768290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c4d970 0x557013c51580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0874e48700 1 -- 172.21.15.31:0/2580768290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557013c51d70 con 0x557013c42ac0 2024-04-26T17:58:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0874e48700 1 --2- 172.21.15.31:0/2580768290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c42ac0 0x557013c51040 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f08600d0390 tx=0x7f086000d8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295365949f17081d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0858ff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:58:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0858ff9700 1 -- 172.21.15.31:0/2580768290 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08600b0060 con 0x557013c42ac0 2024-04-26T17:58:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0858ff9700 1 -- 172.21.15.31:0/2580768290 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f0860005640 con 0x557013c42ac0 2024-04-26T17:58:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f088213fb80 1 -- 172.21.15.31:0/2580768290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0868021070 con 0x557013c42ac0 2024-04-26T17:58:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.450+0000 7f0858ff9700 1 -- 172.21.15.31:0/2580768290 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f086009c3b0 con 0x557013c42ac0 2024-04-26T17:58:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.451+0000 7f088213fb80 10 client.15600 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.451+0000 7f088213fb80 1 -- 172.21.15.31:0/2580768290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557013cc3960 con 0x557013c42ac0 2024-04-26T17:58:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.451+0000 7f0858ff9700 1 -- 172.21.15.31:0/2580768290 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0868021070 con 0x557013c42ac0 2024-04-26T17:58:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.451+0000 7f0858ff9700 1 -- 172.21.15.31:0/2580768290 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f08600a7070 con 0x557013c42ac0 2024-04-26T17:58:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.451+0000 7f088213fb80 10 client.15600 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:45.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.451+0000 7f088213fb80 10 client.15600 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T17:58:45.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.451+0000 7f088213fb80 20 client.15600 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:45.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.451+0000 7f088213fb80 1 --2- 172.21.15.31:0/2580768290 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x557013cc3fa0 0x557013cc6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:45.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.451+0000 7f088213fb80 4 client.15600 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "502c2818", "release": true, "await": true}] 2024-04-26T17:58:45.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.451+0000 7f088213fb80 1 -- 172.21.15.31:0/2580768290 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "quiesce db", "set_id": "502c2818", "release": true, "await": true}) -- 0x557013a15350 con 0x557013cc3fa0 2024-04-26T17:58:45.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.451+0000 7f0875649700 1 --2- 172.21.15.31:0/2580768290 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x557013cc3fa0 0x557013cc6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:45.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.452+0000 7f0875649700 1 --2- 172.21.15.31:0/2580768290 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x557013cc3fa0 0x557013cc6390 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a366c17ed0cea12 server_cookie=81495385e4f1c53f in_seq=0 out_seq=0 2024-04-26T17:58:45.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.452+0000 7f0858ff9700 10 client.15600 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:45.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.488+0000 7f0858ff9700 1 -- 172.21.15.31:0/2580768290 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x557013a15350 con 0x557013cc3fa0 2024-04-26T17:58:45.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.488+0000 7f0858ff9700 10 client.15600 handle_command_reply: tid=0 2024-04-26T17:58:45.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.488+0000 7f088213fb80 1 client.15600 shutdown 2024-04-26T17:58:45.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.488+0000 7f088213fb80 1 -- 172.21.15.31:0/2580768290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c42ac0 msgr2=0x557013c51040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:45.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.488+0000 7f088213fb80 1 --2- 172.21.15.31:0/2580768290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c42ac0 0x557013c51040 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f08600d0390 tx=0x7f086000d8d0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.488+0000 7f088213fb80 1 -- 172.21.15.31:0/2580768290 shutdown_connections 2024-04-26T17:58:45.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.488+0000 7f088213fb80 1 --2- 172.21.15.31:0/2580768290 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x557013cc3fa0 0x557013cc6390 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.488+0000 7f088213fb80 1 --2- 172.21.15.31:0/2580768290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557013c4d970 0x557013c51580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.488+0000 7f088213fb80 1 --2- 172.21.15.31:0/2580768290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557013c421a0 0x557013c55cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.488+0000 7f0858ff9700 0 client.15600 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:45.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.488+0000 7f088213fb80 1 --2- 172.21.15.31:0/2580768290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013c42ac0 0x557013c51040 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.488+0000 7f088213fb80 1 -- 172.21.15.31:0/2580768290 >> 172.21.15.31:0/2580768290 conn(0x557013b6fe40 msgr2=0x557013c440a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:45.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.489+0000 7f088213fb80 1 -- 172.21.15.31:0/2580768290 shutdown_connections 2024-04-26T17:58:45.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.489+0000 7f088213fb80 1 -- 172.21.15.31:0/2580768290 wait complete. 2024-04-26T17:58:45.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.489+0000 7f088213fb80 20 client.15600 trim_cache size 0 max 16384 2024-04-26T17:58:45.491 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T17:58:45.492 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 29, 2024-04-26T17:58:45.492 INFO:teuthology.orchestra.run.smithi031.stdout: "leader": 24473, 2024-04-26T17:58:45.492 INFO:teuthology.orchestra.run.smithi031.stdout: "set_version": 8, 2024-04-26T17:58:45.492 INFO:teuthology.orchestra.run.smithi031.stdout: "sets": { 2024-04-26T17:58:45.492 INFO:teuthology.orchestra.run.smithi031.stdout: "502c2818": { 2024-04-26T17:58:45.492 INFO:teuthology.orchestra.run.smithi031.stdout: "version": 8, 2024-04-26T17:58:45.492 INFO:teuthology.orchestra.run.smithi031.stdout: "age_ref": 0.0, 2024-04-26T17:58:45.492 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T17:58:45.492 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RELEASED", 2024-04-26T17:58:45.492 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T17:58:45.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T17:58:45.492 INFO:teuthology.orchestra.run.smithi031.stdout: "timeout": 60.0, 2024-04-26T17:58:45.493 INFO:teuthology.orchestra.run.smithi031.stdout: "expiration": 90.0, 2024-04-26T17:58:45.493 INFO:teuthology.orchestra.run.smithi031.stdout: "members": { 2024-04-26T17:58:45.493 INFO:teuthology.orchestra.run.smithi031.stdout: "file:/": { 2024-04-26T17:58:45.493 INFO:teuthology.orchestra.run.smithi031.stdout: "excluded": false, 2024-04-26T17:58:45.493 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T17:58:45.493 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RELEASED", 2024-04-26T17:58:45.493 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T17:58:45.493 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:45.493 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:45.493 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:45.493 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:45.493 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T17:58:45.493 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T17:58:45.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.490+0000 7f08778ad700 1 -- 172.21.15.31:0/1226147601 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f085c0a3f00 msgr2=0x7f085c0a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:45.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.490+0000 7f08778ad700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f085c0a3f00 0x7f085c0a63c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f086c005950 tx=0x7f086c00a300 comp rx=0 tx=0).stop 2024-04-26T17:58:45.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.490+0000 7f08778ad700 1 -- 172.21.15.31:0/1226147601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870102f50 msgr2=0x7f087019fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:45.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.490+0000 7f08778ad700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870102f50 0x7f087019fc90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f086000d950 tx=0x7f086000dc60 comp rx=0 tx=0).stop 2024-04-26T17:58:45.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.490+0000 7f08778ad700 1 -- 172.21.15.31:0/1226147601 shutdown_connections 2024-04-26T17:58:45.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.490+0000 7f08778ad700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f085c0a3f00 0x7f085c0a63c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.491+0000 7f08778ad700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0870107680 0x7f0870199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.491+0000 7f08778ad700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08701025a0 0x7f087019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.491+0000 7f08778ad700 1 --2- 172.21.15.31:0/1226147601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0870102f50 0x7f087019fc90 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:45.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.491+0000 7f08778ad700 1 -- 172.21.15.31:0/1226147601 >> 172.21.15.31:0/1226147601 conn(0x7f08700793c0 msgr2=0x7f0870106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:45.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.491+0000 7f08778ad700 1 -- 172.21.15.31:0/1226147601 shutdown_connections 2024-04-26T17:58:45.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T17:58:45.491+0000 7f08778ad700 1 -- 172.21.15.31:0/1226147601 wait complete. 2024-04-26T17:58:45.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:45 smithi186 ceph-mon[32797]: pgmap v70: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 31 KiB/s wr, 11 op/s 2024-04-26T17:58:46.378 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '502c2818', seconds elapsed: 43.4 2024-04-26T17:58:47.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:47 smithi031 ceph-mon[29164]: pgmap v71: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 212 KiB/s wr, 30 op/s 2024-04-26T17:58:47.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:47 smithi097 ceph-mon[32419]: pgmap v71: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 212 KiB/s wr, 30 op/s 2024-04-26T17:58:47.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:47 smithi186 ceph-mon[32797]: pgmap v71: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 212 KiB/s wr, 30 op/s 2024-04-26T17:58:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 85.0 s, 1902.7 tps, lat 408.018 ms stddev 3694.698 2024-04-26T17:58:49.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:49 smithi031 ceph-mon[29164]: pgmap v72: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 243 KiB/s wr, 34 op/s 2024-04-26T17:58:49.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:49 smithi097 ceph-mon[32419]: pgmap v72: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 243 KiB/s wr, 34 op/s 2024-04-26T17:58:49.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:49 smithi186 ceph-mon[32797]: pgmap v72: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 243 KiB/s wr, 34 op/s 2024-04-26T17:58:51.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:51 smithi031 ceph-mon[29164]: pgmap v73: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 237 KiB/s wr, 31 op/s 2024-04-26T17:58:51.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:51 smithi097 ceph-mon[32419]: pgmap v73: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 237 KiB/s wr, 31 op/s 2024-04-26T17:58:51.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:51 smithi186 ceph-mon[32797]: pgmap v73: 97 pgs: 97 active+clean; 1.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 237 KiB/s wr, 31 op/s 2024-04-26T17:58:53.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 90.0 s, 3642.4 tps, lat 27.436 ms stddev 28.219 2024-04-26T17:58:53.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:53 smithi097 ceph-mon[32419]: pgmap v74: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.5 MiB/s wr, 352 op/s 2024-04-26T17:58:53.520 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:53 smithi186 ceph-mon[32797]: pgmap v74: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.5 MiB/s wr, 352 op/s 2024-04-26T17:58:53.537 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:53 smithi031 ceph-mon[29164]: pgmap v74: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.5 MiB/s wr, 352 op/s 2024-04-26T17:58:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:55 smithi097 ceph-mon[32419]: pgmap v75: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 3.5 MiB/s wr, 347 op/s 2024-04-26T17:58:55.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:55 smithi186 ceph-mon[32797]: pgmap v75: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 3.5 MiB/s wr, 347 op/s 2024-04-26T17:58:55.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:55 smithi031 ceph-mon[29164]: pgmap v75: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 3.5 MiB/s wr, 347 op/s 2024-04-26T17:58:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.974+0000 7f864c1a4640 1 -- 172.21.15.31:0/3620133206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644101860 msgr2=0x7f8644101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.974+0000 7f864c1a4640 1 --2- 172.21.15.31:0/3620133206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644101860 0x7f8644101ce0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8640009930 tx=0x7f864002a420 comp rx=0 tx=0).stop 2024-04-26T17:58:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.974+0000 7f864c1a4640 1 -- 172.21.15.31:0/3620133206 shutdown_connections 2024-04-26T17:58:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.974+0000 7f864c1a4640 1 --2- 172.21.15.31:0/3620133206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8644102220 0x7f864410e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.974+0000 7f864c1a4640 1 --2- 172.21.15.31:0/3620133206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644101860 0x7f8644101ce0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.974+0000 7f864c1a4640 1 --2- 172.21.15.31:0/3620133206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8644107820 0x7f8644107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.974+0000 7f864c1a4640 1 -- 172.21.15.31:0/3620133206 >> 172.21.15.31:0/3620133206 conn(0x7f86440789b0 msgr2=0x7f8644078dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.974+0000 7f864c1a4640 1 -- 172.21.15.31:0/3620133206 shutdown_connections 2024-04-26T17:58:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.974+0000 7f864c1a4640 1 -- 172.21.15.31:0/3620133206 wait complete. 2024-04-26T17:58:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.975+0000 7f864c1a4640 1 Processor -- start 2024-04-26T17:58:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.976+0000 7f864c1a4640 1 -- start start 2024-04-26T17:58:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.977+0000 7f864c1a4640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8644101860 0x7f8644074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.977+0000 7f864c1a4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644102220 0x7f864406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.977+0000 7f864c1a4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8644107820 0x7f864406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.977+0000 7f864c1a4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8644111ae0 con 0x7f8644102220 2024-04-26T17:58:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.977+0000 7f864c1a4640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f8644111960 con 0x7f8644101860 2024-04-26T17:58:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.977+0000 7f864c1a4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8644111c60 con 0x7f8644107820 2024-04-26T17:58:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.977+0000 7f864973f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644102220 0x7f864406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.977+0000 7f8649f40640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8644101860 0x7f8644074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.977+0000 7f8649f40640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8644101860 0x7f8644074150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:54296/0 (socket says 172.21.15.31:54296) 2024-04-26T17:58:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.977+0000 7f864973f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644102220 0x7f864406d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60512/0 (socket says 172.21.15.31:60512) 2024-04-26T17:58:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.977+0000 7f8649f40640 1 -- 172.21.15.31:0/947694700 learned_addr learned my addr 172.21.15.31:0/947694700 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.977+0000 7f864a741640 1 --2- 172.21.15.31:0/947694700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8644107820 0x7f864406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.978+0000 7f864973f640 1 -- 172.21.15.31:0/947694700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8644107820 msgr2=0x7f864406d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.978+0000 7f864973f640 1 --2- 172.21.15.31:0/947694700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8644107820 0x7f864406d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.978+0000 7f864973f640 1 -- 172.21.15.31:0/947694700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8644101860 msgr2=0x7f8644074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.978+0000 7f864973f640 1 --2- 172.21.15.31:0/947694700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8644101860 0x7f8644074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.978+0000 7f864973f640 1 -- 172.21.15.31:0/947694700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864406dd50 con 0x7f8644102220 2024-04-26T17:58:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.978+0000 7f8649f40640 1 --2- 172.21.15.31:0/947694700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8644101860 0x7f8644074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:58:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.978+0000 7f864973f640 1 --2- 172.21.15.31:0/947694700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644102220 0x7f864406d220 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8640031440 tx=0x7f8640004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.979+0000 7f863affd640 1 -- 172.21.15.31:0/947694700 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864003f070 con 0x7f8644102220 2024-04-26T17:58:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.979+0000 7f863affd640 1 -- 172.21.15.31:0/947694700 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8640003aa0 con 0x7f8644102220 2024-04-26T17:58:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.979+0000 7f863affd640 1 -- 172.21.15.31:0/947694700 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640035070 con 0x7f8644102220 2024-04-26T17:58:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.979+0000 7f864c1a4640 1 -- 172.21.15.31:0/947694700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f864406dfe0 con 0x7f8644102220 2024-04-26T17:58:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.979+0000 7f864c1a4640 1 -- 172.21.15.31:0/947694700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8644071eb0 con 0x7f8644102220 2024-04-26T17:58:56.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.983+0000 7f864ef96740 1 Processor -- start 2024-04-26T17:58:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.984+0000 7f864ef96740 1 -- start start 2024-04-26T17:58:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.984+0000 7f864ef96740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116ffa0 0x5588c1170380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.984+0000 7f864ef96740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ba10 0x5588c116be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.984+0000 7f864ef96740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116c3d0 0x5588c1177ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.984+0000 7f864ef96740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5588c10c8460 con 0x5588c116ba10 2024-04-26T17:58:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.984+0000 7f864ef96740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5588c1082590 con 0x5588c116ffa0 2024-04-26T17:58:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.984+0000 7f864ef96740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5588c10c4420 con 0x5588c116c3d0 2024-04-26T17:58:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.984+0000 7f8649f40640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ba10 0x5588c116be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.984+0000 7f863affd640 1 -- 172.21.15.31:0/947694700 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f864003b460 con 0x7f8644102220 2024-04-26T17:58:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.984+0000 7f8649f40640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ba10 0x5588c116be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60524/0 (socket says 172.21.15.31:60524) 2024-04-26T17:58:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.984+0000 7f8649f40640 1 -- 172.21.15.31:0/4269611890 learned_addr learned my addr 172.21.15.31:0/4269611890 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.984+0000 7f864a741640 1 --2- 172.21.15.31:0/4269611890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116ffa0 0x5588c1170380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.985+0000 7f864973f640 1 --2- 172.21.15.31:0/4269611890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116c3d0 0x5588c1177ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.985+0000 7f8649f40640 1 -- 172.21.15.31:0/4269611890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116c3d0 msgr2=0x5588c1177ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.985+0000 7f8649f40640 1 --2- 172.21.15.31:0/4269611890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116c3d0 0x5588c1177ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.985+0000 7f8649f40640 1 -- 172.21.15.31:0/4269611890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116ffa0 msgr2=0x5588c1170380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.985+0000 7f8649f40640 1 --2- 172.21.15.31:0/4269611890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116ffa0 0x5588c1170380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.985+0000 7f8649f40640 1 -- 172.21.15.31:0/4269611890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588c108f0d0 con 0x5588c116ba10 2024-04-26T17:58:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.985+0000 7f8649f40640 1 --2- 172.21.15.31:0/4269611890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ba10 0x5588c116be90 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f863400e480 tx=0x7f8634005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a58e172d910c9fc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.985+0000 7f863affd640 1 --2- 172.21.15.31:0/947694700 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f86300a3f20 0x7f86300a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.985+0000 7f862ffff640 1 -- 172.21.15.31:0/4269611890 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863401b020 con 0x5588c116ba10 2024-04-26T17:58:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.985+0000 7f862ffff640 1 -- 172.21.15.31:0/4269611890 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8634017ab0 con 0x5588c116ba10 2024-04-26T17:58:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.985+0000 7f862ffff640 1 -- 172.21.15.31:0/4269611890 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8634016ad0 con 0x5588c116ba10 2024-04-26T17:58:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.985+0000 7f863affd640 1 -- 172.21.15.31:0/947694700 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f86400f0af0 con 0x7f8644102220 2024-04-26T17:58:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.985+0000 7f864a741640 1 --2- 172.21.15.31:0/947694700 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f86300a3f20 0x7f86300a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.986+0000 7f864ef96740 1 -- 172.21.15.31:0/4269611890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ba10 msgr2=0x5588c116be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.986+0000 7f864ef96740 1 --2- 172.21.15.31:0/4269611890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ba10 0x5588c116be90 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f863400e480 tx=0x7f8634005f90 comp rx=0 tx=0).stop 2024-04-26T17:58:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.987+0000 7f864ef96740 1 -- 172.21.15.31:0/4269611890 shutdown_connections 2024-04-26T17:58:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.987+0000 7f864ef96740 1 --2- 172.21.15.31:0/4269611890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116c3d0 0x5588c1177ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.987+0000 7f864ef96740 1 --2- 172.21.15.31:0/4269611890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ba10 0x5588c116be90 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.987+0000 7f864ef96740 1 --2- 172.21.15.31:0/4269611890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116ffa0 0x5588c1170380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.987+0000 7f864ef96740 1 -- 172.21.15.31:0/4269611890 >> 172.21.15.31:0/4269611890 conn(0x5588c115dec0 msgr2=0x5588c115e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.987+0000 7f864a741640 1 --2- 172.21.15.31:0/947694700 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f86300a3f20 0x7f86300a63e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f863c005fd0 tx=0x7f863c009500 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.988+0000 7f864ef96740 1 -- 172.21.15.31:0/4269611890 shutdown_connections 2024-04-26T17:58:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.988+0000 7f864ef96740 1 -- 172.21.15.31:0/4269611890 wait complete. 2024-04-26T17:58:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.988+0000 7f864ef96740 1 -- start start 2024-04-26T17:58:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.988+0000 7f864ef96740 1 Processor -- start 2024-04-26T17:58:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.989+0000 7f864ef96740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116ba10 0x5588c1175320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.989+0000 7f864ef96740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116c3d0 0x5588c1175860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.989+0000 7f864ef96740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ffa0 0x5588c1175da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.989+0000 7f864ef96740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5588c11784f0 con 0x5588c116ffa0 2024-04-26T17:58:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.989+0000 7f864ef96740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5588c104fb60 con 0x5588c116c3d0 2024-04-26T17:58:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.989+0000 7f864ef96740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5588c1178670 con 0x5588c116ba10 2024-04-26T17:58:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.989+0000 7f864ef96740 10 client.0 _ll_register_callbacks cb 0x5588c10742b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.991+0000 7f864a741640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ffa0 0x5588c1175da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.991+0000 7f864a741640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ffa0 0x5588c1175da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60540/0 (socket says 172.21.15.31:60540) 2024-04-26T17:58:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.991+0000 7f864a741640 1 -- 172.21.15.31:0/3177279853 learned_addr learned my addr 172.21.15.31:0/3177279853 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.991+0000 7f864973f640 1 --2- 172.21.15.31:0/3177279853 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116c3d0 0x5588c1175860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.991+0000 7f8649f40640 1 --2- 172.21.15.31:0/3177279853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116ba10 0x5588c1175320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.993+0000 7f864973f640 1 -- 172.21.15.31:0/3177279853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116ba10 msgr2=0x5588c1175320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.993+0000 7f864973f640 1 --2- 172.21.15.31:0/3177279853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116ba10 0x5588c1175320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.993+0000 7f864973f640 1 -- 172.21.15.31:0/3177279853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ffa0 msgr2=0x5588c1175da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.993+0000 7f864973f640 1 --2- 172.21.15.31:0/3177279853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ffa0 0x5588c1175da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.993+0000 7f864973f640 1 -- 172.21.15.31:0/3177279853 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588c0dda210 con 0x5588c116c3d0 2024-04-26T17:58:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.994+0000 7f864973f640 1 --2- 172.21.15.31:0/3177279853 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116c3d0 0x5588c1175860 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f86400ee290 tx=0x7f86400b8e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fab3f69f0f3c7a41 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.994+0000 7f862d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T17:58:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.994+0000 7f862d7fa640 1 -- 172.21.15.31:0/3177279853 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640048070 con 0x5588c116c3d0 2024-04-26T17:58:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.994+0000 7f864ef96740 1 -- 172.21.15.31:0/3177279853 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8634017ab0 con 0x5588c116c3d0 2024-04-26T17:58:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.994+0000 7f862d7fa640 1 -- 172.21.15.31:0/3177279853 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f8640003c50 con 0x5588c116c3d0 2024-04-26T17:58:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.995+0000 7f862d7fa640 1 -- 172.21.15.31:0/3177279853 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f86400bc4f0 con 0x5588c116c3d0 2024-04-26T17:58:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.995+0000 7f864ef96740 10 client.25310 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.995+0000 7f864ef96740 1 -- 172.21.15.31:0/3177279853 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5588c0ea6620 con 0x5588c116c3d0 2024-04-26T17:58:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.997+0000 7f862d7fa640 1 -- 172.21.15.31:0/3177279853 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8634017ab0 con 0x5588c116c3d0 2024-04-26T17:58:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.997+0000 7f862d7fa640 1 -- 172.21.15.31:0/3177279853 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f864004b060 con 0x5588c116c3d0 2024-04-26T17:58:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.998+0000 7f864ef96740 10 client.25310 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.998+0000 7f864ef96740 10 client.25310 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:58:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.998+0000 7f864ef96740 20 client.25310 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.998+0000 7f864ef96740 1 --2- 172.21.15.31:0/3177279853 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5588c1166110 0x5588c1168530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.998+0000 7f864ef96740 4 client.25310 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:58:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.998+0000 7f864ef96740 1 -- 172.21.15.31:0/3177279853 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588c10d6070 con 0x5588c1166110 2024-04-26T17:58:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:56.999+0000 7f8649f40640 1 --2- 172.21.15.31:0/3177279853 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5588c1166110 0x5588c1168530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.002+0000 7f8649f40640 1 --2- 172.21.15.31:0/3177279853 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5588c1166110 0x5588c1168530 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f93166e04fffd4 server_cookie=13e75f332698f3cf in_seq=0 out_seq=0 2024-04-26T17:58:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.002+0000 7f862d7fa640 10 client.25310 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.003+0000 7f862d7fa640 1 -- 172.21.15.31:0/3177279853 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588c10d6070 con 0x5588c1166110 2024-04-26T17:58:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.003+0000 7f862d7fa640 10 client.25310 handle_command_reply: tid=0 2024-04-26T17:58:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.003+0000 7f864ef96740 1 client.25310 shutdown 2024-04-26T17:58:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.004+0000 7f864ef96740 1 -- 172.21.15.31:0/3177279853 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116c3d0 msgr2=0x5588c1175860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.004+0000 7f864ef96740 1 --2- 172.21.15.31:0/3177279853 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116c3d0 0x5588c1175860 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f86400ee290 tx=0x7f86400b8e30 comp rx=0 tx=0).stop 2024-04-26T17:58:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.005+0000 7f864ef96740 1 -- 172.21.15.31:0/3177279853 shutdown_connections 2024-04-26T17:58:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.005+0000 7f864ef96740 1 --2- 172.21.15.31:0/3177279853 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5588c1166110 0x5588c1168530 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.005+0000 7f864ef96740 1 --2- 172.21.15.31:0/3177279853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ffa0 0x5588c1175da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.005+0000 7f864ef96740 1 --2- 172.21.15.31:0/3177279853 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116c3d0 0x5588c1175860 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.005+0000 7f864ef96740 1 --2- 172.21.15.31:0/3177279853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116ba10 0x5588c1175320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.005+0000 7f864ef96740 1 -- 172.21.15.31:0/3177279853 >> 172.21.15.31:0/3177279853 conn(0x5588c115dec0 msgr2=0x5588c116b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.005+0000 7f862d7fa640 0 client.25310 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.005+0000 7f864ef96740 1 -- 172.21.15.31:0/3177279853 shutdown_connections 2024-04-26T17:58:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.005+0000 7f864ef96740 1 -- 172.21.15.31:0/3177279853 wait complete. 2024-04-26T17:58:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.006+0000 7f864ef96740 20 client.25310 trim_cache size 0 max 16384 2024-04-26T17:58:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.032+0000 7f864ef96740 1 Processor -- start 2024-04-26T17:58:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.032+0000 7f864ef96740 1 -- start start 2024-04-26T17:58:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.033+0000 7f864ef96740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116ba10 0x5588c116dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.033+0000 7f864ef96740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116c3d0 0x5588c11813f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.033+0000 7f864ef96740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116ffa0 0x5588c1172e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.033+0000 7f864ef96740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f86441b7260 con 0x5588c116c3d0 2024-04-26T17:58:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.033+0000 7f864ef96740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f86441b4ad0 con 0x5588c116ffa0 2024-04-26T17:58:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.034+0000 7f864ef96740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86441b2920 con 0x5588c116ba10 2024-04-26T17:58:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.034+0000 7f8649f40640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116ba10 0x5588c116dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.034+0000 7f864973f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116c3d0 0x5588c11813f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.034+0000 7f864973f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116c3d0 0x5588c11813f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60548/0 (socket says 172.21.15.31:60548) 2024-04-26T17:58:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.034+0000 7f864973f640 1 -- 172.21.15.31:0/2067144020 learned_addr learned my addr 172.21.15.31:0/2067144020 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.034+0000 7f864a741640 1 --2- 172.21.15.31:0/2067144020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116ffa0 0x5588c1172e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.035+0000 7f864973f640 1 -- 172.21.15.31:0/2067144020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116ba10 msgr2=0x5588c116dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.035+0000 7f864973f640 1 --2- 172.21.15.31:0/2067144020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116ba10 0x5588c116dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.035+0000 7f864973f640 1 -- 172.21.15.31:0/2067144020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116ffa0 msgr2=0x5588c1172e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.035+0000 7f864973f640 1 --2- 172.21.15.31:0/2067144020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116ffa0 0x5588c1172e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.035+0000 7f864973f640 1 -- 172.21.15.31:0/2067144020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588c11706f0 con 0x5588c116c3d0 2024-04-26T17:58:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.035+0000 7f864973f640 1 --2- 172.21.15.31:0/2067144020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116c3d0 0x5588c11813f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8640005c40 tx=0x7f8640003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=996090653fd016b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.035+0000 7f862f7fe640 1 -- 172.21.15.31:0/2067144020 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640048070 con 0x5588c116c3d0 2024-04-26T17:58:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.035+0000 7f862f7fe640 1 -- 172.21.15.31:0/2067144020 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86400f0df0 con 0x5588c116c3d0 2024-04-26T17:58:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.035+0000 7f862f7fe640 1 -- 172.21.15.31:0/2067144020 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86400c5310 con 0x5588c116c3d0 2024-04-26T17:58:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.036+0000 7f864ef96740 1 -- 172.21.15.31:0/2067144020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116c3d0 msgr2=0x5588c11813f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.036+0000 7f864ef96740 1 --2- 172.21.15.31:0/2067144020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116c3d0 0x5588c11813f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8640005c40 tx=0x7f8640003f50 comp rx=0 tx=0).stop 2024-04-26T17:58:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.036+0000 7f864ef96740 1 -- 172.21.15.31:0/2067144020 shutdown_connections 2024-04-26T17:58:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.036+0000 7f864ef96740 1 --2- 172.21.15.31:0/2067144020 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116ffa0 0x5588c1172e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.036+0000 7f864ef96740 1 --2- 172.21.15.31:0/2067144020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116c3d0 0x5588c11813f0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.036+0000 7f864ef96740 1 --2- 172.21.15.31:0/2067144020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116ba10 0x5588c116dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.036+0000 7f864ef96740 1 -- 172.21.15.31:0/2067144020 >> 172.21.15.31:0/2067144020 conn(0x5588c115dec0 msgr2=0x5588c116ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.036+0000 7f864ef96740 1 -- 172.21.15.31:0/2067144020 shutdown_connections 2024-04-26T17:58:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.036+0000 7f864ef96740 1 -- 172.21.15.31:0/2067144020 wait complete. 2024-04-26T17:58:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.036+0000 7f864ef96740 1 -- start start 2024-04-26T17:58:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.036+0000 7f864ef96740 1 Processor -- start 2024-04-26T17:58:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.037+0000 7f864ef96740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116ba10 0x5588c1197e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.037+0000 7f864ef96740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116c3d0 0x5588c117d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.037+0000 7f864ef96740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ffa0 0x5588c117de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.037+0000 7f864ef96740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f86441b2f90 con 0x5588c116ffa0 2024-04-26T17:58:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.037+0000 7f864ef96740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f86441b4400 con 0x5588c116c3d0 2024-04-26T17:58:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.037+0000 7f864ef96740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86441b3660 con 0x5588c116ba10 2024-04-26T17:58:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.037+0000 7f864ef96740 10 client.0 _ll_register_callbacks cb 0x7f8640009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:58:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.037+0000 7f864a741640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ffa0 0x5588c117de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.037+0000 7f864a741640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ffa0 0x5588c117de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60556/0 (socket says 172.21.15.31:60556) 2024-04-26T17:58:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.037+0000 7f864a741640 1 -- 172.21.15.31:0/3590340243 learned_addr learned my addr 172.21.15.31:0/3590340243 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:58:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.037+0000 7f864973f640 1 --2- 172.21.15.31:0/3590340243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116c3d0 0x5588c117d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.038+0000 7f864a741640 1 -- 172.21.15.31:0/3590340243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116ba10 msgr2=0x5588c1197e30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T17:58:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.038+0000 7f864a741640 1 --2- 172.21.15.31:0/3590340243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116ba10 0x5588c1197e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.038+0000 7f864a741640 1 -- 172.21.15.31:0/3590340243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116c3d0 msgr2=0x5588c117d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:58:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.038+0000 7f864a741640 1 --2- 172.21.15.31:0/3590340243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116c3d0 0x5588c117d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.038+0000 7f864a741640 1 -- 172.21.15.31:0/3590340243 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588c11721c0 con 0x5588c116ffa0 2024-04-26T17:58:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.039+0000 7f864a741640 1 --2- 172.21.15.31:0/3590340243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ffa0 0x5588c117de80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f863c0041a0 tx=0x7f863c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de4002f300277ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:58:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.039+0000 7f862d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:58:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.039+0000 7f862d7fa640 1 -- 172.21.15.31:0/3590340243 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c004b20 con 0x5588c116ffa0 2024-04-26T17:58:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.039+0000 7f862d7fa640 1 -- 172.21.15.31:0/3590340243 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f863c004cc0 con 0x5588c116ffa0 2024-04-26T17:58:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.039+0000 7f864ef96740 1 -- 172.21.15.31:0/3590340243 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86400f0df0 con 0x5588c116ffa0 2024-04-26T17:58:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.039+0000 7f862d7fa640 1 -- 172.21.15.31:0/3590340243 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f863c00f9e0 con 0x5588c116ffa0 2024-04-26T17:58:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.039+0000 7f864ef96740 10 client.15618 fetch_fsmap learned FSMap version 29 2024-04-26T17:58:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.039+0000 7f864ef96740 1 -- 172.21.15.31:0/3590340243 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5588c11f4880 con 0x5588c116ffa0 2024-04-26T17:58:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.040+0000 7f862d7fa640 1 -- 172.21.15.31:0/3590340243 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86400f0df0 con 0x5588c116ffa0 2024-04-26T17:58:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.040+0000 7f862d7fa640 1 -- 172.21.15.31:0/3590340243 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f863c027070 con 0x5588c116ffa0 2024-04-26T17:58:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.040+0000 7f864ef96740 10 client.15618 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:58:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.040+0000 7f864ef96740 10 client.15618 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:58:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.040+0000 7f864ef96740 20 client.15618 populate_metadata read hostname 'smithi031' 2024-04-26T17:58:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.040+0000 7f864ef96740 1 --2- 172.21.15.31:0/3590340243 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5588c11f50e0 0x5588c11f7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:58:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.040+0000 7f864ef96740 4 client.15618 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:58:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.040+0000 7f864ef96740 1 -- 172.21.15.31:0/3590340243 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5588c10cfee0 con 0x5588c11f50e0 2024-04-26T17:58:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.040+0000 7f8649f40640 1 --2- 172.21.15.31:0/3590340243 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5588c11f50e0 0x5588c11f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:58:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.040+0000 7f8649f40640 1 --2- 172.21.15.31:0/3590340243 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5588c11f50e0 0x5588c11f7500 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8323626c839a4aec server_cookie=b2024e50a2c0988d in_seq=0 out_seq=0 2024-04-26T17:58:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.040+0000 7f862d7fa640 10 client.15618 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:57.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.049+0000 7f862d7fa640 1 -- 172.21.15.31:0/3590340243 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5588c10cfee0 con 0x5588c11f50e0 2024-04-26T17:58:57.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.049+0000 7f862d7fa640 10 client.15618 handle_command_reply: tid=0 2024-04-26T17:58:57.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.049+0000 7f864ef96740 1 client.15618 shutdown 2024-04-26T17:58:57.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864ef96740 1 -- 172.21.15.31:0/3590340243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ffa0 msgr2=0x5588c117de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:57.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864ef96740 1 --2- 172.21.15.31:0/3590340243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ffa0 0x5588c117de80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f863c0041a0 tx=0x7f863c004ab0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864ef96740 1 -- 172.21.15.31:0/3590340243 shutdown_connections 2024-04-26T17:58:57.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864ef96740 1 --2- 172.21.15.31:0/3590340243 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5588c11f50e0 0x5588c11f7500 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864ef96740 1 --2- 172.21.15.31:0/3590340243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588c116ffa0 0x5588c117de80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864ef96740 1 --2- 172.21.15.31:0/3590340243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588c116c3d0 0x5588c117d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864ef96740 1 --2- 172.21.15.31:0/3590340243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588c116ba10 0x5588c1197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864ef96740 1 -- 172.21.15.31:0/3590340243 >> 172.21.15.31:0/3590340243 conn(0x5588c115dec0 msgr2=0x5588c1173bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f862d7fa640 0 client.15618 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:58:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864ef96740 1 -- 172.21.15.31:0/3590340243 shutdown_connections 2024-04-26T17:58:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864ef96740 1 -- 172.21.15.31:0/3590340243 wait complete. 2024-04-26T17:58:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864ef96740 20 client.15618 trim_cache size 0 max 16384 2024-04-26T17:58:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:58:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:58:57.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:58:57.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:58:57.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864c1a4640 1 -- 172.21.15.31:0/947694700 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f86300a3f20 msgr2=0x7f86300a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:57.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864c1a4640 1 --2- 172.21.15.31:0/947694700 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f86300a3f20 0x7f86300a63e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f863c005fd0 tx=0x7f863c009500 comp rx=0 tx=0).stop 2024-04-26T17:58:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864c1a4640 1 -- 172.21.15.31:0/947694700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644102220 msgr2=0x7f864406d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:58:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.050+0000 7f864c1a4640 1 --2- 172.21.15.31:0/947694700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644102220 0x7f864406d220 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8640031440 tx=0x7f8640004270 comp rx=0 tx=0).stop 2024-04-26T17:58:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.051+0000 7f864c1a4640 1 -- 172.21.15.31:0/947694700 shutdown_connections 2024-04-26T17:58:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.051+0000 7f864c1a4640 1 --2- 172.21.15.31:0/947694700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8644107820 0x7f864406d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.051+0000 7f864c1a4640 1 --2- 172.21.15.31:0/947694700 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f86300a3f20 0x7f86300a63e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.051+0000 7f864c1a4640 1 --2- 172.21.15.31:0/947694700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644102220 0x7f864406d220 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.051+0000 7f864c1a4640 1 --2- 172.21.15.31:0/947694700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8644101860 0x7f8644074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:58:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.051+0000 7f864c1a4640 1 -- 172.21.15.31:0/947694700 >> 172.21.15.31:0/947694700 conn(0x7f86440789b0 msgr2=0x7f86441043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:58:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.051+0000 7f864c1a4640 1 -- 172.21.15.31:0/947694700 shutdown_connections 2024-04-26T17:58:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:58:57.051+0000 7f864c1a4640 1 -- 172.21.15.31:0/947694700 wait complete. 2024-04-26T17:58:57.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:57 smithi097 ceph-mon[32419]: pgmap v76: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.8 MiB/s wr, 542 op/s 2024-04-26T17:58:57.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:57 smithi186 ceph-mon[32797]: pgmap v76: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.8 MiB/s wr, 542 op/s 2024-04-26T17:58:57.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:57 smithi031 ceph-mon[29164]: pgmap v76: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.8 MiB/s wr, 542 op/s 2024-04-26T17:58:58.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 95.0 s, 3337.2 tps, lat 29.926 ms stddev 29.747 2024-04-26T17:58:59.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:58:59 smithi097 ceph-mon[32419]: pgmap v77: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 641 op/s 2024-04-26T17:58:59.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:58:59 smithi186 ceph-mon[32797]: pgmap v77: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 641 op/s 2024-04-26T17:58:59.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:58:59 smithi031 ceph-mon[29164]: pgmap v77: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 641 op/s 2024-04-26T17:59:01.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:01 smithi097 ceph-mon[32419]: pgmap v78: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.8 MiB/s wr, 634 op/s 2024-04-26T17:59:01.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:01 smithi186 ceph-mon[32797]: pgmap v78: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.8 MiB/s wr, 634 op/s 2024-04-26T17:59:01.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:01 smithi031 ceph-mon[29164]: pgmap v78: 97 pgs: 97 active+clean; 1.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.8 MiB/s wr, 634 op/s 2024-04-26T17:59:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 100.0 s, 2336.4 tps, lat 42.651 ms stddev 82.661 2024-04-26T17:59:03.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:03 smithi097 ceph-mon[32419]: pgmap v79: 97 pgs: 97 active+clean; 1.3 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 968 op/s 2024-04-26T17:59:03.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:03 smithi186 ceph-mon[32797]: pgmap v79: 97 pgs: 97 active+clean; 1.3 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 968 op/s 2024-04-26T17:59:03.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:03 smithi031 ceph-mon[29164]: pgmap v79: 97 pgs: 97 active+clean; 1.3 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 968 op/s 2024-04-26T17:59:05.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:05 smithi097 ceph-mon[32419]: pgmap v80: 97 pgs: 97 active+clean; 1.3 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.5 MiB/s wr, 647 op/s 2024-04-26T17:59:05.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:05 smithi186 ceph-mon[32797]: pgmap v80: 97 pgs: 97 active+clean; 1.3 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.5 MiB/s wr, 647 op/s 2024-04-26T17:59:05.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:05 smithi031 ceph-mon[29164]: pgmap v80: 97 pgs: 97 active+clean; 1.3 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.5 MiB/s wr, 647 op/s 2024-04-26T17:59:07.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:07 smithi097 ceph-mon[32419]: pgmap v81: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 23 MiB/s wr, 1.55k op/s 2024-04-26T17:59:07.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:07 smithi186 ceph-mon[32797]: pgmap v81: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 23 MiB/s wr, 1.55k op/s 2024-04-26T17:59:07.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:07 smithi031 ceph-mon[29164]: pgmap v81: 97 pgs: 97 active+clean; 1.4 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 23 MiB/s wr, 1.55k op/s 2024-04-26T17:59:08.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 105.0 s, 3367.4 tps, lat 29.819 ms stddev 36.225 2024-04-26T17:59:09.361 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:09 smithi031 ceph-mon[29164]: pgmap v82: 97 pgs: 97 active+clean; 1.5 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 32 MiB/s wr, 1.99k op/s 2024-04-26T17:59:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:09 smithi097 ceph-mon[32419]: pgmap v82: 97 pgs: 97 active+clean; 1.5 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 32 MiB/s wr, 1.99k op/s 2024-04-26T17:59:09.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:09 smithi186 ceph-mon[32797]: pgmap v82: 97 pgs: 97 active+clean; 1.5 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 32 MiB/s wr, 1.99k op/s 2024-04-26T17:59:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.174+0000 7f32fc60f640 1 -- 172.21.15.31:0/1419311938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f41032f0 msgr2=0x7f32f410f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.174+0000 7f32fc60f640 1 --2- 172.21.15.31:0/1419311938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f41032f0 0x7f32f410f800 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009cd0 tx=0x7f32ec02a8e0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.174+0000 7f32fc60f640 1 -- 172.21.15.31:0/1419311938 shutdown_connections 2024-04-26T17:59:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.174+0000 7f32fc60f640 1 --2- 172.21.15.31:0/1419311938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f41032f0 0x7f32f410f800 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.174+0000 7f32fc60f640 1 --2- 172.21.15.31:0/1419311938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32f4102930 0x7f32f4102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.174+0000 7f32fc60f640 1 --2- 172.21.15.31:0/1419311938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32f4108940 0x7f32f4108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.174+0000 7f32fc60f640 1 -- 172.21.15.31:0/1419311938 >> 172.21.15.31:0/1419311938 conn(0x7f32f40fe420 msgr2=0x7f32f4100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.175+0000 7f32fc60f640 1 -- 172.21.15.31:0/1419311938 shutdown_connections 2024-04-26T17:59:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.175+0000 7f32fc60f640 1 -- 172.21.15.31:0/1419311938 wait complete. 2024-04-26T17:59:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.175+0000 7f32fc60f640 1 Processor -- start 2024-04-26T17:59:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.175+0000 7f32fc60f640 1 -- start start 2024-04-26T17:59:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fc60f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32f4102930 0x7f32f410ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fc60f640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32f41032f0 0x7f32f410f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fc60f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f4108940 0x7f32f41a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fc60f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f32f4112c00 con 0x7f32f4108940 2024-04-26T17:59:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fc60f640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f32f4112a80 con 0x7f32f41032f0 2024-04-26T17:59:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fc60f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f32f4112d80 con 0x7f32f4102930 2024-04-26T17:59:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fabac640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f4108940 0x7f32f41a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fabac640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f4108940 0x7f32f41a3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47768/0 (socket says 172.21.15.31:47768) 2024-04-26T17:59:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fabac640 1 -- 172.21.15.31:0/521545281 learned_addr learned my addr 172.21.15.31:0/521545281 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32f9baa640 1 --2- 172.21.15.31:0/521545281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32f41032f0 0x7f32f410f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fa3ab640 1 --2- 172.21.15.31:0/521545281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32f4102930 0x7f32f410ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fabac640 1 -- 172.21.15.31:0/521545281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32f4102930 msgr2=0x7f32f410ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fabac640 1 --2- 172.21.15.31:0/521545281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32f4102930 0x7f32f410ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fabac640 1 -- 172.21.15.31:0/521545281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32f41032f0 msgr2=0x7f32f410f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fabac640 1 --2- 172.21.15.31:0/521545281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32f41032f0 0x7f32f410f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fabac640 1 -- 172.21.15.31:0/521545281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f41a46a0 con 0x7f32f4108940 2024-04-26T17:59:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.176+0000 7f32fabac640 1 --2- 172.21.15.31:0/521545281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f4108940 0x7f32f41a3f70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f32ec033040 tx=0x7f32ec009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.177+0000 7f32eb7fe640 1 -- 172.21.15.31:0/521545281 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec004040 con 0x7f32f4108940 2024-04-26T17:59:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.177+0000 7f32fc60f640 1 -- 172.21.15.31:0/521545281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32f41a4930 con 0x7f32f4108940 2024-04-26T17:59:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.177+0000 7f32eb7fe640 1 -- 172.21.15.31:0/521545281 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32ec004690 con 0x7f32f4108940 2024-04-26T17:59:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.177+0000 7f32eb7fe640 1 -- 172.21.15.31:0/521545281 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec030d90 con 0x7f32f4108940 2024-04-26T17:59:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.177+0000 7f32fc60f640 1 -- 172.21.15.31:0/521545281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32f41ac220 con 0x7f32f4108940 2024-04-26T17:59:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.181+0000 7f32ff3fd740 1 Processor -- start 2024-04-26T17:59:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.182+0000 7f32eb7fe640 1 -- 172.21.15.31:0/521545281 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f32ec0041e0 con 0x7f32f4108940 2024-04-26T17:59:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.182+0000 7f32ff3fd740 1 -- start start 2024-04-26T17:59:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32ff3fd740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32ff3fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c1820 0x55ec053cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32ff3fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 0x55ec053ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32ff3fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ec05327460 con 0x55ec053cc0e0 2024-04-26T17:59:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32ff3fd740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ec052e1590 con 0x55ec053c0f00 2024-04-26T17:59:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32ff3fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ec05323420 con 0x55ec053c1820 2024-04-26T17:59:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32f9baa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32f9baa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:52902/0 (socket says 172.21.15.31:52902) 2024-04-26T17:59:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32f9baa640 1 -- 172.21.15.31:0/3655069972 learned_addr learned my addr 172.21.15.31:0/3655069972 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32f9baa640 1 --2- 172.21.15.31:0/3655069972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 0x55ec053ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32fa3ab640 1 --2- 172.21.15.31:0/3655069972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c1820 0x55ec053cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32f9baa640 1 -- 172.21.15.31:0/3655069972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c1820 msgr2=0x55ec053cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32f9baa640 1 --2- 172.21.15.31:0/3655069972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c1820 0x55ec053cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32eb7fe640 1 --2- 172.21.15.31:0/521545281 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f32e00a3e40 0x7f32e00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32f9baa640 1 -- 172.21.15.31:0/3655069972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 msgr2=0x55ec053ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32f9baa640 1 --2- 172.21.15.31:0/3655069972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 0x55ec053ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32f9baa640 1 -- 172.21.15.31:0/3655069972 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec052ee0d0 con 0x55ec053c0f00 2024-04-26T17:59:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32fa3ab640 1 --2- 172.21.15.31:0/3655069972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c1820 0x55ec053cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:59:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32eb7fe640 1 -- 172.21.15.31:0/521545281 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f32ec02e030 con 0x7f32f4108940 2024-04-26T17:59:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.183+0000 7f32fabac640 1 --2- 172.21.15.31:0/521545281 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f32e00a3e40 0x7f32e00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.184+0000 7f32f9baa640 1 --2- 172.21.15.31:0/3655069972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c12e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f32e4009fd0 tx=0x7f32e4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=844cc02a04875d64 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.184+0000 7f32e8bfe640 1 -- 172.21.15.31:0/3655069972 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e4004030 con 0x55ec053c0f00 2024-04-26T17:59:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.184+0000 7f32e8bfe640 1 -- 172.21.15.31:0/3655069972 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32e4005020 con 0x55ec053c0f00 2024-04-26T17:59:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.184+0000 7f32e8bfe640 1 -- 172.21.15.31:0/3655069972 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e4004be0 con 0x55ec053c0f00 2024-04-26T17:59:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.184+0000 7f32ff3fd740 1 -- 172.21.15.31:0/3655069972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 msgr2=0x55ec053c12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.184+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/3655069972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c12e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f32e4009fd0 tx=0x7f32e4005850 comp rx=0 tx=0).stop 2024-04-26T17:59:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.184+0000 7f32fabac640 1 --2- 172.21.15.31:0/521545281 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f32e00a3e40 0x7f32e00a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0ba150 tx=0x7f32ec0095c0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 -- 172.21.15.31:0/3655069972 shutdown_connections 2024-04-26T17:59:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/3655069972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 0x55ec053ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/3655069972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c1820 0x55ec053cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/3655069972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c12e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 -- 172.21.15.31:0/3655069972 >> 172.21.15.31:0/3655069972 conn(0x55ec053bd410 msgr2=0x55ec053bd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 -- 172.21.15.31:0/3655069972 shutdown_connections 2024-04-26T17:59:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 -- 172.21.15.31:0/3655069972 wait complete. 2024-04-26T17:59:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 -- start start 2024-04-26T17:59:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 Processor -- start 2024-04-26T17:59:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c0f00 0x55ec053c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c1820 0x55ec053c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 0x55ec053c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ec053ced10 con 0x55ec053cc0e0 2024-04-26T17:59:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ec052aeb60 con 0x55ec053c1820 2024-04-26T17:59:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ec053cee90 con 0x55ec053c0f00 2024-04-26T17:59:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32f9baa640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c0f00 0x55ec053c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32ff3fd740 10 client.0 _ll_register_callbacks cb 0x55ec052d32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:59:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32f9baa640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c0f00 0x55ec053c90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:40848/0 (socket says 172.21.15.31:40848) 2024-04-26T17:59:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32f9baa640 1 -- 172.21.15.31:0/654594136 learned_addr learned my addr 172.21.15.31:0/654594136 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.185+0000 7f32f9baa640 1 --2- 172.21.15.31:0/654594136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 0x55ec053c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.186+0000 7f32fa3ab640 1 --2- 172.21.15.31:0/654594136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c1820 0x55ec053c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.186+0000 7f32f9baa640 1 -- 172.21.15.31:0/654594136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c1820 msgr2=0x55ec053c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.186+0000 7f32f9baa640 1 --2- 172.21.15.31:0/654594136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c1820 0x55ec053c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.186+0000 7f32f9baa640 1 -- 172.21.15.31:0/654594136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 msgr2=0x55ec053c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.186+0000 7f32f9baa640 1 --2- 172.21.15.31:0/654594136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 0x55ec053c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.186+0000 7f32f9baa640 1 -- 172.21.15.31:0/654594136 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec05039210 con 0x55ec053c0f00 2024-04-26T17:59:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.186+0000 7f32f9baa640 1 --2- 172.21.15.31:0/654594136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c0f00 0x55ec053c90a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f32e4004ae0 tx=0x7f32e4004840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14f38df6b00cc58b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.186+0000 7f32ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T17:59:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.187+0000 7f32ddffb640 1 -- 172.21.15.31:0/654594136 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e4004c00 con 0x55ec053c0f00 2024-04-26T17:59:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.187+0000 7f32ddffb640 1 -- 172.21.15.31:0/654594136 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f32e4004da0 con 0x55ec053c0f00 2024-04-26T17:59:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.187+0000 7f32ff3fd740 1 -- 172.21.15.31:0/654594136 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32e4005020 con 0x55ec053c0f00 2024-04-26T17:59:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.187+0000 7f32ddffb640 1 -- 172.21.15.31:0/654594136 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f32e4004030 con 0x55ec053c0f00 2024-04-26T17:59:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.187+0000 7f32ff3fd740 10 client.25252 fetch_fsmap learned FSMap version 29 2024-04-26T17:59:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.187+0000 7f32ff3fd740 1 -- 172.21.15.31:0/654594136 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ec05105620 con 0x55ec053c0f00 2024-04-26T17:59:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.187+0000 7f32ddffb640 1 -- 172.21.15.31:0/654594136 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32e4005020 con 0x55ec053c0f00 2024-04-26T17:59:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.188+0000 7f32ddffb640 1 -- 172.21.15.31:0/654594136 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f32e40035f0 con 0x55ec053c0f00 2024-04-26T17:59:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.188+0000 7f32ff3fd740 10 client.25252 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:59:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.188+0000 7f32ff3fd740 10 client.25252 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:59:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.188+0000 7f32ff3fd740 20 client.25252 populate_metadata read hostname 'smithi031' 2024-04-26T17:59:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.188+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/654594136 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec053dc0e0 0x55ec053de4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.188+0000 7f32ff3fd740 4 client.25252 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:59:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.188+0000 7f32ff3fd740 1 -- 172.21.15.31:0/654594136 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec05335070 con 0x55ec053dc0e0 2024-04-26T17:59:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.188+0000 7f32fa3ab640 1 --2- 172.21.15.31:0/654594136 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec053dc0e0 0x55ec053de4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.189+0000 7f32fa3ab640 1 --2- 172.21.15.31:0/654594136 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec053dc0e0 0x55ec053de4d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b84d4bb15416800c server_cookie=da2d0c159e3830a5 in_seq=0 out_seq=0 2024-04-26T17:59:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.189+0000 7f32ddffb640 10 client.25252 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ddffb640 1 -- 172.21.15.31:0/654594136 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ec05335070 con 0x55ec053dc0e0 2024-04-26T17:59:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ddffb640 10 client.25252 handle_command_reply: tid=0 2024-04-26T17:59:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ff3fd740 1 client.25252 shutdown 2024-04-26T17:59:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ff3fd740 1 -- 172.21.15.31:0/654594136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c0f00 msgr2=0x55ec053c90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/654594136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c0f00 0x55ec053c90a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f32e4004ae0 tx=0x7f32e4004840 comp rx=0 tx=0).stop 2024-04-26T17:59:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ff3fd740 1 -- 172.21.15.31:0/654594136 shutdown_connections 2024-04-26T17:59:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/654594136 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec053dc0e0 0x55ec053de4d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/654594136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 0x55ec053c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/654594136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c1820 0x55ec053c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ddffb640 0 client.25252 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/654594136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c0f00 0x55ec053c90a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ff3fd740 1 -- 172.21.15.31:0/654594136 >> 172.21.15.31:0/654594136 conn(0x55ec053bd410 msgr2=0x55ec053c3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ff3fd740 1 -- 172.21.15.31:0/654594136 shutdown_connections 2024-04-26T17:59:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ff3fd740 1 -- 172.21.15.31:0/654594136 wait complete. 2024-04-26T17:59:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.190+0000 7f32ff3fd740 20 client.25252 trim_cache size 0 max 16384 2024-04-26T17:59:10.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.211+0000 7f32ff3fd740 1 Processor -- start 2024-04-26T17:59:10.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32ff3fd740 1 -- start start 2024-04-26T17:59:10.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32ff3fd740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32ff3fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053c1820 0x55ec053e3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32ff3fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053cc0e0 0x55ec053e4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32ff3fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f32f41b7280 con 0x55ec053c1820 2024-04-26T17:59:10.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32ff3fd740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f32f41b4af0 con 0x55ec053c0f00 2024-04-26T17:59:10.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32ff3fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f32f41b2940 con 0x55ec053cc0e0 2024-04-26T17:59:10.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32f9baa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32fa3ab640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053c1820 0x55ec053e3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32f9baa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:52930/0 (socket says 172.21.15.31:52930) 2024-04-26T17:59:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32f9baa640 1 -- 172.21.15.31:0/3709994595 learned_addr learned my addr 172.21.15.31:0/3709994595 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32f9baa640 1 --2- 172.21.15.31:0/3709994595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053cc0e0 0x55ec053e4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32f9baa640 1 -- 172.21.15.31:0/3709994595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053cc0e0 msgr2=0x55ec053e4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32f9baa640 1 --2- 172.21.15.31:0/3709994595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053cc0e0 0x55ec053e4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32f9baa640 1 -- 172.21.15.31:0/3709994595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053c1820 msgr2=0x55ec053e3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32f9baa640 1 --2- 172.21.15.31:0/3709994595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053c1820 0x55ec053e3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.213+0000 7f32f9baa640 1 -- 172.21.15.31:0/3709994595 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec053ce4d0 con 0x55ec053c0f00 2024-04-26T17:59:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.214+0000 7f32f9baa640 1 --2- 172.21.15.31:0/3709994595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c38c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f32e40058c0 tx=0x7f32e4005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7b52e1c30fb67e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.214+0000 7f32dffff640 1 -- 172.21.15.31:0/3709994595 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e4036040 con 0x55ec053c0f00 2024-04-26T17:59:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.214+0000 7f32dffff640 1 -- 172.21.15.31:0/3709994595 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32e40048b0 con 0x55ec053c0f00 2024-04-26T17:59:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.214+0000 7f32dffff640 1 -- 172.21.15.31:0/3709994595 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e40041d0 con 0x55ec053c0f00 2024-04-26T17:59:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.215+0000 7f32ff3fd740 1 -- 172.21.15.31:0/3709994595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 msgr2=0x55ec053c38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.215+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/3709994595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c38c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f32e40058c0 tx=0x7f32e4005f90 comp rx=0 tx=0).stop 2024-04-26T17:59:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.215+0000 7f32ff3fd740 1 -- 172.21.15.31:0/3709994595 shutdown_connections 2024-04-26T17:59:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.215+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/3709994595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053cc0e0 0x55ec053e4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.215+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/3709994595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053c1820 0x55ec053e3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.215+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/3709994595 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c38c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.215+0000 7f32ff3fd740 1 -- 172.21.15.31:0/3709994595 >> 172.21.15.31:0/3709994595 conn(0x55ec053bd410 msgr2=0x55ec053c7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.215+0000 7f32ff3fd740 1 -- 172.21.15.31:0/3709994595 shutdown_connections 2024-04-26T17:59:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.215+0000 7f32ff3fd740 1 -- 172.21.15.31:0/3709994595 wait complete. 2024-04-26T17:59:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.216+0000 7f32ff3fd740 1 -- start start 2024-04-26T17:59:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.216+0000 7f32ff3fd740 1 Processor -- start 2024-04-26T17:59:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.216+0000 7f32ff3fd740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.216+0000 7f32ff3fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c1820 0x55ec053c6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.216+0000 7f32ff3fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 0x55ec053c6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.216+0000 7f32ff3fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f32f41b2fb0 con 0x55ec053cc0e0 2024-04-26T17:59:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.216+0000 7f32ff3fd740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f32f41b4420 con 0x55ec053c0f00 2024-04-26T17:59:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.216+0000 7f32ff3fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f32f41b3680 con 0x55ec053c1820 2024-04-26T17:59:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.216+0000 7f32ff3fd740 10 client.0 _ll_register_callbacks cb 0x7f32e4009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:59:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.216+0000 7f32f9baa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.216+0000 7f32f9baa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:52934/0 (socket says 172.21.15.31:52934) 2024-04-26T17:59:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.216+0000 7f32f9baa640 1 -- 172.21.15.31:0/712513579 learned_addr learned my addr 172.21.15.31:0/712513579 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.216+0000 7f32fa3ab640 1 --2- 172.21.15.31:0/712513579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c1820 0x55ec053c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.216+0000 7f32f9baa640 1 --2- 172.21.15.31:0/712513579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 0x55ec053c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.217+0000 7f32fa3ab640 1 -- 172.21.15.31:0/712513579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 msgr2=0x55ec053c5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.217+0000 7f32fa3ab640 1 --2- 172.21.15.31:0/712513579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.217+0000 7f32fa3ab640 1 -- 172.21.15.31:0/712513579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 msgr2=0x55ec053c6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.217+0000 7f32fa3ab640 1 --2- 172.21.15.31:0/712513579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 0x55ec053c6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.217+0000 7f32fa3ab640 1 -- 172.21.15.31:0/712513579 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec054415a0 con 0x55ec053c1820 2024-04-26T17:59:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.217+0000 7f32f9baa640 1 --2- 172.21.15.31:0/712513579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:59:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.218+0000 7f32fa3ab640 1 --2- 172.21.15.31:0/712513579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c1820 0x55ec053c6200 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x55ec053dec70 tx=0x7f32f00044d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d89fd7b8e99b752b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.218+0000 7f32ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T17:59:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.218+0000 7f32ddffb640 1 -- 172.21.15.31:0/712513579 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f0005260 con 0x55ec053c1820 2024-04-26T17:59:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.219+0000 7f32ddffb640 1 -- 172.21.15.31:0/712513579 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f32f0005400 con 0x55ec053c1820 2024-04-26T17:59:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.219+0000 7f32ff3fd740 1 -- 172.21.15.31:0/712513579 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32e40048b0 con 0x55ec053c1820 2024-04-26T17:59:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.219+0000 7f32ddffb640 1 -- 172.21.15.31:0/712513579 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f32f0005620 con 0x55ec053c1820 2024-04-26T17:59:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.219+0000 7f32ff3fd740 10 client.25258 fetch_fsmap learned FSMap version 29 2024-04-26T17:59:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.219+0000 7f32ff3fd740 1 -- 172.21.15.31:0/712513579 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ec05442850 con 0x55ec053c1820 2024-04-26T17:59:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.219+0000 7f32ddffb640 1 -- 172.21.15.31:0/712513579 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32e40048b0 con 0x55ec053c1820 2024-04-26T17:59:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.219+0000 7f32ddffb640 1 -- 172.21.15.31:0/712513579 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f32f0003a40 con 0x55ec053c1820 2024-04-26T17:59:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.220+0000 7f32ff3fd740 10 client.25258 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T17:59:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.220+0000 7f32ff3fd740 10 client.25258 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:59:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.220+0000 7f32ff3fd740 20 client.25258 populate_metadata read hostname 'smithi031' 2024-04-26T17:59:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.220+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/712513579 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec05442f90 0x55ec054453b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.220+0000 7f32ff3fd740 4 client.25258 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:59:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.220+0000 7f32ff3fd740 1 -- 172.21.15.31:0/712513579 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ec0532eee0 con 0x55ec05442f90 2024-04-26T17:59:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.220+0000 7f32f9baa640 1 --2- 172.21.15.31:0/712513579 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec05442f90 0x55ec054453b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.221+0000 7f32f9baa640 1 --2- 172.21.15.31:0/712513579 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec05442f90 0x55ec054453b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56099abac8f5c9e8 server_cookie=207942fd705578bd in_seq=0 out_seq=0 2024-04-26T17:59:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.221+0000 7f32ddffb640 10 client.25258 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.236+0000 7f32ddffb640 1 -- 172.21.15.31:0/712513579 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ec0532eee0 con 0x55ec05442f90 2024-04-26T17:59:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.236+0000 7f32ddffb640 10 client.25258 handle_command_reply: tid=0 2024-04-26T17:59:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.236+0000 7f32ff3fd740 1 client.25258 shutdown 2024-04-26T17:59:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.237+0000 7f32ff3fd740 1 -- 172.21.15.31:0/712513579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c1820 msgr2=0x55ec053c6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.237+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/712513579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c1820 0x55ec053c6200 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x55ec053dec70 tx=0x7f32f00044d0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.237+0000 7f32ff3fd740 1 -- 172.21.15.31:0/712513579 shutdown_connections 2024-04-26T17:59:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.237+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/712513579 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ec05442f90 0x55ec054453b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.237+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/712513579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec053cc0e0 0x55ec053c6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.237+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/712513579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec053c1820 0x55ec053c6200 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.237+0000 7f32ff3fd740 1 --2- 172.21.15.31:0/712513579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ec053c0f00 0x55ec053c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.237+0000 7f32ff3fd740 1 -- 172.21.15.31:0/712513579 >> 172.21.15.31:0/712513579 conn(0x55ec053bd410 msgr2=0x55ec053bbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.237+0000 7f32ddffb640 0 client.25258 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.237+0000 7f32ff3fd740 1 -- 172.21.15.31:0/712513579 shutdown_connections 2024-04-26T17:59:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.237+0000 7f32ff3fd740 1 -- 172.21.15.31:0/712513579 wait complete. 2024-04-26T17:59:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.237+0000 7f32ff3fd740 20 client.25258 trim_cache size 0 max 16384 2024-04-26T17:59:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:59:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:59:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:59:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:59:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.238+0000 7f32fc60f640 1 -- 172.21.15.31:0/521545281 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f32e00a3e40 msgr2=0x7f32e00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.238+0000 7f32fc60f640 1 --2- 172.21.15.31:0/521545281 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f32e00a3e40 0x7f32e00a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0ba150 tx=0x7f32ec0095c0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.238+0000 7f32fc60f640 1 -- 172.21.15.31:0/521545281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f4108940 msgr2=0x7f32f41a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.238+0000 7f32fc60f640 1 --2- 172.21.15.31:0/521545281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f4108940 0x7f32f41a3f70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f32ec033040 tx=0x7f32ec009360 comp rx=0 tx=0).stop 2024-04-26T17:59:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.238+0000 7f32fc60f640 1 -- 172.21.15.31:0/521545281 shutdown_connections 2024-04-26T17:59:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.238+0000 7f32fc60f640 1 --2- 172.21.15.31:0/521545281 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f32e00a3e40 0x7f32e00a6300 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.238+0000 7f32fc60f640 1 --2- 172.21.15.31:0/521545281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32f4108940 0x7f32f41a3f70 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.238+0000 7f32fc60f640 1 --2- 172.21.15.31:0/521545281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f32f41032f0 0x7f32f410f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.238+0000 7f32fc60f640 1 --2- 172.21.15.31:0/521545281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32f4102930 0x7f32f410ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.238+0000 7f32fc60f640 1 -- 172.21.15.31:0/521545281 >> 172.21.15.31:0/521545281 conn(0x7f32f40fe420 msgr2=0x7f32f40fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.239+0000 7f32fc60f640 1 -- 172.21.15.31:0/521545281 shutdown_connections 2024-04-26T17:59:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:10.239+0000 7f32fc60f640 1 -- 172.21.15.31:0/521545281 wait complete. 2024-04-26T17:59:11.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:11 smithi097 ceph-mon[32419]: pgmap v83: 97 pgs: 97 active+clean; 1.5 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 30 MiB/s wr, 1.87k op/s 2024-04-26T17:59:11.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:11 smithi186 ceph-mon[32797]: pgmap v83: 97 pgs: 97 active+clean; 1.5 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 30 MiB/s wr, 1.87k op/s 2024-04-26T17:59:11.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:11 smithi031 ceph-mon[29164]: pgmap v83: 97 pgs: 97 active+clean; 1.5 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 30 MiB/s wr, 1.87k op/s 2024-04-26T17:59:13.095 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 110.0 s, 3238.4 tps, lat 30.854 ms stddev 30.506 2024-04-26T17:59:13.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:13 smithi097 ceph-mon[32419]: pgmap v84: 97 pgs: 97 active+clean; 1.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 33 MiB/s wr, 2.19k op/s 2024-04-26T17:59:13.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:13 smithi186 ceph-mon[32797]: pgmap v84: 97 pgs: 97 active+clean; 1.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 33 MiB/s wr, 2.19k op/s 2024-04-26T17:59:13.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:13 smithi031 ceph-mon[29164]: pgmap v84: 97 pgs: 97 active+clean; 1.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 33 MiB/s wr, 2.19k op/s 2024-04-26T17:59:15.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:15 smithi097 ceph-mon[32419]: pgmap v85: 97 pgs: 97 active+clean; 1.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 30 MiB/s wr, 1.85k op/s 2024-04-26T17:59:15.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:15 smithi186 ceph-mon[32797]: pgmap v85: 97 pgs: 97 active+clean; 1.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 30 MiB/s wr, 1.85k op/s 2024-04-26T17:59:15.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:15 smithi031 ceph-mon[29164]: pgmap v85: 97 pgs: 97 active+clean; 1.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 30 MiB/s wr, 1.85k op/s 2024-04-26T17:59:17.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:17 smithi097 ceph-mon[32419]: pgmap v86: 97 pgs: 97 active+clean; 1.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 31 MiB/s wr, 2.03k op/s 2024-04-26T17:59:17.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:17 smithi186 ceph-mon[32797]: pgmap v86: 97 pgs: 97 active+clean; 1.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 31 MiB/s wr, 2.03k op/s 2024-04-26T17:59:17.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:17 smithi031 ceph-mon[29164]: pgmap v86: 97 pgs: 97 active+clean; 1.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 31 MiB/s wr, 2.03k op/s 2024-04-26T17:59:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 115.0 s, 3685.3 tps, lat 27.219 ms stddev 27.724 2024-04-26T17:59:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:19 smithi097 ceph-mon[32419]: pgmap v87: 97 pgs: 97 active+clean; 1.6 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 18 MiB/s wr, 1.25k op/s 2024-04-26T17:59:19.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:19 smithi186 ceph-mon[32797]: pgmap v87: 97 pgs: 97 active+clean; 1.6 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 18 MiB/s wr, 1.25k op/s 2024-04-26T17:59:19.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:19 smithi031 ceph-mon[29164]: pgmap v87: 97 pgs: 97 active+clean; 1.6 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 18 MiB/s wr, 1.25k op/s 2024-04-26T17:59:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:21 smithi097 ceph-mon[32419]: pgmap v88: 97 pgs: 97 active+clean; 1.6 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.7 MiB/s wr, 621 op/s 2024-04-26T17:59:21.429 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:21 smithi031 ceph-mon[29164]: pgmap v88: 97 pgs: 97 active+clean; 1.6 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.7 MiB/s wr, 621 op/s 2024-04-26T17:59:21.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:21 smithi186 ceph-mon[32797]: pgmap v88: 97 pgs: 97 active+clean; 1.6 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.7 MiB/s wr, 621 op/s 2024-04-26T17:59:22.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:22 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:59:22.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:22 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:59:22.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:22 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:59:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 120.0 s, 3905.9 tps, lat 25.453 ms stddev 25.267 2024-04-26T17:59:23.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.365+0000 7f691159a640 1 -- 172.21.15.31:0/2452663038 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c030d70 con 0x7f6914104400 2024-04-26T17:59:23.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.366+0000 7f6918c6d640 1 -- 172.21.15.31:0/2452663038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6914104400 msgr2=0x7f691410c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:23.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.366+0000 7f6918c6d640 1 --2- 172.21.15.31:0/2452663038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6914104400 0x7f691410c8e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f690c009be0 tx=0x7f690c02a750 comp rx=0 tx=0).stop 2024-04-26T17:59:23.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.366+0000 7f6918c6d640 1 -- 172.21.15.31:0/2452663038 shutdown_connections 2024-04-26T17:59:23.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.366+0000 7f6918c6d640 1 --2- 172.21.15.31:0/2452663038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f691410cf00 0x7f691410f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.366+0000 7f6918c6d640 1 --2- 172.21.15.31:0/2452663038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6914104400 0x7f691410c8e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.366+0000 7f6918c6d640 1 --2- 172.21.15.31:0/2452663038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6914103ae0 0x7f6914103ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.366+0000 7f6918c6d640 1 -- 172.21.15.31:0/2452663038 >> 172.21.15.31:0/2452663038 conn(0x7f69140fd300 msgr2=0x7f69140ff720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:23.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.367+0000 7f6918c6d640 1 -- 172.21.15.31:0/2452663038 shutdown_connections 2024-04-26T17:59:23.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.367+0000 7f6918c6d640 1 -- 172.21.15.31:0/2452663038 wait complete. 2024-04-26T17:59:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.368+0000 7f6918c6d640 1 Processor -- start 2024-04-26T17:59:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.369+0000 7f6918c6d640 1 -- start start 2024-04-26T17:59:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.369+0000 7f6918c6d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6914103ae0 0x7f691419fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.369+0000 7f6918c6d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6914104400 0x7f69141a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.369+0000 7f6918c6d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f691410cf00 0x7f691410bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.369+0000 7f6918c6d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6914112c00 con 0x7f691410cf00 2024-04-26T17:59:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.369+0000 7f6918c6d640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6914112a80 con 0x7f6914104400 2024-04-26T17:59:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.369+0000 7f6918c6d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6914112d80 con 0x7f6914103ae0 2024-04-26T17:59:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.369+0000 7f6911d9b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6914104400 0x7f69141a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.369+0000 7f6911d9b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6914104400 0x7f69141a0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:39828/0 (socket says 172.21.15.31:39828) 2024-04-26T17:59:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.369+0000 7f6911d9b640 1 -- 172.21.15.31:0/1138611061 learned_addr learned my addr 172.21.15.31:0/1138611061 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.369+0000 7f691259c640 1 --2- 172.21.15.31:0/1138611061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6914103ae0 0x7f691419fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.369+0000 7f6912d9d640 1 --2- 172.21.15.31:0/1138611061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f691410cf00 0x7f691410bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.370+0000 7f6911d9b640 1 -- 172.21.15.31:0/1138611061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6914103ae0 msgr2=0x7f691419fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.370+0000 7f6911d9b640 1 --2- 172.21.15.31:0/1138611061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6914103ae0 0x7f691419fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.370+0000 7f6911d9b640 1 -- 172.21.15.31:0/1138611061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f691410cf00 msgr2=0x7f691410bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.370+0000 7f6911d9b640 1 --2- 172.21.15.31:0/1138611061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f691410cf00 0x7f691410bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.370+0000 7f6911d9b640 1 -- 172.21.15.31:0/1138611061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f691410c580 con 0x7f6914104400 2024-04-26T17:59:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.370+0000 7f6911d9b640 1 --2- 172.21.15.31:0/1138611061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6914104400 0x7f69141a0300 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f690c009d00 tx=0x7f690c003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.370+0000 7f69037fe640 1 -- 172.21.15.31:0/1138611061 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c041020 con 0x7f6914104400 2024-04-26T17:59:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.370+0000 7f69037fe640 1 -- 172.21.15.31:0/1138611061 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f690c0318f0 con 0x7f6914104400 2024-04-26T17:59:23.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.370+0000 7f69037fe640 1 -- 172.21.15.31:0/1138611061 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c036040 con 0x7f6914104400 2024-04-26T17:59:23.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.370+0000 7f6918c6d640 1 -- 172.21.15.31:0/1138611061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f691410c7b0 con 0x7f6914104400 2024-04-26T17:59:23.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.371+0000 7f6918c6d640 1 -- 172.21.15.31:0/1138611061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f691419b920 con 0x7f6914104400 2024-04-26T17:59:23.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.373+0000 7f69037fe640 1 -- 172.21.15.31:0/1138611061 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f690c044020 con 0x7f6914104400 2024-04-26T17:59:23.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.374+0000 7f69037fe640 1 --2- 172.21.15.31:0/1138611061 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f68fc0a3f70 0x7f68fc0a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.374+0000 7f691b985740 1 Processor -- start 2024-04-26T17:59:23.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.374+0000 7f691259c640 1 --2- 172.21.15.31:0/1138611061 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f68fc0a3f70 0x7f68fc0a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.374+0000 7f69037fe640 1 -- 172.21.15.31:0/1138611061 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f690c0f1280 con 0x7f6914104400 2024-04-26T17:59:23.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.374+0000 7f691b985740 1 -- start start 2024-04-26T17:59:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.375+0000 7f691b985740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706adda2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.375+0000 7f691259c640 1 --2- 172.21.15.31:0/1138611061 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f68fc0a3f70 0x7f68fc0a6430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f69080095f0 tx=0x7f6908005f90 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.375+0000 7f691b985740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706adda820 0x55706ade4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.375+0000 7f691b985740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706ade50e0 0x55706ade74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.375+0000 7f691b985740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55706ad40460 con 0x55706adda820 2024-04-26T17:59:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.375+0000 7f691b985740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55706acfa590 con 0x55706add9f00 2024-04-26T17:59:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.375+0000 7f691b985740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55706ad3c420 con 0x55706ade50e0 2024-04-26T17:59:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.375+0000 7f691259c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706adda2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.375+0000 7f6911d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706adda820 0x55706ade4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.375+0000 7f691259c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706adda2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:39840/0 (socket says 172.21.15.31:39840) 2024-04-26T17:59:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.375+0000 7f691259c640 1 -- 172.21.15.31:0/739739643 learned_addr learned my addr 172.21.15.31:0/739739643 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.376+0000 7f691259c640 1 -- 172.21.15.31:0/739739643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706ade50e0 msgr2=0x55706ade74d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T17:59:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.376+0000 7f691259c640 1 --2- 172.21.15.31:0/739739643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706ade50e0 0x55706ade74d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.376+0000 7f691259c640 1 -- 172.21.15.31:0/739739643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706adda820 msgr2=0x55706ade4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:23.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.376+0000 7f691259c640 1 --2- 172.21.15.31:0/739739643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706adda820 0x55706ade4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.376+0000 7f691259c640 1 -- 172.21.15.31:0/739739643 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55706ad070d0 con 0x55706add9f00 2024-04-26T17:59:23.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.376+0000 7f6911d9b640 1 --2- 172.21.15.31:0/739739643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706adda820 0x55706ade4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:59:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.376+0000 7f691259c640 1 --2- 172.21.15.31:0/739739643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706adda2e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f69080093e0 tx=0x7f6908014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8063ee3c4005865 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.376+0000 7f6900ff9640 1 -- 172.21.15.31:0/739739643 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690801f620 con 0x55706add9f00 2024-04-26T17:59:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.377+0000 7f6900ff9640 1 -- 172.21.15.31:0/739739643 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f690801fc70 con 0x55706add9f00 2024-04-26T17:59:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.377+0000 7f6900ff9640 1 -- 172.21.15.31:0/739739643 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690801dbf0 con 0x55706add9f00 2024-04-26T17:59:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.378+0000 7f691b985740 1 -- 172.21.15.31:0/739739643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 msgr2=0x55706adda2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.378+0000 7f691b985740 1 --2- 172.21.15.31:0/739739643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706adda2e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f69080093e0 tx=0x7f6908014000 comp rx=0 tx=0).stop 2024-04-26T17:59:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.378+0000 7f691b985740 1 -- 172.21.15.31:0/739739643 shutdown_connections 2024-04-26T17:59:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.378+0000 7f691b985740 1 --2- 172.21.15.31:0/739739643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706ade50e0 0x55706ade74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.378+0000 7f691b985740 1 --2- 172.21.15.31:0/739739643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706adda820 0x55706ade4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.378+0000 7f691b985740 1 --2- 172.21.15.31:0/739739643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706adda2e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.378+0000 7f691b985740 1 -- 172.21.15.31:0/739739643 >> 172.21.15.31:0/739739643 conn(0x55706add6410 msgr2=0x55706add67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.378+0000 7f691b985740 1 -- 172.21.15.31:0/739739643 shutdown_connections 2024-04-26T17:59:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.378+0000 7f691b985740 1 -- 172.21.15.31:0/739739643 wait complete. 2024-04-26T17:59:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.379+0000 7f691b985740 1 -- start start 2024-04-26T17:59:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.379+0000 7f691b985740 1 Processor -- start 2024-04-26T17:59:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.379+0000 7f691b985740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706ade20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f691259c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706ade20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f691259c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706ade20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:39844/0 (socket says 172.21.15.31:39844) 2024-04-26T17:59:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f691259c640 1 -- 172.21.15.31:0/451979098 learned_addr learned my addr 172.21.15.31:0/451979098 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f691b985740 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706adda820 0x55706ade2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f691b985740 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706ade50e0 0x55706ade2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f6912d9d640 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706ade50e0 0x55706ade2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f691b985740 1 -- 172.21.15.31:0/451979098 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55706ade7d10 con 0x55706adda820 2024-04-26T17:59:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f691b985740 1 -- 172.21.15.31:0/451979098 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55706acc7b60 con 0x55706add9f00 2024-04-26T17:59:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f691b985740 1 -- 172.21.15.31:0/451979098 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55706ade7e90 con 0x55706ade50e0 2024-04-26T17:59:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f6911d9b640 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706adda820 0x55706ade2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f691b985740 10 client.0 _ll_register_callbacks cb 0x55706acec2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:59:23.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f691259c640 1 -- 172.21.15.31:0/451979098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706ade50e0 msgr2=0x55706ade2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:23.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f691259c640 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706ade50e0 0x55706ade2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f691259c640 1 -- 172.21.15.31:0/451979098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706adda820 msgr2=0x55706ade2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f691259c640 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706adda820 0x55706ade2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.380+0000 7f691259c640 1 -- 172.21.15.31:0/451979098 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55706aa52210 con 0x55706add9f00 2024-04-26T17:59:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.381+0000 7f6912d9d640 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706ade50e0 0x55706ade2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:59:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.381+0000 7f691b985740 1 -- 172.21.15.31:0/451979098 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f690801fc70 con 0x55706add9f00 2024-04-26T17:59:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.381+0000 7f691259c640 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706ade20e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f690801db80 tx=0x7f690801dbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4431eb35c1cbaa83 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.381+0000 7f68f9ffb640 10 client.25343 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T17:59:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.381+0000 7f68f9ffb640 1 -- 172.21.15.31:0/451979098 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6908005040 con 0x55706add9f00 2024-04-26T17:59:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.381+0000 7f68f9ffb640 1 -- 172.21.15.31:0/451979098 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f69080051e0 con 0x55706add9f00 2024-04-26T17:59:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.381+0000 7f68f9ffb640 1 -- 172.21.15.31:0/451979098 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6908005420 con 0x55706add9f00 2024-04-26T17:59:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.381+0000 7f691b985740 10 client.25343 fetch_fsmap learned FSMap version 30 2024-04-26T17:59:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.381+0000 7f691b985740 1 -- 172.21.15.31:0/451979098 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({fsmap=30}) -- 0x55706ab1e620 con 0x55706add9f00 2024-04-26T17:59:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.382+0000 7f68f9ffb640 1 -- 172.21.15.31:0/451979098 <== mon.2 v2:172.21.15.97:3300/0 4 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f690801f420 con 0x55706add9f00 2024-04-26T17:59:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.382+0000 7f691b985740 10 client.25343 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T17:59:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.382+0000 7f691b985740 10 client.25343 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:59:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.382+0000 7f691b985740 20 client.25343 populate_metadata read hostname 'smithi031' 2024-04-26T17:59:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.382+0000 7f691b985740 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55706adf4e00 0x55706adf71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.382+0000 7f691b985740 4 client.25343 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:59:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.382+0000 7f691b985740 1 -- 172.21.15.31:0/451979098 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55706ad4e070 con 0x55706adf4e00 2024-04-26T17:59:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.382+0000 7f6911d9b640 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55706adf4e00 0x55706adf71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.382+0000 7f68f9ffb640 10 client.25343 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.382+0000 7f6911d9b640 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55706adf4e00 0x55706adf71f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=560c8224c637ec8f server_cookie=fbe7fcbd07679a29 in_seq=0 out_seq=0 2024-04-26T17:59:23.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:23 smithi097 ceph-mon[32419]: pgmap v89: 97 pgs: 97 active+clean; 1.6 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-26T17:59:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f68f9ffb640 1 -- 172.21.15.31:0/451979098 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55706ad4e070 con 0x55706adf4e00 2024-04-26T17:59:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f68f9ffb640 10 client.25343 handle_command_reply: tid=0 2024-04-26T17:59:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f691b985740 1 client.25343 shutdown 2024-04-26T17:59:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f691b985740 1 -- 172.21.15.31:0/451979098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 msgr2=0x55706ade20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f691b985740 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706ade20e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f690801db80 tx=0x7f690801dbb0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f691b985740 1 -- 172.21.15.31:0/451979098 shutdown_connections 2024-04-26T17:59:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f691b985740 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55706adf4e00 0x55706adf71f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f68f9ffb640 0 client.25343 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f691b985740 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706ade50e0 0x55706ade2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f691b985740 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706adda820 0x55706ade2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f691b985740 1 --2- 172.21.15.31:0/451979098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706ade20e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f691b985740 1 -- 172.21.15.31:0/451979098 >> 172.21.15.31:0/451979098 conn(0x55706add6410 msgr2=0x55706addc190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f691b985740 1 -- 172.21.15.31:0/451979098 shutdown_connections 2024-04-26T17:59:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f691b985740 1 -- 172.21.15.31:0/451979098 wait complete. 2024-04-26T17:59:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.383+0000 7f691b985740 20 client.25343 trim_cache size 0 max 16384 2024-04-26T17:59:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.405+0000 7f691b985740 1 Processor -- start 2024-04-26T17:59:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.405+0000 7f691b985740 1 -- start start 2024-04-26T17:59:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.405+0000 7f691b985740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706add9f00 0x55706adf0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.405+0000 7f691b985740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706adda820 0x55706addc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f691b985740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706ade50e0 0x55706adfc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f691b985740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f69141bfac0 con 0x55706add9f00 2024-04-26T17:59:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f691b985740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f69141bd330 con 0x55706ade50e0 2024-04-26T17:59:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f691b985740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f69141bb180 con 0x55706adda820 2024-04-26T17:59:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f691259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706add9f00 0x55706adf0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f6911d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706adda820 0x55706addc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f691259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706add9f00 0x55706adf0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50530/0 (socket says 172.21.15.31:50530) 2024-04-26T17:59:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f691259c640 1 -- 172.21.15.31:0/2559121667 learned_addr learned my addr 172.21.15.31:0/2559121667 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f6912d9d640 1 --2- 172.21.15.31:0/2559121667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706ade50e0 0x55706adfc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f6911d9b640 1 -- 172.21.15.31:0/2559121667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706ade50e0 msgr2=0x55706adfc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f6911d9b640 1 --2- 172.21.15.31:0/2559121667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706ade50e0 0x55706adfc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f6911d9b640 1 -- 172.21.15.31:0/2559121667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706add9f00 msgr2=0x55706adf0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f6911d9b640 1 --2- 172.21.15.31:0/2559121667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706add9f00 0x55706adf0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f6911d9b640 1 -- 172.21.15.31:0/2559121667 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55706add9cc0 con 0x55706adda820 2024-04-26T17:59:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f6911d9b640 1 --2- 172.21.15.31:0/2559121667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706adda820 0x55706addc910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f690c0eea20 tx=0x7f690c003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=445c3c97f5ac98a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f68fbfff640 1 -- 172.21.15.31:0/2559121667 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c0f0040 con 0x55706adda820 2024-04-26T17:59:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f68fbfff640 1 -- 172.21.15.31:0/2559121667 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f690c0c5ee0 con 0x55706adda820 2024-04-26T17:59:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.406+0000 7f68fbfff640 1 -- 172.21.15.31:0/2559121667 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c0c5a50 con 0x55706adda820 2024-04-26T17:59:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.407+0000 7f691b985740 1 -- 172.21.15.31:0/2559121667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706adda820 msgr2=0x55706addc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.407+0000 7f691b985740 1 --2- 172.21.15.31:0/2559121667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706adda820 0x55706addc910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f690c0eea20 tx=0x7f690c003f10 comp rx=0 tx=0).stop 2024-04-26T17:59:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.407+0000 7f691b985740 1 -- 172.21.15.31:0/2559121667 shutdown_connections 2024-04-26T17:59:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.407+0000 7f691b985740 1 --2- 172.21.15.31:0/2559121667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706ade50e0 0x55706adfc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.407+0000 7f691b985740 1 --2- 172.21.15.31:0/2559121667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706adda820 0x55706addc910 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.407+0000 7f691b985740 1 --2- 172.21.15.31:0/2559121667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706add9f00 0x55706adf0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.407+0000 7f691b985740 1 -- 172.21.15.31:0/2559121667 >> 172.21.15.31:0/2559121667 conn(0x55706add6410 msgr2=0x55706ade2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.407+0000 7f691b985740 1 -- 172.21.15.31:0/2559121667 shutdown_connections 2024-04-26T17:59:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.407+0000 7f691b985740 1 -- 172.21.15.31:0/2559121667 wait complete. 2024-04-26T17:59:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691b985740 1 -- start start 2024-04-26T17:59:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691b985740 1 Processor -- start 2024-04-26T17:59:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691b985740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706addee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691b985740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706adda820 0x55706addf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691b985740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706ade50e0 0x55706addf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691b985740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f69141bb7f0 con 0x55706ade50e0 2024-04-26T17:59:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691b985740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f69141bcc60 con 0x55706add9f00 2024-04-26T17:59:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691b985740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f69141bbec0 con 0x55706adda820 2024-04-26T17:59:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f6912d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706ade50e0 0x55706addf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f6912d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706ade50e0 0x55706addf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50538/0 (socket says 172.21.15.31:50538) 2024-04-26T17:59:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f6912d9d640 1 -- 172.21.15.31:0/3093639772 learned_addr learned my addr 172.21.15.31:0/3093639772 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691b985740 10 client.0 _ll_register_callbacks cb 0x7f690c02ac50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:59:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691259c640 1 --2- 172.21.15.31:0/3093639772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706addee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f6911d9b640 1 --2- 172.21.15.31:0/3093639772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706adda820 0x55706addf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691259c640 1 -- 172.21.15.31:0/3093639772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706adda820 msgr2=0x55706addf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691259c640 1 --2- 172.21.15.31:0/3093639772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706adda820 0x55706addf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691259c640 1 -- 172.21.15.31:0/3093639772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706ade50e0 msgr2=0x55706addf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691259c640 1 --2- 172.21.15.31:0/3093639772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706ade50e0 0x55706addf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f691259c640 1 -- 172.21.15.31:0/3093639772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55706ae5a140 con 0x55706add9f00 2024-04-26T17:59:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.408+0000 7f6912d9d640 1 --2- 172.21.15.31:0/3093639772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706ade50e0 0x55706addf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:59:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.409+0000 7f691259c640 1 --2- 172.21.15.31:0/3093639772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706addee80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f69080095c0 tx=0x7f690802e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11b8c2dfab2106ef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.409+0000 7f68f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T17:59:23.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.409+0000 7f68f9ffb640 1 -- 172.21.15.31:0/3093639772 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6908048060 con 0x55706add9f00 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.409+0000 7f68f9ffb640 1 -- 172.21.15.31:0/3093639772 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f6908043480 con 0x55706add9f00 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.409+0000 7f691b985740 1 -- 172.21.15.31:0/3093639772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f690c0c5ee0 con 0x55706add9f00 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.409+0000 7f68f9ffb640 1 -- 172.21.15.31:0/3093639772 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6908043680 con 0x55706add9f00 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.409+0000 7f691b985740 10 client.25349 fetch_fsmap learned FSMap version 30 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.409+0000 7f691b985740 1 -- 172.21.15.31:0/3093639772 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55706ae5b410 con 0x55706add9f00 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.409+0000 7f68f9ffb640 1 -- 172.21.15.31:0/3093639772 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f690c0c5ee0 con 0x55706add9f00 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.409+0000 7f68f9ffb640 1 -- 172.21.15.31:0/3093639772 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f6908030400 con 0x55706add9f00 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.410+0000 7f691b985740 10 client.25349 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.410+0000 7f691b985740 10 client.25349 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.410+0000 7f691b985740 20 client.25349 populate_metadata read hostname 'smithi031' 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.410+0000 7f691b985740 1 --2- 172.21.15.31:0/3093639772 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55706adf4e00 0x55706ae5dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.410+0000 7f691b985740 4 client.25349 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.410+0000 7f691b985740 1 -- 172.21.15.31:0/3093639772 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55706ad47ee0 con 0x55706adf4e00 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.410+0000 7f6911d9b640 1 --2- 172.21.15.31:0/3093639772 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55706adf4e00 0x55706ae5dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.411+0000 7f68f9ffb640 10 client.25349 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.411+0000 7f6911d9b640 1 --2- 172.21.15.31:0/3093639772 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55706adf4e00 0x55706ae5dab0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63e52bc325a6666 server_cookie=2d8e647d0896bbd in_seq=0 out_seq=0 2024-04-26T17:59:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.425+0000 7f68f9ffb640 1 -- 172.21.15.31:0/3093639772 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55706ad47ee0 con 0x55706adf4e00 2024-04-26T17:59:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.425+0000 7f68f9ffb640 10 client.25349 handle_command_reply: tid=0 2024-04-26T17:59:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.425+0000 7f691b985740 1 client.25349 shutdown 2024-04-26T17:59:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.426+0000 7f691b985740 1 -- 172.21.15.31:0/3093639772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 msgr2=0x55706addee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.426+0000 7f691b985740 1 --2- 172.21.15.31:0/3093639772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706addee80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f69080095c0 tx=0x7f690802e000 comp rx=0 tx=0).stop 2024-04-26T17:59:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.426+0000 7f691b985740 1 -- 172.21.15.31:0/3093639772 shutdown_connections 2024-04-26T17:59:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.426+0000 7f691b985740 1 --2- 172.21.15.31:0/3093639772 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55706adf4e00 0x55706ae5dab0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.426+0000 7f691b985740 1 --2- 172.21.15.31:0/3093639772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55706ade50e0 0x55706addf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.426+0000 7f68f9ffb640 0 client.25349 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.426+0000 7f691b985740 1 --2- 172.21.15.31:0/3093639772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55706adda820 0x55706addf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.426+0000 7f691b985740 1 --2- 172.21.15.31:0/3093639772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55706add9f00 0x55706addee80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.426+0000 7f691b985740 1 -- 172.21.15.31:0/3093639772 >> 172.21.15.31:0/3093639772 conn(0x55706add6410 msgr2=0x55706ade20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.427+0000 7f691b985740 1 -- 172.21.15.31:0/3093639772 shutdown_connections 2024-04-26T17:59:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.427+0000 7f691b985740 1 -- 172.21.15.31:0/3093639772 wait complete. 2024-04-26T17:59:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.428+0000 7f691b985740 20 client.25349 trim_cache size 0 max 16384 2024-04-26T17:59:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:59:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:59:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:59:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:59:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.429+0000 7f6918c6d640 1 -- 172.21.15.31:0/1138611061 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f68fc0a3f70 msgr2=0x7f68fc0a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.429+0000 7f6918c6d640 1 --2- 172.21.15.31:0/1138611061 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f68fc0a3f70 0x7f68fc0a6430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f69080095f0 tx=0x7f6908005f90 comp rx=0 tx=0).stop 2024-04-26T17:59:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.429+0000 7f6918c6d640 1 -- 172.21.15.31:0/1138611061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6914104400 msgr2=0x7f69141a0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.429+0000 7f6918c6d640 1 --2- 172.21.15.31:0/1138611061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6914104400 0x7f69141a0300 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f690c009d00 tx=0x7f690c003ba0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.430+0000 7f6918c6d640 1 -- 172.21.15.31:0/1138611061 shutdown_connections 2024-04-26T17:59:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.430+0000 7f6918c6d640 1 --2- 172.21.15.31:0/1138611061 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f68fc0a3f70 0x7f68fc0a6430 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.430+0000 7f6918c6d640 1 --2- 172.21.15.31:0/1138611061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f691410cf00 0x7f691410bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.430+0000 7f6918c6d640 1 --2- 172.21.15.31:0/1138611061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6914104400 0x7f69141a0300 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.430+0000 7f6918c6d640 1 --2- 172.21.15.31:0/1138611061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6914103ae0 0x7f691419fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.430+0000 7f6918c6d640 1 -- 172.21.15.31:0/1138611061 >> 172.21.15.31:0/1138611061 conn(0x7f69140fd300 msgr2=0x7f6914101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.430+0000 7f6918c6d640 1 -- 172.21.15.31:0/1138611061 shutdown_connections 2024-04-26T17:59:23.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:23.430+0000 7f6918c6d640 1 -- 172.21.15.31:0/1138611061 wait complete. 2024-04-26T17:59:23.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:23 smithi186 ceph-mon[32797]: pgmap v89: 97 pgs: 97 active+clean; 1.6 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-26T17:59:23.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:23 smithi031 ceph-mon[29164]: pgmap v89: 97 pgs: 97 active+clean; 1.6 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-26T17:59:25.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:25 smithi097 ceph-mon[32419]: pgmap v90: 97 pgs: 97 active+clean; 1.6 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.1 MiB/s wr, 727 op/s 2024-04-26T17:59:25.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:25 smithi186 ceph-mon[32797]: pgmap v90: 97 pgs: 97 active+clean; 1.6 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.1 MiB/s wr, 727 op/s 2024-04-26T17:59:25.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:25 smithi031 ceph-mon[29164]: pgmap v90: 97 pgs: 97 active+clean; 1.6 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.1 MiB/s wr, 727 op/s 2024-04-26T17:59:27.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:27 smithi097 ceph-mon[32419]: pgmap v91: 97 pgs: 97 active+clean; 1.6 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-26T17:59:27.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:27 smithi186 ceph-mon[32797]: pgmap v91: 97 pgs: 97 active+clean; 1.6 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-26T17:59:27.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:27 smithi031 ceph-mon[29164]: pgmap v91: 97 pgs: 97 active+clean; 1.6 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-26T17:59:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 125.0 s, 3275.0 tps, lat 30.699 ms stddev 32.114 2024-04-26T17:59:29.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:29 smithi097 ceph-mon[32419]: pgmap v92: 97 pgs: 97 active+clean; 1.6 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 12 MiB/s wr, 863 op/s 2024-04-26T17:59:29.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:29 smithi186 ceph-mon[32797]: pgmap v92: 97 pgs: 97 active+clean; 1.6 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 12 MiB/s wr, 863 op/s 2024-04-26T17:59:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:29 smithi031 ceph-mon[29164]: pgmap v92: 97 pgs: 97 active+clean; 1.6 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 12 MiB/s wr, 863 op/s 2024-04-26T17:59:31.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:31 smithi097 ceph-mon[32419]: pgmap v93: 97 pgs: 97 active+clean; 1.6 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 728 op/s 2024-04-26T17:59:31.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:31 smithi186 ceph-mon[32797]: pgmap v93: 97 pgs: 97 active+clean; 1.6 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 728 op/s 2024-04-26T17:59:31.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:31 smithi031 ceph-mon[29164]: pgmap v93: 97 pgs: 97 active+clean; 1.6 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 728 op/s 2024-04-26T17:59:33.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 130.0 s, 3792.0 tps, lat 26.338 ms stddev 25.886 2024-04-26T17:59:33.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:33 smithi097 ceph-mon[32419]: pgmap v94: 97 pgs: 97 active+clean; 1.6 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 13 MiB/s wr, 1.04k op/s 2024-04-26T17:59:33.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:33 smithi186 ceph-mon[32797]: pgmap v94: 97 pgs: 97 active+clean; 1.6 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 13 MiB/s wr, 1.04k op/s 2024-04-26T17:59:33.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:33 smithi031 ceph-mon[29164]: pgmap v94: 97 pgs: 97 active+clean; 1.6 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 13 MiB/s wr, 1.04k op/s 2024-04-26T17:59:35.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:35 smithi186 ceph-mon[32797]: pgmap v95: 97 pgs: 97 active+clean; 1.6 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.1 MiB/s wr, 623 op/s 2024-04-26T17:59:35.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:35 smithi031 ceph-mon[29164]: pgmap v95: 97 pgs: 97 active+clean; 1.6 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.1 MiB/s wr, 623 op/s 2024-04-26T17:59:35.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:35 smithi097 ceph-mon[32419]: pgmap v95: 97 pgs: 97 active+clean; 1.6 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.1 MiB/s wr, 623 op/s 2024-04-26T17:59:36.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.569+0000 7fb18a259640 1 -- 172.21.15.31:0/1169655523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184113720 msgr2=0x7fb184115b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:36.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.569+0000 7fb18a259640 1 --2- 172.21.15.31:0/1169655523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184113720 0x7fb184115b10 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009ca0 tx=0x7fb17c02a960 comp rx=0 tx=0).stop 2024-04-26T17:59:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.570+0000 7fb18a259640 1 -- 172.21.15.31:0/1169655523 shutdown_connections 2024-04-26T17:59:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.570+0000 7fb18a259640 1 --2- 172.21.15.31:0/1169655523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184113720 0x7fb184115b10 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.570+0000 7fb18a259640 1 --2- 172.21.15.31:0/1169655523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb184101320 0x7fb1841131e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.570+0000 7fb18a259640 1 --2- 172.21.15.31:0/1169655523 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb184100a00 0x7fb184100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.570+0000 7fb18a259640 1 -- 172.21.15.31:0/1169655523 >> 172.21.15.31:0/1169655523 conn(0x7fb184077930 msgr2=0x7fb1840fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.570+0000 7fb18a259640 1 -- 172.21.15.31:0/1169655523 shutdown_connections 2024-04-26T17:59:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.570+0000 7fb18a259640 1 -- 172.21.15.31:0/1169655523 wait complete. 2024-04-26T17:59:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.571+0000 7fb18a259640 1 Processor -- start 2024-04-26T17:59:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.572+0000 7fb18a259640 1 -- start start 2024-04-26T17:59:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.572+0000 7fb18a259640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb184100a00 0x7fb184112b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.572+0000 7fb18a259640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101320 0x7fb184113050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.572+0000 7fb18a259640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb184113720 0x7fb18410cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.573+0000 7fb18a259640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb1841092c0 con 0x7fb184101320 2024-04-26T17:59:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.573+0000 7fb18a259640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb184109140 con 0x7fb184113720 2024-04-26T17:59:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.573+0000 7fb1837fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb184100a00 0x7fb184112b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.573+0000 7fb1837fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb184100a00 0x7fb184112b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48530/0 (socket says 172.21.15.31:48530) 2024-04-26T17:59:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.573+0000 7fb1837fe640 1 -- 172.21.15.31:0/2247984017 learned_addr learned my addr 172.21.15.31:0/2247984017 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.573+0000 7fb18a259640 1 -- 172.21.15.31:0/2247984017 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb184109440 con 0x7fb184100a00 2024-04-26T17:59:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.573+0000 7fb183fff640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb184113720 0x7fb18410cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.573+0000 7fb182ffd640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101320 0x7fb184113050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.573+0000 7fb1837fe640 1 -- 172.21.15.31:0/2247984017 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb184113720 msgr2=0x7fb18410cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.573+0000 7fb1837fe640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb184113720 0x7fb18410cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.573+0000 7fb1837fe640 1 -- 172.21.15.31:0/2247984017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101320 msgr2=0x7fb184113050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.573+0000 7fb1837fe640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101320 0x7fb184113050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.573+0000 7fb1837fe640 1 -- 172.21.15.31:0/2247984017 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18410d480 con 0x7fb184100a00 2024-04-26T17:59:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.574+0000 7fb182ffd640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101320 0x7fb184113050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:59:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.574+0000 7fb1837fe640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb184100a00 0x7fb184112b10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb17400d950 tx=0x7fb17400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.574+0000 7fb180ff9640 1 -- 172.21.15.31:0/2247984017 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb174017070 con 0x7fb184100a00 2024-04-26T17:59:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.574+0000 7fb180ff9640 1 -- 172.21.15.31:0/2247984017 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb174009c30 con 0x7fb184100a00 2024-04-26T17:59:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.574+0000 7fb18a259640 1 -- 172.21.15.31:0/2247984017 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18410d770 con 0x7fb184100a00 2024-04-26T17:59:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.574+0000 7fb18a259640 1 -- 172.21.15.31:0/2247984017 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1841b47a0 con 0x7fb184100a00 2024-04-26T17:59:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.574+0000 7fb180ff9640 1 -- 172.21.15.31:0/2247984017 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17400f620 con 0x7fb184100a00 2024-04-26T17:59:36.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.578+0000 7fb18cf73740 1 Processor -- start 2024-04-26T17:59:36.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.578+0000 7fb180ff9640 1 -- 172.21.15.31:0/2247984017 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb174010460 con 0x7fb184100a00 2024-04-26T17:59:36.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.578+0000 7fb18cf73740 1 -- start start 2024-04-26T17:59:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb18cf73740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779df00 0x55a7a779e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb18cf73740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779e820 0x55a7a77a8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb18cf73740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a77a90e0 0x55a7a77ab4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb18cf73740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a7a7704460 con 0x55a7a77a90e0 2024-04-26T17:59:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb18cf73740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55a7a76be590 con 0x55a7a779df00 2024-04-26T17:59:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb18cf73740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a7a7700420 con 0x55a7a779e820 2024-04-26T17:59:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb183fff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779df00 0x55a7a779e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb183fff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779df00 0x55a7a779e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:43544/0 (socket says 172.21.15.31:43544) 2024-04-26T17:59:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb183fff640 1 -- 172.21.15.31:0/2721368796 learned_addr learned my addr 172.21.15.31:0/2721368796 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb182ffd640 1 --2- 172.21.15.31:0/2721368796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a77a90e0 0x55a7a77ab4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb1837fe640 1 --2- 172.21.15.31:0/2721368796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779e820 0x55a7a77a8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb182ffd640 1 -- 172.21.15.31:0/2721368796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779e820 msgr2=0x55a7a77a8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb182ffd640 1 --2- 172.21.15.31:0/2721368796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779e820 0x55a7a77a8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb182ffd640 1 -- 172.21.15.31:0/2721368796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779df00 msgr2=0x55a7a779e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb182ffd640 1 --2- 172.21.15.31:0/2721368796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779df00 0x55a7a779e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb182ffd640 1 -- 172.21.15.31:0/2721368796 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7a76cb0d0 con 0x55a7a77a90e0 2024-04-26T17:59:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb180ff9640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fb16c0a3f10 0x7fb16c0a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.579+0000 7fb1837fe640 1 --2- 172.21.15.31:0/2721368796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779e820 0x55a7a77a8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:59:36.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.580+0000 7fb182ffd640 1 --2- 172.21.15.31:0/2721368796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a77a90e0 0x55a7a77ab4d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb18410e3d0 tx=0x7fb17800bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c393b7771fbc2d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:36.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.580+0000 7fb183fff640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fb16c0a3f10 0x7fb16c0a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.580+0000 7fb180ff9640 1 -- 172.21.15.31:0/2247984017 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fb1740d02c0 con 0x7fb184100a00 2024-04-26T17:59:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.580+0000 7fb169ffb640 1 -- 172.21.15.31:0/2721368796 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17800dda0 con 0x55a7a77a90e0 2024-04-26T17:59:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.580+0000 7fb169ffb640 1 -- 172.21.15.31:0/2721368796 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1780054d0 con 0x55a7a77a90e0 2024-04-26T17:59:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.580+0000 7fb169ffb640 1 -- 172.21.15.31:0/2721368796 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17800f680 con 0x55a7a77a90e0 2024-04-26T17:59:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.581+0000 7fb18cf73740 1 -- 172.21.15.31:0/2721368796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a77a90e0 msgr2=0x55a7a77ab4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.581+0000 7fb183fff640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fb16c0a3f10 0x7fb16c0a63d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009620 tx=0x7fb17c005d80 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.581+0000 7fb18cf73740 1 --2- 172.21.15.31:0/2721368796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a77a90e0 0x55a7a77ab4d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb18410e3d0 tx=0x7fb17800bec0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.581+0000 7fb18cf73740 1 -- 172.21.15.31:0/2721368796 shutdown_connections 2024-04-26T17:59:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.581+0000 7fb18cf73740 1 --2- 172.21.15.31:0/2721368796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a77a90e0 0x55a7a77ab4d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.581+0000 7fb18cf73740 1 --2- 172.21.15.31:0/2721368796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779e820 0x55a7a77a8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.581+0000 7fb18cf73740 1 --2- 172.21.15.31:0/2721368796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779df00 0x55a7a779e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.581+0000 7fb18cf73740 1 -- 172.21.15.31:0/2721368796 >> 172.21.15.31:0/2721368796 conn(0x55a7a779a410 msgr2=0x55a7a779a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.581+0000 7fb18cf73740 1 -- 172.21.15.31:0/2721368796 shutdown_connections 2024-04-26T17:59:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.581+0000 7fb18cf73740 1 -- 172.21.15.31:0/2721368796 wait complete. 2024-04-26T17:59:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.581+0000 7fb18cf73740 1 -- start start 2024-04-26T17:59:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.582+0000 7fb18cf73740 1 Processor -- start 2024-04-26T17:59:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.582+0000 7fb18cf73740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779df00 0x55a7a77a6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.582+0000 7fb18cf73740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779e820 0x55a7a77a6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.582+0000 7fb18cf73740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a77a90e0 0x55a7a77a6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.582+0000 7fb18cf73740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a7a77abd10 con 0x55a7a77a90e0 2024-04-26T17:59:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.582+0000 7fb18cf73740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55a7a768bb60 con 0x55a7a779e820 2024-04-26T17:59:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.582+0000 7fb18cf73740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a7a77abe90 con 0x55a7a779df00 2024-04-26T17:59:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.582+0000 7fb18cf73740 10 client.0 _ll_register_callbacks cb 0x55a7a76b02b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:59:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.582+0000 7fb1837fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779df00 0x55a7a77a6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.582+0000 7fb1837fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779df00 0x55a7a77a6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48554/0 (socket says 172.21.15.31:48554) 2024-04-26T17:59:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.583+0000 7fb1837fe640 1 -- 172.21.15.31:0/2866064203 learned_addr learned my addr 172.21.15.31:0/2866064203 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.583+0000 7fb182ffd640 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779e820 0x55a7a77a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.583+0000 7fb183fff640 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a77a90e0 0x55a7a77a6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.583+0000 7fb1837fe640 1 -- 172.21.15.31:0/2866064203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779e820 msgr2=0x55a7a77a6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.583+0000 7fb1837fe640 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779e820 0x55a7a77a6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.583+0000 7fb1837fe640 1 -- 172.21.15.31:0/2866064203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a77a90e0 msgr2=0x55a7a77a6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.583+0000 7fb1837fe640 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a77a90e0 0x55a7a77a6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.583+0000 7fb1837fe640 1 -- 172.21.15.31:0/2866064203 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7a7416210 con 0x55a7a779df00 2024-04-26T17:59:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.583+0000 7fb183fff640 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a77a90e0 0x55a7a77a6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:59:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.583+0000 7fb182ffd640 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779e820 0x55a7a77a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:59:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.583+0000 7fb1837fe640 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779df00 0x55a7a77a6020 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb174010d90 tx=0x7fb1740985e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47cb85b1663fca0d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.583+0000 7fb162ffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T17:59:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.584+0000 7fb162ffd640 1 -- 172.21.15.31:0/2866064203 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb174020070 con 0x55a7a779df00 2024-04-26T17:59:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.584+0000 7fb162ffd640 1 -- 172.21.15.31:0/2866064203 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fb1740a0150 con 0x55a7a779df00 2024-04-26T17:59:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.585+0000 7fb18cf73740 1 -- 172.21.15.31:0/2866064203 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1780054d0 con 0x55a7a779df00 2024-04-26T17:59:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.585+0000 7fb162ffd640 1 -- 172.21.15.31:0/2866064203 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb1740a0600 con 0x55a7a779df00 2024-04-26T17:59:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.585+0000 7fb18cf73740 10 client.25279 fetch_fsmap learned FSMap version 30 2024-04-26T17:59:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.585+0000 7fb18cf73740 1 -- 172.21.15.31:0/2866064203 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a7a74e2620 con 0x55a7a779df00 2024-04-26T17:59:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.586+0000 7fb162ffd640 1 -- 172.21.15.31:0/2866064203 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1780054d0 con 0x55a7a779df00 2024-04-26T17:59:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.586+0000 7fb162ffd640 1 -- 172.21.15.31:0/2866064203 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fb1740981a0 con 0x55a7a779df00 2024-04-26T17:59:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.586+0000 7fb18cf73740 10 client.25279 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T17:59:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.586+0000 7fb18cf73740 10 client.25279 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:59:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.586+0000 7fb18cf73740 20 client.25279 populate_metadata read hostname 'smithi031' 2024-04-26T17:59:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.586+0000 7fb18cf73740 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a7a77ca1e0 0x55a7a77cc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.586+0000 7fb18cf73740 4 client.25279 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:59:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.586+0000 7fb18cf73740 1 -- 172.21.15.31:0/2866064203 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7a7712070 con 0x55a7a77ca1e0 2024-04-26T17:59:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.586+0000 7fb182ffd640 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a7a77ca1e0 0x55a7a77cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.587+0000 7fb182ffd640 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a7a77ca1e0 0x55a7a77cc630 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b7228a719ce3732 server_cookie=443255acf865978f in_seq=0 out_seq=0 2024-04-26T17:59:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.587+0000 7fb162ffd640 10 client.25279 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.587+0000 7fb162ffd640 1 -- 172.21.15.31:0/2866064203 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a7a7712070 con 0x55a7a77ca1e0 2024-04-26T17:59:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.587+0000 7fb162ffd640 10 client.25279 handle_command_reply: tid=0 2024-04-26T17:59:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.587+0000 7fb18cf73740 1 client.25279 shutdown 2024-04-26T17:59:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.587+0000 7fb18cf73740 1 -- 172.21.15.31:0/2866064203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779df00 msgr2=0x55a7a77a6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.587+0000 7fb18cf73740 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779df00 0x55a7a77a6020 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb174010d90 tx=0x7fb1740985e0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.588+0000 7fb18cf73740 1 -- 172.21.15.31:0/2866064203 shutdown_connections 2024-04-26T17:59:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.588+0000 7fb18cf73740 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a7a77ca1e0 0x55a7a77cc630 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.588+0000 7fb18cf73740 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a77a90e0 0x55a7a77a6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.588+0000 7fb18cf73740 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779e820 0x55a7a77a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.588+0000 7fb18cf73740 1 --2- 172.21.15.31:0/2866064203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779df00 0x55a7a77a6020 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.588+0000 7fb18cf73740 1 -- 172.21.15.31:0/2866064203 >> 172.21.15.31:0/2866064203 conn(0x55a7a779a410 msgr2=0x55a7a77a0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.588+0000 7fb162ffd640 0 client.25279 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.588+0000 7fb18cf73740 1 -- 172.21.15.31:0/2866064203 shutdown_connections 2024-04-26T17:59:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.588+0000 7fb18cf73740 1 -- 172.21.15.31:0/2866064203 wait complete. 2024-04-26T17:59:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.588+0000 7fb18cf73740 20 client.25279 trim_cache size 0 max 16384 2024-04-26T17:59:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.609+0000 7fb18cf73740 1 Processor -- start 2024-04-26T17:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.609+0000 7fb18cf73740 1 -- start start 2024-04-26T17:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.609+0000 7fb18cf73740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779df00 0x55a7a77b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.609+0000 7fb18cf73740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a779e820 0x55a7a77a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.609+0000 7fb18cf73740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a77a90e0 0x55a7a7799430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.609+0000 7fb18cf73740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb170010f90 con 0x55a7a779e820 2024-04-26T17:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.610+0000 7fb18cf73740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb17000e800 con 0x55a7a77a90e0 2024-04-26T17:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.610+0000 7fb18cf73740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb17000c650 con 0x55a7a779df00 2024-04-26T17:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.610+0000 7fb182ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a779e820 0x55a7a77a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.610+0000 7fb182ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a779e820 0x55a7a77a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49612/0 (socket says 172.21.15.31:49612) 2024-04-26T17:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.610+0000 7fb182ffd640 1 -- 172.21.15.31:0/441063695 learned_addr learned my addr 172.21.15.31:0/441063695 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.610+0000 7fb1837fe640 1 --2- 172.21.15.31:0/441063695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779df00 0x55a7a77b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.610+0000 7fb183fff640 1 --2- 172.21.15.31:0/441063695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a77a90e0 0x55a7a7799430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.610+0000 7fb182ffd640 1 -- 172.21.15.31:0/441063695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779df00 msgr2=0x55a7a77b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.610+0000 7fb182ffd640 1 --2- 172.21.15.31:0/441063695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779df00 0x55a7a77b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.610+0000 7fb182ffd640 1 -- 172.21.15.31:0/441063695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a77a90e0 msgr2=0x55a7a7799430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.610+0000 7fb182ffd640 1 --2- 172.21.15.31:0/441063695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a77a90e0 0x55a7a7799430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.610+0000 7fb182ffd640 1 -- 172.21.15.31:0/441063695 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7a77a4fd0 con 0x55a7a779e820 2024-04-26T17:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.610+0000 7fb182ffd640 1 --2- 172.21.15.31:0/441063695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a779e820 0x55a7a77a08c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x55a7a779b5a0 tx=0x7fb178005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80fb2abacf7a1941 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.611+0000 7fb1697fa640 1 -- 172.21.15.31:0/441063695 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17801c040 con 0x55a7a779e820 2024-04-26T17:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.611+0000 7fb1697fa640 1 -- 172.21.15.31:0/441063695 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb178005e50 con 0x55a7a779e820 2024-04-26T17:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.611+0000 7fb1697fa640 1 -- 172.21.15.31:0/441063695 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb178004ff0 con 0x55a7a779e820 2024-04-26T17:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.612+0000 7fb18cf73740 1 -- 172.21.15.31:0/441063695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a779e820 msgr2=0x55a7a77a08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.612+0000 7fb18cf73740 1 --2- 172.21.15.31:0/441063695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a779e820 0x55a7a77a08c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x55a7a779b5a0 tx=0x7fb178005780 comp rx=0 tx=0).stop 2024-04-26T17:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.612+0000 7fb18cf73740 1 -- 172.21.15.31:0/441063695 shutdown_connections 2024-04-26T17:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.612+0000 7fb18cf73740 1 --2- 172.21.15.31:0/441063695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a77a90e0 0x55a7a7799430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.612+0000 7fb18cf73740 1 --2- 172.21.15.31:0/441063695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a779e820 0x55a7a77a08c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.612+0000 7fb18cf73740 1 --2- 172.21.15.31:0/441063695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a779df00 0x55a7a77b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.612+0000 7fb18cf73740 1 -- 172.21.15.31:0/441063695 >> 172.21.15.31:0/441063695 conn(0x55a7a779a410 msgr2=0x55a7a7798d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.612+0000 7fb18cf73740 1 -- 172.21.15.31:0/441063695 shutdown_connections 2024-04-26T17:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.612+0000 7fb18cf73740 1 -- 172.21.15.31:0/441063695 wait complete. 2024-04-26T17:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.612+0000 7fb18cf73740 1 -- start start 2024-04-26T17:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.613+0000 7fb18cf73740 1 Processor -- start 2024-04-26T17:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.613+0000 7fb18cf73740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779df00 0x55a7a77a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.613+0000 7fb18cf73740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a779e820 0x55a7a77a32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.613+0000 7fb18cf73740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a77a90e0 0x55a7a77a3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.613+0000 7fb18cf73740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb17000ccc0 con 0x55a7a779e820 2024-04-26T17:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.613+0000 7fb18cf73740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fb17000e130 con 0x55a7a779df00 2024-04-26T17:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.613+0000 7fb18cf73740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb17000d390 con 0x55a7a77a90e0 2024-04-26T17:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.613+0000 7fb18cf73740 10 client.0 _ll_register_callbacks cb 0x7fb17409bc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.613+0000 7fb182ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a779e820 0x55a7a77a32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.613+0000 7fb182ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a779e820 0x55a7a77a32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49614/0 (socket says 172.21.15.31:49614) 2024-04-26T17:59:36.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.613+0000 7fb182ffd640 1 -- 172.21.15.31:0/1220971092 learned_addr learned my addr 172.21.15.31:0/1220971092 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:36.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.613+0000 7fb183fff640 1 --2- 172.21.15.31:0/1220971092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a77a90e0 0x55a7a77a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.614+0000 7fb182ffd640 1 -- 172.21.15.31:0/1220971092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a77a90e0 msgr2=0x55a7a77a3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:36.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.614+0000 7fb182ffd640 1 --2- 172.21.15.31:0/1220971092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a77a90e0 0x55a7a77a3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.614+0000 7fb182ffd640 1 -- 172.21.15.31:0/1220971092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779df00 msgr2=0x55a7a77a2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:36.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.614+0000 7fb182ffd640 1 --2- 172.21.15.31:0/1220971092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779df00 0x55a7a77a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.614+0000 7fb182ffd640 1 -- 172.21.15.31:0/1220971092 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7a779e650 con 0x55a7a779e820 2024-04-26T17:59:36.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.614+0000 7fb183fff640 1 --2- 172.21.15.31:0/1220971092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a77a90e0 0x55a7a77a3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T17:59:36.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.614+0000 7fb182ffd640 1 --2- 172.21.15.31:0/1220971092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a779e820 0x55a7a77a32d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb17800b980 tx=0x7fb17800dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a47d9b02d2a3cc23 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:36.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.614+0000 7fb162ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T17:59:36.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.615+0000 7fb162ffd640 1 -- 172.21.15.31:0/1220971092 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb178004940 con 0x55a7a779e820 2024-04-26T17:59:36.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.615+0000 7fb162ffd640 1 -- 172.21.15.31:0/1220971092 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fb178004cd0 con 0x55a7a779e820 2024-04-26T17:59:36.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.615+0000 7fb18cf73740 1 -- 172.21.15.31:0/1220971092 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb178005e50 con 0x55a7a779e820 2024-04-26T17:59:36.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.615+0000 7fb162ffd640 1 -- 172.21.15.31:0/1220971092 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb178004ed0 con 0x55a7a779e820 2024-04-26T17:59:36.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.615+0000 7fb18cf73740 10 client.15669 fetch_fsmap learned FSMap version 30 2024-04-26T17:59:36.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.615+0000 7fb18cf73740 1 -- 172.21.15.31:0/1220971092 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a7a7830980 con 0x55a7a779e820 2024-04-26T17:59:36.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.616+0000 7fb162ffd640 1 -- 172.21.15.31:0/1220971092 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb178005e50 con 0x55a7a779e820 2024-04-26T17:59:36.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.616+0000 7fb162ffd640 1 -- 172.21.15.31:0/1220971092 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fb17802a040 con 0x55a7a779e820 2024-04-26T17:59:36.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.616+0000 7fb18cf73740 10 client.15669 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T17:59:36.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.616+0000 7fb18cf73740 10 client.15669 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:59:36.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.616+0000 7fb18cf73740 20 client.15669 populate_metadata read hostname 'smithi031' 2024-04-26T17:59:36.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.616+0000 7fb18cf73740 1 --2- 172.21.15.31:0/1220971092 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a7a78310c0 0x55a7a78334e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:36.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.616+0000 7fb18cf73740 4 client.15669 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:59:36.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.616+0000 7fb18cf73740 1 -- 172.21.15.31:0/1220971092 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a7a770bee0 con 0x55a7a78310c0 2024-04-26T17:59:36.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.616+0000 7fb1837fe640 1 --2- 172.21.15.31:0/1220971092 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a7a78310c0 0x55a7a78334e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:36.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.617+0000 7fb1837fe640 1 --2- 172.21.15.31:0/1220971092 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a7a78310c0 0x55a7a78334e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3af0ad24322c79cf server_cookie=a3a3d6682415e71b in_seq=0 out_seq=0 2024-04-26T17:59:36.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.617+0000 7fb162ffd640 10 client.15669 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb162ffd640 1 -- 172.21.15.31:0/1220971092 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a7a770bee0 con 0x55a7a78310c0 2024-04-26T17:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb162ffd640 10 client.15669 handle_command_reply: tid=0 2024-04-26T17:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb18cf73740 1 client.15669 shutdown 2024-04-26T17:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb18cf73740 1 -- 172.21.15.31:0/1220971092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a779e820 msgr2=0x55a7a77a32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb18cf73740 1 --2- 172.21.15.31:0/1220971092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a779e820 0x55a7a77a32d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb17800b980 tx=0x7fb17800dd80 comp rx=0 tx=0).stop 2024-04-26T17:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb18cf73740 1 -- 172.21.15.31:0/1220971092 shutdown_connections 2024-04-26T17:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb18cf73740 1 --2- 172.21.15.31:0/1220971092 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a7a78310c0 0x55a7a78334e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb18cf73740 1 --2- 172.21.15.31:0/1220971092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a7a77a90e0 0x55a7a77a3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb18cf73740 1 --2- 172.21.15.31:0/1220971092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a779e820 0x55a7a77a32d0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb162ffd640 0 client.15669 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb18cf73740 1 --2- 172.21.15.31:0/1220971092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a7a779df00 0x55a7a77a2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb18cf73740 1 -- 172.21.15.31:0/1220971092 >> 172.21.15.31:0/1220971092 conn(0x55a7a779a410 msgr2=0x55a7a77995b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb18cf73740 1 -- 172.21.15.31:0/1220971092 shutdown_connections 2024-04-26T17:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb18cf73740 1 -- 172.21.15.31:0/1220971092 wait complete. 2024-04-26T17:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.628+0000 7fb18cf73740 20 client.15669 trim_cache size 0 max 16384 2024-04-26T17:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.629+0000 7fb18a259640 1 -- 172.21.15.31:0/2247984017 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fb16c0a3f10 msgr2=0x7fb16c0a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.629+0000 7fb18a259640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fb16c0a3f10 0x7fb16c0a63d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009620 tx=0x7fb17c005d80 comp rx=0 tx=0).stop 2024-04-26T17:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.629+0000 7fb18a259640 1 -- 172.21.15.31:0/2247984017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb184100a00 msgr2=0x7fb184112b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.629+0000 7fb18a259640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb184100a00 0x7fb184112b10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb17400d950 tx=0x7fb17400de10 comp rx=0 tx=0).stop 2024-04-26T17:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.629+0000 7fb18a259640 1 -- 172.21.15.31:0/2247984017 shutdown_connections 2024-04-26T17:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.629+0000 7fb18a259640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb184113720 0x7fb18410cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.629+0000 7fb18a259640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fb16c0a3f10 0x7fb16c0a63d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.629+0000 7fb18a259640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101320 0x7fb184113050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.629+0000 7fb18a259640 1 --2- 172.21.15.31:0/2247984017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb184100a00 0x7fb184112b10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.629+0000 7fb18a259640 1 -- 172.21.15.31:0/2247984017 >> 172.21.15.31:0/2247984017 conn(0x7fb184077930 msgr2=0x7fb184113ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.630+0000 7fb18a259640 1 -- 172.21.15.31:0/2247984017 shutdown_connections 2024-04-26T17:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:36.630+0000 7fb18a259640 1 -- 172.21.15.31:0/2247984017 wait complete. 2024-04-26T17:59:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:37 smithi186 ceph-mon[32797]: pgmap v96: 97 pgs: 97 active+clean; 1.7 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 11 MiB/s wr, 850 op/s 2024-04-26T17:59:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:37 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:59:37.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:37 smithi031 ceph-mon[29164]: pgmap v96: 97 pgs: 97 active+clean; 1.7 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 11 MiB/s wr, 850 op/s 2024-04-26T17:59:37.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:37 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:59:37.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:37 smithi097 ceph-mon[32419]: pgmap v96: 97 pgs: 97 active+clean; 1.7 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 11 MiB/s wr, 850 op/s 2024-04-26T17:59:37.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:37 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T17:59:38.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 135.0 s, 3178.6 tps, lat 31.344 ms stddev 32.558 2024-04-26T17:59:38.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:38 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:59:38.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:38 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:59:38.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:38 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:59:38.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:38 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:59:38.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:38 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:59:38.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:38 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:59:38.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:38 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T17:59:38.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:38 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T17:59:38.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:38 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T17:59:39.460 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:39 smithi031 ceph-mon[29164]: pgmap v97: 97 pgs: 97 active+clean; 1.7 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.4 MiB/s wr, 797 op/s 2024-04-26T17:59:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:39 smithi186 ceph-mon[32797]: pgmap v97: 97 pgs: 97 active+clean; 1.7 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.4 MiB/s wr, 797 op/s 2024-04-26T17:59:39.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:39 smithi097 ceph-mon[32419]: pgmap v97: 97 pgs: 97 active+clean; 1.7 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.4 MiB/s wr, 797 op/s 2024-04-26T17:59:41.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:41 smithi186 ceph-mon[32797]: pgmap v98: 97 pgs: 97 active+clean; 1.7 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.6 MiB/s wr, 630 op/s 2024-04-26T17:59:41.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:41 smithi031 ceph-mon[29164]: pgmap v98: 97 pgs: 97 active+clean; 1.7 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.6 MiB/s wr, 630 op/s 2024-04-26T17:59:41.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:41 smithi097 ceph-mon[32419]: pgmap v98: 97 pgs: 97 active+clean; 1.7 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.6 MiB/s wr, 630 op/s 2024-04-26T17:59:43.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 140.0 s, 3722.4 tps, lat 26.973 ms stddev 27.926 2024-04-26T17:59:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:43 smithi186 ceph-mon[32797]: pgmap v99: 97 pgs: 97 active+clean; 1.7 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 11 MiB/s wr, 939 op/s 2024-04-26T17:59:43.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:43 smithi031 ceph-mon[29164]: pgmap v99: 97 pgs: 97 active+clean; 1.7 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 11 MiB/s wr, 939 op/s 2024-04-26T17:59:43.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:43 smithi097 ceph-mon[32419]: pgmap v99: 97 pgs: 97 active+clean; 1.7 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 11 MiB/s wr, 939 op/s 2024-04-26T17:59:45.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:45 smithi186 ceph-mon[32797]: pgmap v100: 97 pgs: 97 active+clean; 1.7 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.5 MiB/s wr, 624 op/s 2024-04-26T17:59:45.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:45 smithi031 ceph-mon[29164]: pgmap v100: 97 pgs: 97 active+clean; 1.7 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.5 MiB/s wr, 624 op/s 2024-04-26T17:59:45.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:45 smithi097 ceph-mon[32419]: pgmap v100: 97 pgs: 97 active+clean; 1.7 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.5 MiB/s wr, 624 op/s 2024-04-26T17:59:47.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:47 smithi186 ceph-mon[32797]: pgmap v101: 97 pgs: 97 active+clean; 1.7 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 807 op/s 2024-04-26T17:59:47.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:47 smithi031 ceph-mon[29164]: pgmap v101: 97 pgs: 97 active+clean; 1.7 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 807 op/s 2024-04-26T17:59:47.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:47 smithi097 ceph-mon[32419]: pgmap v101: 97 pgs: 97 active+clean; 1.7 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 807 op/s 2024-04-26T17:59:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 145.0 s, 3176.8 tps, lat 31.388 ms stddev 31.784 2024-04-26T17:59:49.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:49 smithi186 ceph-mon[32797]: pgmap v102: 97 pgs: 97 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.8 MiB/s wr, 708 op/s 2024-04-26T17:59:49.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:49 smithi031 ceph-mon[29164]: pgmap v102: 97 pgs: 97 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.8 MiB/s wr, 708 op/s 2024-04-26T17:59:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:49 smithi097 ceph-mon[32419]: pgmap v102: 97 pgs: 97 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.8 MiB/s wr, 708 op/s 2024-04-26T17:59:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.748+0000 7f4c8b7f0640 1 -- 172.21.15.31:0/1400768046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c84109e70 msgr2=0x7f4c8410a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.748+0000 7f4c8b7f0640 1 --2- 172.21.15.31:0/1400768046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c84109e70 0x7f4c8410a2f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009900 tx=0x7f4c7402a410 comp rx=0 tx=0).stop 2024-04-26T17:59:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.748+0000 7f4c8b7f0640 1 -- 172.21.15.31:0/1400768046 shutdown_connections 2024-04-26T17:59:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.748+0000 7f4c8b7f0640 1 --2- 172.21.15.31:0/1400768046 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c84114210 0x7f4c84116600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.748+0000 7f4c8b7f0640 1 --2- 172.21.15.31:0/1400768046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c84109e70 0x7f4c8410a2f0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.748+0000 7f4c8b7f0640 1 --2- 172.21.15.31:0/1400768046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c8406a680 0x7f4c84109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.748+0000 7f4c8b7f0640 1 -- 172.21.15.31:0/1400768046 >> 172.21.15.31:0/1400768046 conn(0x7f4c8406de90 msgr2=0x7f4c8406e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.748+0000 7f4c8b7f0640 1 -- 172.21.15.31:0/1400768046 shutdown_connections 2024-04-26T17:59:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.748+0000 7f4c8b7f0640 1 -- 172.21.15.31:0/1400768046 wait complete. 2024-04-26T17:59:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.749+0000 7f4c8b7f0640 1 Processor -- start 2024-04-26T17:59:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.749+0000 7f4c8b7f0640 1 -- start start 2024-04-26T17:59:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c8b7f0640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c8406a680 0x7f4c841a75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c8b7f0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c84109e70 0x7f4c841a7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c8b7f0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c84114210 0x7f4c841a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c8b7f0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8411b410 con 0x7f4c84114210 2024-04-26T17:59:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c8b7f0640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8411b290 con 0x7f4c8406a680 2024-04-26T17:59:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c8b7f0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8411b590 con 0x7f4c84109e70 2024-04-26T17:59:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c88d8b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c84109e70 0x7f4c841a7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c89d8d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c84114210 0x7f4c841a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c88d8b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c84109e70 0x7f4c841a7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:35240/0 (socket says 172.21.15.31:35240) 2024-04-26T17:59:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c88d8b640 1 -- 172.21.15.31:0/2447836337 learned_addr learned my addr 172.21.15.31:0/2447836337 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c8958c640 1 --2- 172.21.15.31:0/2447836337 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c8406a680 0x7f4c841a75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c88d8b640 1 -- 172.21.15.31:0/2447836337 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c8406a680 msgr2=0x7f4c841a75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c88d8b640 1 --2- 172.21.15.31:0/2447836337 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c8406a680 0x7f4c841a75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c88d8b640 1 -- 172.21.15.31:0/2447836337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c84114210 msgr2=0x7f4c841a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c88d8b640 1 --2- 172.21.15.31:0/2447836337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c84114210 0x7f4c841a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c88d8b640 1 -- 172.21.15.31:0/2447836337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c841a1f30 con 0x7f4c84109e70 2024-04-26T17:59:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.750+0000 7f4c89d8d640 1 --2- 172.21.15.31:0/2447836337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c84114210 0x7f4c841a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T17:59:49.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.752+0000 7f4c88d8b640 1 --2- 172.21.15.31:0/2447836337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c84109e70 0x7f4c841a7af0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4c74005ab0 tx=0x7f4c74003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:49.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.752+0000 7f4c7a7fc640 1 -- 172.21.15.31:0/2447836337 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7403f070 con 0x7f4c84109e70 2024-04-26T17:59:49.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.752+0000 7f4c7a7fc640 1 -- 172.21.15.31:0/2447836337 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c74031b60 con 0x7f4c84109e70 2024-04-26T17:59:49.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.752+0000 7f4c8b7f0640 1 -- 172.21.15.31:0/2447836337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c841a21c0 con 0x7f4c84109e70 2024-04-26T17:59:49.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.752+0000 7f4c8b7f0640 1 -- 172.21.15.31:0/2447836337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c84117fb0 con 0x7f4c84109e70 2024-04-26T17:59:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.754+0000 7f4c7a7fc640 1 -- 172.21.15.31:0/2447836337 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74035030 con 0x7f4c84109e70 2024-04-26T17:59:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.758+0000 7f4c8e5e1740 1 Processor -- start 2024-04-26T17:59:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.759+0000 7f4c8e5e1740 1 -- start start 2024-04-26T17:59:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.759+0000 7f4c8e5e1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e6710 0x5608806e6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.759+0000 7f4c8e5e1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806e7030 0x5608806f13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.759+0000 7f4c8e5e1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806f18f0 0x5608806f3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.759+0000 7f4c8e5e1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560880644460 con 0x5608806e7030 2024-04-26T17:59:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.759+0000 7f4c8e5e1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5608805fe590 con 0x5608806f18f0 2024-04-26T17:59:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.759+0000 7f4c8e5e1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560880640420 con 0x5608806e6710 2024-04-26T17:59:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.759+0000 7f4c89d8d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e6710 0x5608806e6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.759+0000 7f4c89d8d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e6710 0x5608806e6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:35252/0 (socket says 172.21.15.31:35252) 2024-04-26T17:59:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.759+0000 7f4c89d8d640 1 -- 172.21.15.31:0/1008420522 learned_addr learned my addr 172.21.15.31:0/1008420522 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.761+0000 7f4c8958c640 1 --2- 172.21.15.31:0/1008420522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806e7030 0x5608806f13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.761+0000 7f4c89d8d640 1 -- 172.21.15.31:0/1008420522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806f18f0 msgr2=0x5608806f3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.761+0000 7f4c89d8d640 1 --2- 172.21.15.31:0/1008420522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806f18f0 0x5608806f3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.761+0000 7f4c89d8d640 1 -- 172.21.15.31:0/1008420522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806e7030 msgr2=0x5608806f13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.761+0000 7f4c89d8d640 1 --2- 172.21.15.31:0/1008420522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806e7030 0x5608806f13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.761+0000 7f4c89d8d640 1 -- 172.21.15.31:0/1008420522 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56088060b0d0 con 0x5608806e6710 2024-04-26T17:59:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.762+0000 7f4c89d8d640 1 --2- 172.21.15.31:0/1008420522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e6710 0x5608806e6af0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4c80011440 tx=0x7f4c80011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cd0c344c8463729 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.763+0000 7f4c7a7fc640 1 -- 172.21.15.31:0/2447836337 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4c74042070 con 0x7f4c84109e70 2024-04-26T17:59:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.764+0000 7f4c6f7fe640 1 -- 172.21.15.31:0/1008420522 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c80004020 con 0x5608806e6710 2024-04-26T17:59:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.764+0000 7f4c6f7fe640 1 -- 172.21.15.31:0/1008420522 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c800041c0 con 0x5608806e6710 2024-04-26T17:59:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.764+0000 7f4c6f7fe640 1 -- 172.21.15.31:0/1008420522 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8000f8b0 con 0x5608806e6710 2024-04-26T17:59:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.764+0000 7f4c7a7fc640 1 --2- 172.21.15.31:0/2447836337 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f4c700a3db0 0x7f4c700a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.764+0000 7f4c7a7fc640 1 -- 172.21.15.31:0/2447836337 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f4c740f0d20 con 0x7f4c84109e70 2024-04-26T17:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.765+0000 7f4c8958c640 1 --2- 172.21.15.31:0/2447836337 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f4c700a3db0 0x7f4c700a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.765+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/1008420522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e6710 msgr2=0x5608806e6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.765+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/1008420522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e6710 0x5608806e6af0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4c80011440 tx=0x7f4c80011900 comp rx=0 tx=0).stop 2024-04-26T17:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.767+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/1008420522 shutdown_connections 2024-04-26T17:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.767+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/1008420522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806f18f0 0x5608806f3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.767+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/1008420522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806e7030 0x5608806f13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.767+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/1008420522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e6710 0x5608806e6af0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.767+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/1008420522 >> 172.21.15.31:0/1008420522 conn(0x5608806e2c20 msgr2=0x5608806e3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.768+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/1008420522 shutdown_connections 2024-04-26T17:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.768+0000 7f4c8958c640 1 --2- 172.21.15.31:0/2447836337 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f4c700a3db0 0x7f4c700a6270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5608806e7cb0 tx=0x7f4c7c012040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.769+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/1008420522 wait complete. 2024-04-26T17:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.770+0000 7f4c8e5e1740 1 -- start start 2024-04-26T17:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.770+0000 7f4c8e5e1740 1 Processor -- start 2024-04-26T17:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.770+0000 7f4c8e5e1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806e6710 0x5608806dfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.771+0000 7f4c8e5e1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e7030 0x5608806e0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.771+0000 7f4c8e5e1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806f18f0 0x5608806f0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.771+0000 7f4c8e5e1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5608806da0e0 con 0x5608806f18f0 2024-04-26T17:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.771+0000 7f4c8e5e1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5608805cbb60 con 0x5608806e6710 2024-04-26T17:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.771+0000 7f4c8e5e1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5608806da260 con 0x5608806e7030 2024-04-26T17:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.771+0000 7f4c8958c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806e6710 0x5608806dfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.771+0000 7f4c8958c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806e6710 0x5608806dfdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:56688/0 (socket says 172.21.15.31:56688) 2024-04-26T17:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.771+0000 7f4c8958c640 1 -- 172.21.15.31:0/1312164918 learned_addr learned my addr 172.21.15.31:0/1312164918 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.771+0000 7f4c8e5e1740 10 client.0 _ll_register_callbacks cb 0x5608805f02b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:59:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.771+0000 7f4c8958c640 1 -- 172.21.15.31:0/1312164918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e7030 msgr2=0x5608806e0310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T17:59:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.771+0000 7f4c8958c640 1 --2- 172.21.15.31:0/1312164918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e7030 0x5608806e0310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.771+0000 7f4c8958c640 1 -- 172.21.15.31:0/1312164918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806f18f0 msgr2=0x5608806f0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.771+0000 7f4c8958c640 1 --2- 172.21.15.31:0/1312164918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806f18f0 0x5608806f0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.771+0000 7f4c8958c640 1 -- 172.21.15.31:0/1312164918 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560880356210 con 0x5608806e6710 2024-04-26T17:59:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.772+0000 7f4c8958c640 1 --2- 172.21.15.31:0/1312164918 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806e6710 0x5608806dfdd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c010aa0 tx=0x7f4c7c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77fcc95a0a1d0556 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.778+0000 7f4c6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T17:59:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.778+0000 7f4c6cff9640 1 -- 172.21.15.31:0/1312164918 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c02b060 con 0x5608806e6710 2024-04-26T17:59:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.778+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/1312164918 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c800041c0 con 0x5608806e6710 2024-04-26T17:59:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.778+0000 7f4c6cff9640 1 -- 172.21.15.31:0/1312164918 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f4c7c016030 con 0x5608806e6710 2024-04-26T17:59:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.778+0000 7f4c6cff9640 1 -- 172.21.15.31:0/1312164918 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4c7c01ec80 con 0x5608806e6710 2024-04-26T17:59:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.779+0000 7f4c8e5e1740 10 client.25373 fetch_fsmap learned FSMap version 30 2024-04-26T17:59:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.779+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/1312164918 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560880422620 con 0x5608806e6710 2024-04-26T17:59:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.779+0000 7f4c6cff9640 1 -- 172.21.15.31:0/1312164918 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c800041c0 con 0x5608806e6710 2024-04-26T17:59:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.779+0000 7f4c6cff9640 1 -- 172.21.15.31:0/1312164918 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f4c7c01a3b0 con 0x5608806e6710 2024-04-26T17:59:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.779+0000 7f4c8e5e1740 10 client.25373 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T17:59:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.780+0000 7f4c8e5e1740 10 client.25373 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:59:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.780+0000 7f4c8e5e1740 20 client.25373 populate_metadata read hostname 'smithi031' 2024-04-26T17:59:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.780+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/1312164918 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56088070a890 0x56088070cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.780+0000 7f4c8e5e1740 4 client.25373 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T17:59:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.780+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/1312164918 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560880652070 con 0x56088070a890 2024-04-26T17:59:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.780+0000 7f4c88d8b640 1 --2- 172.21.15.31:0/1312164918 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56088070a890 0x56088070cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.782+0000 7f4c88d8b640 1 --2- 172.21.15.31:0/1312164918 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56088070a890 0x56088070cd50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5efcdc8659f1fa69 server_cookie=8954b01008956d09 in_seq=0 out_seq=0 2024-04-26T17:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.782+0000 7f4c6cff9640 10 client.25373 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.783+0000 7f4c6cff9640 1 -- 172.21.15.31:0/1312164918 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560880652070 con 0x56088070a890 2024-04-26T17:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.783+0000 7f4c6cff9640 10 client.25373 handle_command_reply: tid=0 2024-04-26T17:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.783+0000 7f4c8e5e1740 1 client.25373 shutdown 2024-04-26T17:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.785+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/1312164918 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806e6710 msgr2=0x5608806dfdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.785+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/1312164918 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806e6710 0x5608806dfdd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c010aa0 tx=0x7f4c7c004ab0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.789+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/1312164918 shutdown_connections 2024-04-26T17:59:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.789+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/1312164918 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56088070a890 0x56088070cd50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.789+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/1312164918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806f18f0 0x5608806f0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.789+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/1312164918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e7030 0x5608806e0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.789+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/1312164918 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806e6710 0x5608806dfdd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.789+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/1312164918 >> 172.21.15.31:0/1312164918 conn(0x5608806e2c20 msgr2=0x5608806d95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.789+0000 7f4c6cff9640 0 client.25373 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.790+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/1312164918 shutdown_connections 2024-04-26T17:59:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.790+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/1312164918 wait complete. 2024-04-26T17:59:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.791+0000 7f4c8e5e1740 20 client.25373 trim_cache size 0 max 16384 2024-04-26T17:59:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8e5e1740 1 Processor -- start 2024-04-26T17:59:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8e5e1740 1 -- start start 2024-04-26T17:59:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8e5e1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806e6710 0x5608806de4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8e5e1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e7030 0x5608806f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8e5e1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806f18f0 0x560880712660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8958c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806e6710 0x5608806de4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8958c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806e6710 0x5608806de4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46686/0 (socket says 172.21.15.31:46686) 2024-04-26T17:59:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8958c640 1 -- 172.21.15.31:0/273157155 learned_addr learned my addr 172.21.15.31:0/273157155 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/273157155 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4c68011080 con 0x5608806e6710 2024-04-26T17:59:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/273157155 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6800e8f0 con 0x5608806f18f0 2024-04-26T17:59:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/273157155 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6800c740 con 0x5608806e7030 2024-04-26T17:59:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c88d8b640 1 --2- 172.21.15.31:0/273157155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e7030 0x5608806f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8958c640 1 -- 172.21.15.31:0/273157155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e7030 msgr2=0x5608806f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8958c640 1 --2- 172.21.15.31:0/273157155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e7030 0x5608806f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c89d8d640 1 --2- 172.21.15.31:0/273157155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806f18f0 0x560880712660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8958c640 1 -- 172.21.15.31:0/273157155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806f18f0 msgr2=0x560880712660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8958c640 1 --2- 172.21.15.31:0/273157155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806f18f0 0x560880712660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c8958c640 1 -- 172.21.15.31:0/273157155 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608806deb90 con 0x5608806e6710 2024-04-26T17:59:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.819+0000 7f4c89d8d640 1 --2- 172.21.15.31:0/273157155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806f18f0 0x560880712660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T17:59:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.820+0000 7f4c8958c640 1 --2- 172.21.15.31:0/273157155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806e6710 0x5608806de4b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c017040 tx=0x7f4c7c009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7417152c5e83a3b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.820+0000 7f4c6effd640 1 -- 172.21.15.31:0/273157155 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c01a590 con 0x5608806e6710 2024-04-26T17:59:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.820+0000 7f4c6effd640 1 -- 172.21.15.31:0/273157155 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c7c01a9a0 con 0x5608806e6710 2024-04-26T17:59:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.820+0000 7f4c6effd640 1 -- 172.21.15.31:0/273157155 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c027c40 con 0x5608806e6710 2024-04-26T17:59:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.821+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/273157155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806e6710 msgr2=0x5608806de4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.821+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/273157155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806e6710 0x5608806de4b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c017040 tx=0x7f4c7c009ef0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.821+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/273157155 shutdown_connections 2024-04-26T17:59:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.821+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/273157155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806f18f0 0x560880712660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.821+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/273157155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e7030 0x5608806f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.821+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/273157155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806e6710 0x5608806de4b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.821+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/273157155 >> 172.21.15.31:0/273157155 conn(0x5608806e2c20 msgr2=0x5608806e89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.821+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/273157155 shutdown_connections 2024-04-26T17:59:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.821+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/273157155 wait complete. 2024-04-26T17:59:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.821+0000 7f4c8e5e1740 1 -- start start 2024-04-26T17:59:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.821+0000 7f4c8e5e1740 1 Processor -- start 2024-04-26T17:59:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8e5e1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e6710 0x5608806f0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8e5e1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806e7030 0x5608806eb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8e5e1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806f18f0 0x5608806ebe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8e5e1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6800cdb0 con 0x5608806f18f0 2024-04-26T17:59:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8e5e1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6800e220 con 0x5608806e7030 2024-04-26T17:59:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8e5e1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6800d480 con 0x5608806e6710 2024-04-26T17:59:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8958c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e6710 0x5608806f0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8e5e1740 10 client.0 _ll_register_callbacks cb 0x7f4c7c009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T17:59:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8958c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e6710 0x5608806f0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:35280/0 (socket says 172.21.15.31:35280) 2024-04-26T17:59:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8958c640 1 -- 172.21.15.31:0/2868237201 learned_addr learned my addr 172.21.15.31:0/2868237201 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T17:59:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c88d8b640 1 --2- 172.21.15.31:0/2868237201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806e7030 0x5608806eb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c89d8d640 1 --2- 172.21.15.31:0/2868237201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806f18f0 0x5608806ebe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8958c640 1 -- 172.21.15.31:0/2868237201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806e7030 msgr2=0x5608806eb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8958c640 1 --2- 172.21.15.31:0/2868237201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806e7030 0x5608806eb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8958c640 1 -- 172.21.15.31:0/2868237201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806f18f0 msgr2=0x5608806ebe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T17:59:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8958c640 1 --2- 172.21.15.31:0/2868237201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806f18f0 0x5608806ebe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.822+0000 7f4c8958c640 1 -- 172.21.15.31:0/2868237201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56088076fed0 con 0x5608806e6710 2024-04-26T17:59:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c8958c640 1 --2- 172.21.15.31:0/2868237201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e6710 0x5608806f0400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c011f20 tx=0x7f4c7c010900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ee4524c441e578c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T17:59:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T17:59:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c6cff9640 1 -- 172.21.15.31:0/2868237201 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c03d020 con 0x5608806e6710 2024-04-26T17:59:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c6cff9640 1 -- 172.21.15.31:0/2868237201 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f4c7c004c00 con 0x5608806e6710 2024-04-26T17:59:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/2868237201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c7c01a9a0 con 0x5608806e6710 2024-04-26T17:59:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c6cff9640 1 -- 172.21.15.31:0/2868237201 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4c7c048050 con 0x5608806e6710 2024-04-26T17:59:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c8e5e1740 10 client.25300 fetch_fsmap learned FSMap version 30 2024-04-26T17:59:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/2868237201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560880770ff0 con 0x5608806e6710 2024-04-26T17:59:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c6cff9640 1 -- 172.21.15.31:0/2868237201 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c7c01a9a0 con 0x5608806e6710 2024-04-26T17:59:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c6cff9640 1 -- 172.21.15.31:0/2868237201 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f4c7c039440 con 0x5608806e6710 2024-04-26T17:59:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c8e5e1740 10 client.25300 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T17:59:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c8e5e1740 10 client.25300 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T17:59:49.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c8e5e1740 20 client.25300 populate_metadata read hostname 'smithi031' 2024-04-26T17:59:49.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/2868237201 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560880771670 0x560880773af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T17:59:49.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c8e5e1740 4 client.25300 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T17:59:49.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.823+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/2868237201 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56088064bee0 con 0x560880771670 2024-04-26T17:59:49.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.824+0000 7f4c88d8b640 1 --2- 172.21.15.31:0/2868237201 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560880771670 0x560880773af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T17:59:49.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.824+0000 7f4c6cff9640 10 client.25300 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:49.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.824+0000 7f4c88d8b640 1 --2- 172.21.15.31:0/2868237201 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560880771670 0x560880773af0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39c2a2814b6bfc7b server_cookie=e4cb6f163caa1b2e in_seq=0 out_seq=0 2024-04-26T17:59:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.838+0000 7f4c6cff9640 1 -- 172.21.15.31:0/2868237201 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56088064bee0 con 0x560880771670 2024-04-26T17:59:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.838+0000 7f4c6cff9640 10 client.25300 handle_command_reply: tid=0 2024-04-26T17:59:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.838+0000 7f4c8e5e1740 1 client.25300 shutdown 2024-04-26T17:59:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.838+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/2868237201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e6710 msgr2=0x5608806f0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.838+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/2868237201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e6710 0x5608806f0400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c011f20 tx=0x7f4c7c010900 comp rx=0 tx=0).stop 2024-04-26T17:59:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.839+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/2868237201 shutdown_connections 2024-04-26T17:59:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.839+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/2868237201 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560880771670 0x560880773af0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.839+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/2868237201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608806f18f0 0x5608806ebe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.839+0000 7f4c6cff9640 0 client.25300 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T17:59:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.839+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/2868237201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5608806e7030 0x5608806eb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.839+0000 7f4c8e5e1740 1 --2- 172.21.15.31:0/2868237201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608806e6710 0x5608806f0400 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.839+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/2868237201 >> 172.21.15.31:0/2868237201 conn(0x5608806e2c20 msgr2=0x5608806d95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.839+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/2868237201 shutdown_connections 2024-04-26T17:59:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.839+0000 7f4c8e5e1740 1 -- 172.21.15.31:0/2868237201 wait complete. 2024-04-26T17:59:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.839+0000 7f4c8e5e1740 20 client.25300 trim_cache size 0 max 16384 2024-04-26T17:59:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T17:59:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T17:59:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T17:59:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T17:59:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.839+0000 7f4c8b7f0640 1 -- 172.21.15.31:0/2447836337 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f4c700a3db0 msgr2=0x7f4c700a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.839+0000 7f4c8b7f0640 1 --2- 172.21.15.31:0/2447836337 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f4c700a3db0 0x7f4c700a6270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5608806e7cb0 tx=0x7f4c7c012040 comp rx=0 tx=0).stop 2024-04-26T17:59:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.839+0000 7f4c8b7f0640 1 -- 172.21.15.31:0/2447836337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c84109e70 msgr2=0x7f4c841a7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T17:59:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.839+0000 7f4c8b7f0640 1 --2- 172.21.15.31:0/2447836337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c84109e70 0x7f4c841a7af0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4c74005ab0 tx=0x7f4c74003c10 comp rx=0 tx=0).stop 2024-04-26T17:59:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.840+0000 7f4c8b7f0640 1 -- 172.21.15.31:0/2447836337 shutdown_connections 2024-04-26T17:59:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.840+0000 7f4c8b7f0640 1 --2- 172.21.15.31:0/2447836337 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f4c700a3db0 0x7f4c700a6270 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.840+0000 7f4c8b7f0640 1 --2- 172.21.15.31:0/2447836337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c84114210 0x7f4c841a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.840+0000 7f4c8b7f0640 1 --2- 172.21.15.31:0/2447836337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c84109e70 0x7f4c841a7af0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.840+0000 7f4c8b7f0640 1 --2- 172.21.15.31:0/2447836337 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c8406a680 0x7f4c841a75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T17:59:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.840+0000 7f4c8b7f0640 1 -- 172.21.15.31:0/2447836337 >> 172.21.15.31:0/2447836337 conn(0x7f4c8406de90 msgr2=0x7f4c84115130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T17:59:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.840+0000 7f4c8b7f0640 1 -- 172.21.15.31:0/2447836337 shutdown_connections 2024-04-26T17:59:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T17:59:49.840+0000 7f4c8b7f0640 1 -- 172.21.15.31:0/2447836337 wait complete. 2024-04-26T17:59:51.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:51 smithi186 ceph-mon[32797]: pgmap v103: 97 pgs: 97 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 619 op/s 2024-04-26T17:59:51.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:51 smithi031 ceph-mon[29164]: pgmap v103: 97 pgs: 97 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 619 op/s 2024-04-26T17:59:51.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:51 smithi097 ceph-mon[32419]: pgmap v103: 97 pgs: 97 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 619 op/s 2024-04-26T17:59:53.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 150.0 s, 4032.6 tps, lat 24.873 ms stddev 25.345 2024-04-26T17:59:53.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:53 smithi186 ceph-mon[32797]: pgmap v104: 97 pgs: 97 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 12 MiB/s wr, 976 op/s 2024-04-26T17:59:53.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:53 smithi031 ceph-mon[29164]: pgmap v104: 97 pgs: 97 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 12 MiB/s wr, 976 op/s 2024-04-26T17:59:53.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:53 smithi097 ceph-mon[32419]: pgmap v104: 97 pgs: 97 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 12 MiB/s wr, 976 op/s 2024-04-26T17:59:55.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:55 smithi186 ceph-mon[32797]: pgmap v105: 97 pgs: 97 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.0 MiB/s wr, 666 op/s 2024-04-26T17:59:55.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:55 smithi031 ceph-mon[29164]: pgmap v105: 97 pgs: 97 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.0 MiB/s wr, 666 op/s 2024-04-26T17:59:55.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:55 smithi097 ceph-mon[32419]: pgmap v105: 97 pgs: 97 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.0 MiB/s wr, 666 op/s 2024-04-26T17:59:57.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:57 smithi186 ceph-mon[32797]: pgmap v106: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 12 MiB/s wr, 909 op/s 2024-04-26T17:59:57.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:57 smithi031 ceph-mon[29164]: pgmap v106: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 12 MiB/s wr, 909 op/s 2024-04-26T17:59:57.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:57 smithi097 ceph-mon[32419]: pgmap v106: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 12 MiB/s wr, 909 op/s 2024-04-26T17:59:58.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 155.0 s, 3190.8 tps, lat 31.220 ms stddev 32.780 2024-04-26T17:59:58.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:58 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:59:58.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:58 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:59:58.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:58 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T17:59:59.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 17:59:59 smithi186 ceph-mon[32797]: pgmap v107: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 800 op/s 2024-04-26T17:59:59.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 17:59:59 smithi031 ceph-mon[29164]: pgmap v107: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 800 op/s 2024-04-26T17:59:59.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 17:59:59 smithi097 ceph-mon[32419]: pgmap v107: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 800 op/s 2024-04-26T18:00:00.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:00 smithi186 ceph-mon[32797]: overall HEALTH_OK 2024-04-26T18:00:00.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:00 smithi031 ceph-mon[29164]: overall HEALTH_OK 2024-04-26T18:00:00.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:00 smithi097 ceph-mon[32419]: overall HEALTH_OK 2024-04-26T18:00:01.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:01 smithi186 ceph-mon[32797]: pgmap v108: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.5 MiB/s wr, 672 op/s 2024-04-26T18:00:01.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:01 smithi031 ceph-mon[29164]: pgmap v108: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.5 MiB/s wr, 672 op/s 2024-04-26T18:00:01.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:01 smithi097 ceph-mon[32419]: pgmap v108: 97 pgs: 97 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.5 MiB/s wr, 672 op/s 2024-04-26T18:00:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.947+0000 7f4a5a9e4640 1 -- 172.21.15.31:0/2685493077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a54107820 msgr2=0x7f4a54107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.947+0000 7f4a5a9e4640 1 --2- 172.21.15.31:0/2685493077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a54107820 0x7f4a54107c00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009930 tx=0x7f4a4402a420 comp rx=0 tx=0).stop 2024-04-26T18:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.947+0000 7f4a5a9e4640 1 -- 172.21.15.31:0/2685493077 shutdown_connections 2024-04-26T18:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.947+0000 7f4a5a9e4640 1 --2- 172.21.15.31:0/2685493077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a54102220 0x7f4a5410e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.947+0000 7f4a5a9e4640 1 --2- 172.21.15.31:0/2685493077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a54101860 0x7f4a54101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.947+0000 7f4a5a9e4640 1 --2- 172.21.15.31:0/2685493077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a54107820 0x7f4a54107c00 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.947+0000 7f4a5a9e4640 1 -- 172.21.15.31:0/2685493077 >> 172.21.15.31:0/2685493077 conn(0x7f4a540789b0 msgr2=0x7f4a54078dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.947+0000 7f4a5a9e4640 1 -- 172.21.15.31:0/2685493077 shutdown_connections 2024-04-26T18:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.947+0000 7f4a5a9e4640 1 -- 172.21.15.31:0/2685493077 wait complete. 2024-04-26T18:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.947+0000 7f4a5a9e4640 1 Processor -- start 2024-04-26T18:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.947+0000 7f4a5a9e4640 1 -- start start 2024-04-26T18:00:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a5a9e4640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a54101860 0x7f4a5419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a5a9e4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a54102220 0x7f4a5419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a5a9e4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a54107820 0x7f4a5419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a5a9e4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4a54111ae0 con 0x7f4a54102220 2024-04-26T18:00:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a5a9e4640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4a54111960 con 0x7f4a54101860 2024-04-26T18:00:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a5a9e4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a54111c60 con 0x7f4a54107820 2024-04-26T18:00:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a537fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a54102220 0x7f4a5419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a537fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a54102220 0x7f4a5419b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49794/0 (socket says 172.21.15.31:49794) 2024-04-26T18:00:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a537fe640 1 -- 172.21.15.31:0/928357640 learned_addr learned my addr 172.21.15.31:0/928357640 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a53fff640 1 --2- 172.21.15.31:0/928357640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a54101860 0x7f4a5419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a58f81640 1 --2- 172.21.15.31:0/928357640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a54107820 0x7f4a5419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a537fe640 1 -- 172.21.15.31:0/928357640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a54107820 msgr2=0x7f4a5419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a537fe640 1 --2- 172.21.15.31:0/928357640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a54107820 0x7f4a5419fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a537fe640 1 -- 172.21.15.31:0/928357640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a54101860 msgr2=0x7f4a5419b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a537fe640 1 --2- 172.21.15.31:0/928357640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a54101860 0x7f4a5419b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a537fe640 1 -- 172.21.15.31:0/928357640 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a541a0300 con 0x7f4a54102220 2024-04-26T18:00:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.948+0000 7f4a537fe640 1 --2- 172.21.15.31:0/928357640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a54102220 0x7f4a5419b850 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800e9f0 tx=0x7f4a4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.949+0000 7f4a517fa640 1 -- 172.21.15.31:0/928357640 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4800cd80 con 0x7f4a54102220 2024-04-26T18:00:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.949+0000 7f4a517fa640 1 -- 172.21.15.31:0/928357640 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a480054d0 con 0x7f4a54102220 2024-04-26T18:00:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.949+0000 7f4a5a9e4640 1 -- 172.21.15.31:0/928357640 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a541a05f0 con 0x7f4a54102220 2024-04-26T18:00:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.949+0000 7f4a517fa640 1 -- 172.21.15.31:0/928357640 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48010690 con 0x7f4a54102220 2024-04-26T18:00:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.949+0000 7f4a5a9e4640 1 -- 172.21.15.31:0/928357640 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a541a7e80 con 0x7f4a54102220 2024-04-26T18:00:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.951+0000 7f4a517fa640 1 -- 172.21.15.31:0/928357640 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4a48004e90 con 0x7f4a54102220 2024-04-26T18:00:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.951+0000 7f4a5d7d4740 1 Processor -- start 2024-04-26T18:00:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.951+0000 7f4a5d7d4740 1 -- start start 2024-04-26T18:00:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a5d7d4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e23f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a5d7d4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23f820 0x55700e249ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a5d7d4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 0x55700e24c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a5d7d4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55700e1a5460 con 0x55700e23f820 2024-04-26T18:00:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a5d7d4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55700e15f590 con 0x55700e23ef00 2024-04-26T18:00:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a5d7d4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55700e1a1420 con 0x55700e24a0e0 2024-04-26T18:00:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a517fa640 1 --2- 172.21.15.31:0/928357640 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f4a3c0a3dc0 0x7f4a3c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a537fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 0x55700e24c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a53fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23f820 0x55700e249ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a58f81640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e23f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a53fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23f820 0x55700e249ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49810/0 (socket says 172.21.15.31:49810) 2024-04-26T18:00:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a537fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 0x55700e24c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:44258/0 (socket says 172.21.15.31:44258) 2024-04-26T18:00:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a58f81640 1 --2- 172.21.15.31:0/493336355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e23f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:39716/0 (socket says 172.21.15.31:39716) 2024-04-26T18:00:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a53fff640 1 -- 172.21.15.31:0/493336355 learned_addr learned my addr 172.21.15.31:0/493336355 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a517fa640 1 -- 172.21.15.31:0/928357640 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f4a480d0c00 con 0x7f4a54102220 2024-04-26T18:00:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a58f81640 1 --2- 172.21.15.31:0/928357640 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f4a3c0a3dc0 0x7f4a3c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a53fff640 1 -- 172.21.15.31:0/493336355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 msgr2=0x55700e24c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a53fff640 1 --2- 172.21.15.31:0/493336355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 0x55700e24c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a53fff640 1 -- 172.21.15.31:0/493336355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 msgr2=0x55700e23f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a53fff640 1 --2- 172.21.15.31:0/493336355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e23f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a53fff640 1 -- 172.21.15.31:0/493336355 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55700e16c0d0 con 0x55700e23f820 2024-04-26T18:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.952+0000 7f4a53fff640 1 --2- 172.21.15.31:0/493336355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23f820 0x55700e249ba0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4a4402a940 tx=0x7f4a44005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69aa0c030f4eb67f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.953+0000 7f4a58f81640 1 --2- 172.21.15.31:0/928357640 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f4a3c0a3dc0 0x7f4a3c0a6280 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c005fd0 tx=0x7f4a4c009500 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.953+0000 7f4a3a7fc640 1 -- 172.21.15.31:0/493336355 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44040070 con 0x55700e23f820 2024-04-26T18:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.953+0000 7f4a3a7fc640 1 -- 172.21.15.31:0/493336355 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a44004030 con 0x55700e23f820 2024-04-26T18:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.953+0000 7f4a3a7fc640 1 -- 172.21.15.31:0/493336355 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44035040 con 0x55700e23f820 2024-04-26T18:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.953+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/493336355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23f820 msgr2=0x55700e249ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.953+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/493336355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23f820 0x55700e249ba0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4a4402a940 tx=0x7f4a44005be0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.953+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/493336355 shutdown_connections 2024-04-26T18:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.954+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/493336355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 0x55700e24c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.954+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/493336355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23f820 0x55700e249ba0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.954+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/493336355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e23f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.954+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/493336355 >> 172.21.15.31:0/493336355 conn(0x55700e23b410 msgr2=0x55700e23b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.954+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/493336355 shutdown_connections 2024-04-26T18:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.954+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/493336355 wait complete. 2024-04-26T18:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.954+0000 7f4a5d7d4740 1 -- start start 2024-04-26T18:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.954+0000 7f4a5d7d4740 1 Processor -- start 2024-04-26T18:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.954+0000 7f4a5d7d4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e246fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.954+0000 7f4a5d7d4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e23f820 0x55700e2474f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a5d7d4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e24a0e0 0x55700e247a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a5d7d4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55700e24ccb0 con 0x55700e24a0e0 2024-04-26T18:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a5d7d4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55700e12cb60 con 0x55700e23ef00 2024-04-26T18:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a5d7d4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55700e24ce30 con 0x55700e23f820 2024-04-26T18:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a5d7d4740 10 client.0 _ll_register_callbacks cb 0x55700e1512b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a53fff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e246fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a58f81640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e24a0e0 0x55700e247a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a537fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e23f820 0x55700e2474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a58f81640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e24a0e0 0x55700e247a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49814/0 (socket says 172.21.15.31:49814) 2024-04-26T18:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a537fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e23f820 0x55700e2474f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:44274/0 (socket says 172.21.15.31:44274) 2024-04-26T18:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a53fff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e246fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:39730/0 (socket says 172.21.15.31:39730) 2024-04-26T18:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a58f81640 1 -- 172.21.15.31:0/3241836759 learned_addr learned my addr 172.21.15.31:0/3241836759 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a58f81640 1 -- 172.21.15.31:0/3241836759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e23f820 msgr2=0x55700e2474f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a58f81640 1 --2- 172.21.15.31:0/3241836759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e23f820 0x55700e2474f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a58f81640 1 -- 172.21.15.31:0/3241836759 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 msgr2=0x55700e246fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a58f81640 1 --2- 172.21.15.31:0/3241836759 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e246fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a58f81640 1 -- 172.21.15.31:0/3241836759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55700deb7210 con 0x55700e24a0e0 2024-04-26T18:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a537fe640 1 --2- 172.21.15.31:0/3241836759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e23f820 0x55700e2474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a58f81640 1 --2- 172.21.15.31:0/3241836759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e24a0e0 0x55700e247a30 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00ff20 tx=0x7f4a4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a9f16037a7a68a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a337fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a337fe640 1 -- 172.21.15.31:0/3241836759 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c02e060 con 0x55700e24a0e0 2024-04-26T18:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a337fe640 1 -- 172.21.15.31:0/3241836759 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f4a4c021cc0 con 0x55700e24a0e0 2024-04-26T18:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.955+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/3241836759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a44004030 con 0x55700e24a0e0 2024-04-26T18:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.956+0000 7f4a337fe640 1 -- 172.21.15.31:0/3241836759 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4a4c033460 con 0x55700e24a0e0 2024-04-26T18:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.956+0000 7f4a5d7d4740 10 client.15696 fetch_fsmap learned FSMap version 31 2024-04-26T18:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.956+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/3241836759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55700df83620 con 0x55700e24a0e0 2024-04-26T18:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.956+0000 7f4a337fe640 1 -- 172.21.15.31:0/3241836759 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a44004030 con 0x55700e24a0e0 2024-04-26T18:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.956+0000 7f4a337fe640 1 -- 172.21.15.31:0/3241836759 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 31) ==== 8350+0+0 (secure 0 0 0) 0x7f4a4c01d070 con 0x55700e24a0e0 2024-04-26T18:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.956+0000 7f4a5d7d4740 10 client.15696 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T18:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.956+0000 7f4a5d7d4740 10 client.15696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.956+0000 7f4a5d7d4740 20 client.15696 populate_metadata read hostname 'smithi031' 2024-04-26T18:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.956+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/3241836759 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55700e25a1a0 0x55700e25c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.956+0000 7f4a5d7d4740 4 client.15696 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.956+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/3241836759 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55700e1b3070 con 0x55700e25a1a0 2024-04-26T18:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.956+0000 7f4a53fff640 1 --2- 172.21.15.31:0/3241836759 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55700e25a1a0 0x55700e25c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.957+0000 7f4a53fff640 1 --2- 172.21.15.31:0/3241836759 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55700e25a1a0 0x55700e25c5d0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d459ee79e70ac793 server_cookie=c8a10b6ce76972cb in_seq=0 out_seq=0 2024-04-26T18:00:02.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.957+0000 7f4a337fe640 10 client.15696 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.957+0000 7f4a337fe640 1 -- 172.21.15.31:0/3241836759 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55700e1b3070 con 0x55700e25a1a0 2024-04-26T18:00:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.957+0000 7f4a337fe640 10 client.15696 handle_command_reply: tid=0 2024-04-26T18:00:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.958+0000 7f4a5d7d4740 1 client.15696 shutdown 2024-04-26T18:00:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.958+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/3241836759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e24a0e0 msgr2=0x55700e247a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.958+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/3241836759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e24a0e0 0x55700e247a30 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00ff20 tx=0x7f4a4c004a40 comp rx=0 tx=0).stop 2024-04-26T18:00:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.958+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/3241836759 shutdown_connections 2024-04-26T18:00:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.958+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/3241836759 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55700e25a1a0 0x55700e25c5d0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.958+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/3241836759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e24a0e0 0x55700e247a30 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.958+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/3241836759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e23f820 0x55700e2474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.958+0000 7f4a337fe640 0 client.15696 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.958+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/3241836759 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e246fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.958+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/3241836759 >> 172.21.15.31:0/3241836759 conn(0x55700e23b410 msgr2=0x55700e241190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.958+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/3241836759 shutdown_connections 2024-04-26T18:00:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.958+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/3241836759 wait complete. 2024-04-26T18:00:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.958+0000 7f4a5d7d4740 20 client.15696 trim_cache size 0 max 16384 2024-04-26T18:00:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a5d7d4740 1 Processor -- start 2024-04-26T18:00:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a5d7d4740 1 -- start start 2024-04-26T18:00:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a5d7d4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23ef00 0x55700e2418c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a5d7d4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23f820 0x55700e23b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a5d7d4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 0x55700e261ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a5d7d4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4a541b2f10 con 0x55700e23ef00 2024-04-26T18:00:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a5d7d4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4a541b07b0 con 0x55700e23f820 2024-04-26T18:00:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a5d7d4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a541ae600 con 0x55700e24a0e0 2024-04-26T18:00:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a53fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23ef00 0x55700e2418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a537fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23f820 0x55700e23b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a58f81640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 0x55700e261ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a53fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23ef00 0x55700e2418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49820/0 (socket says 172.21.15.31:49820) 2024-04-26T18:00:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a58f81640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 0x55700e261ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:44278/0 (socket says 172.21.15.31:44278) 2024-04-26T18:00:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a53fff640 1 -- 172.21.15.31:0/3122049211 learned_addr learned my addr 172.21.15.31:0/3122049211 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a53fff640 1 -- 172.21.15.31:0/3122049211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 msgr2=0x55700e261ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a53fff640 1 --2- 172.21.15.31:0/3122049211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 0x55700e261ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a53fff640 1 -- 172.21.15.31:0/3122049211 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23f820 msgr2=0x55700e23b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a53fff640 1 --2- 172.21.15.31:0/3122049211 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23f820 0x55700e23b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.974+0000 7f4a53fff640 1 -- 172.21.15.31:0/3122049211 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55700e245fa0 con 0x55700e23ef00 2024-04-26T18:00:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.975+0000 7f4a53fff640 1 --2- 172.21.15.31:0/3122049211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23ef00 0x55700e2418c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4a4402a940 tx=0x7f4a44031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d3848961fa2774 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.975+0000 7f4a39ffb640 1 -- 172.21.15.31:0/3122049211 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44040040 con 0x55700e23ef00 2024-04-26T18:00:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.975+0000 7f4a39ffb640 1 -- 172.21.15.31:0/3122049211 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a44055070 con 0x55700e23ef00 2024-04-26T18:00:02.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.975+0000 7f4a39ffb640 1 -- 172.21.15.31:0/3122049211 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44035040 con 0x55700e23ef00 2024-04-26T18:00:02.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.976+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/3122049211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23ef00 msgr2=0x55700e2418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:02.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.976+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/3122049211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23ef00 0x55700e2418c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4a4402a940 tx=0x7f4a44031400 comp rx=0 tx=0).stop 2024-04-26T18:00:02.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.976+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/3122049211 shutdown_connections 2024-04-26T18:00:02.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.976+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/3122049211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 0x55700e261ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.976+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/3122049211 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23f820 0x55700e23b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.976+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/3122049211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23ef00 0x55700e2418c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.976+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/3122049211 >> 172.21.15.31:0/3122049211 conn(0x55700e23b410 msgr2=0x55700e23a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.976+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/3122049211 shutdown_connections 2024-04-26T18:00:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.976+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/3122049211 wait complete. 2024-04-26T18:00:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.976+0000 7f4a5d7d4740 1 -- start start 2024-04-26T18:00:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a5d7d4740 1 Processor -- start 2024-04-26T18:00:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a5d7d4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e2524f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a5d7d4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23f820 0x55700e243a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a5d7d4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 0x55700e243ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a5d7d4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4a541aec70 con 0x55700e23f820 2024-04-26T18:00:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a5d7d4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f4a541b00e0 con 0x55700e23ef00 2024-04-26T18:00:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a5d7d4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a541af340 con 0x55700e24a0e0 2024-04-26T18:00:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a5d7d4740 10 client.0 _ll_register_callbacks cb 0x7f4a4c00fdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:00:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a537fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23f820 0x55700e243a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a53fff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e2524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a537fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23f820 0x55700e243a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49830/0 (socket says 172.21.15.31:49830) 2024-04-26T18:00:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a58f81640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 0x55700e243ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a537fe640 1 -- 172.21.15.31:0/1449005425 learned_addr learned my addr 172.21.15.31:0/1449005425 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a537fe640 1 -- 172.21.15.31:0/1449005425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 msgr2=0x55700e243ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a537fe640 1 --2- 172.21.15.31:0/1449005425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 0x55700e243ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a537fe640 1 -- 172.21.15.31:0/1449005425 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 msgr2=0x55700e2524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a537fe640 1 --2- 172.21.15.31:0/1449005425 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e2524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a537fe640 1 -- 172.21.15.31:0/1449005425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55700e244820 con 0x55700e23f820 2024-04-26T18:00:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a53fff640 1 --2- 172.21.15.31:0/1449005425 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e2524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:00:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a537fe640 1 --2- 172.21.15.31:0/1449005425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23f820 0x55700e243a90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4a48004a40 tx=0x7f4a4806b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142e82d922673414 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.977+0000 7f4a337fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a337fe640 1 -- 172.21.15.31:0/1449005425 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a480a5570 con 0x55700e23f820 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a337fe640 1 -- 172.21.15.31:0/1449005425 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f4a4806b1c0 con 0x55700e23f820 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/1449005425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a44055070 con 0x55700e23f820 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a337fe640 1 -- 172.21.15.31:0/1449005425 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4a4809cb90 con 0x55700e23f820 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a5d7d4740 10 client.15702 fetch_fsmap learned FSMap version 31 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/1449005425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55700e2c07f0 con 0x55700e23f820 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a337fe640 1 -- 172.21.15.31:0/1449005425 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a44055070 con 0x55700e23f820 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a337fe640 1 -- 172.21.15.31:0/1449005425 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 31) ==== 8350+0+0 (secure 0 0 0) 0x7f4a480988e0 con 0x55700e23f820 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a5d7d4740 10 client.15702 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a5d7d4740 10 client.15702 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a5d7d4740 20 client.15702 populate_metadata read hostname 'smithi031' 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/1449005425 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55700e2c0fa0 0x55700e2c33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a5d7d4740 4 client.15702 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/1449005425 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55700e1acee0 con 0x55700e2c0fa0 2024-04-26T18:00:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.978+0000 7f4a53fff640 1 --2- 172.21.15.31:0/1449005425 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55700e2c0fa0 0x55700e2c33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.979+0000 7f4a337fe640 10 client.15702 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.979+0000 7f4a53fff640 1 --2- 172.21.15.31:0/1449005425 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55700e2c0fa0 0x55700e2c33c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87926e571eccc0c1 server_cookie=ca49f95d54000bde in_seq=0 out_seq=0 2024-04-26T18:00:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a337fe640 1 -- 172.21.15.31:0/1449005425 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55700e1acee0 con 0x55700e2c0fa0 2024-04-26T18:00:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a337fe640 10 client.15702 handle_command_reply: tid=0 2024-04-26T18:00:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a5d7d4740 1 client.15702 shutdown 2024-04-26T18:00:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/1449005425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23f820 msgr2=0x55700e243a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/1449005425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23f820 0x55700e243a90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4a48004a40 tx=0x7f4a4806b600 comp rx=0 tx=0).stop 2024-04-26T18:00:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/1449005425 shutdown_connections 2024-04-26T18:00:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/1449005425 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55700e2c0fa0 0x55700e2c33c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/1449005425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55700e24a0e0 0x55700e243ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/1449005425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55700e23f820 0x55700e243a90 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a5d7d4740 1 --2- 172.21.15.31:0/1449005425 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55700e23ef00 0x55700e2524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a337fe640 0 client.15702 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/1449005425 >> 172.21.15.31:0/1449005425 conn(0x55700e23b410 msgr2=0x55700e23a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/1449005425 shutdown_connections 2024-04-26T18:00:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a5d7d4740 1 -- 172.21.15.31:0/1449005425 wait complete. 2024-04-26T18:00:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.992+0000 7f4a5d7d4740 20 client.15702 trim_cache size 0 max 16384 2024-04-26T18:00:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:00:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:00:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:00:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:00:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.993+0000 7f4a5a9e4640 1 -- 172.21.15.31:0/928357640 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f4a3c0a3dc0 msgr2=0x7f4a3c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.993+0000 7f4a5a9e4640 1 --2- 172.21.15.31:0/928357640 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f4a3c0a3dc0 0x7f4a3c0a6280 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c005fd0 tx=0x7f4a4c009500 comp rx=0 tx=0).stop 2024-04-26T18:00:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.993+0000 7f4a5a9e4640 1 -- 172.21.15.31:0/928357640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a54102220 msgr2=0x7f4a5419b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.993+0000 7f4a5a9e4640 1 --2- 172.21.15.31:0/928357640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a54102220 0x7f4a5419b850 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800e9f0 tx=0x7f4a4800edf0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.993+0000 7f4a5a9e4640 1 -- 172.21.15.31:0/928357640 shutdown_connections 2024-04-26T18:00:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.993+0000 7f4a5a9e4640 1 --2- 172.21.15.31:0/928357640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a54107820 0x7f4a5419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.993+0000 7f4a5a9e4640 1 --2- 172.21.15.31:0/928357640 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f4a3c0a3dc0 0x7f4a3c0a6280 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.993+0000 7f4a5a9e4640 1 --2- 172.21.15.31:0/928357640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a54102220 0x7f4a5419b850 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.993+0000 7f4a5a9e4640 1 --2- 172.21.15.31:0/928357640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a54101860 0x7f4a5419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.993+0000 7f4a5a9e4640 1 -- 172.21.15.31:0/928357640 >> 172.21.15.31:0/928357640 conn(0x7f4a540789b0 msgr2=0x7f4a5406a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.993+0000 7f4a5a9e4640 1 -- 172.21.15.31:0/928357640 shutdown_connections 2024-04-26T18:00:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:02.993+0000 7f4a5a9e4640 1 -- 172.21.15.31:0/928357640 wait complete. 2024-04-26T18:00:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 160.0 s, 1937.4 tps, lat 50.520 ms stddev 83.572 2024-04-26T18:00:03.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:03 smithi186 ceph-mon[32797]: pgmap v109: 97 pgs: 97 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 992 op/s 2024-04-26T18:00:03.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:03 smithi031 ceph-mon[29164]: pgmap v109: 97 pgs: 97 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 992 op/s 2024-04-26T18:00:03.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:03 smithi097 ceph-mon[32419]: pgmap v109: 97 pgs: 97 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 992 op/s 2024-04-26T18:00:05.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:05 smithi186 ceph-mon[32797]: pgmap v110: 97 pgs: 97 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.9 MiB/s wr, 635 op/s 2024-04-26T18:00:05.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:05 smithi031 ceph-mon[29164]: pgmap v110: 97 pgs: 97 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.9 MiB/s wr, 635 op/s 2024-04-26T18:00:05.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:05 smithi097 ceph-mon[32419]: pgmap v110: 97 pgs: 97 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.9 MiB/s wr, 635 op/s 2024-04-26T18:00:07.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:07 smithi186 ceph-mon[32797]: pgmap v111: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 19 MiB/s wr, 984 op/s 2024-04-26T18:00:07.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:07 smithi031 ceph-mon[29164]: pgmap v111: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 19 MiB/s wr, 984 op/s 2024-04-26T18:00:07.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:07 smithi097 ceph-mon[32419]: pgmap v111: 97 pgs: 97 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 19 MiB/s wr, 984 op/s 2024-04-26T18:00:08.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 165.0 s, 1890.0 tps, lat 54.265 ms stddev 114.810 2024-04-26T18:00:08.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:08 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:00:08.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:08 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:00:08.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:08 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:00:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:09 smithi186 ceph-mon[32797]: pgmap v112: 97 pgs: 97 active+clean; 2.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 37 MiB/s wr, 1.43k op/s 2024-04-26T18:00:09.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:09 smithi031 ceph-mon[29164]: pgmap v112: 97 pgs: 97 active+clean; 2.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 37 MiB/s wr, 1.43k op/s 2024-04-26T18:00:09.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:09 smithi097 ceph-mon[32419]: pgmap v112: 97 pgs: 97 active+clean; 2.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 37 MiB/s wr, 1.43k op/s 2024-04-26T18:00:11.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:11 smithi186 ceph-mon[32797]: pgmap v113: 97 pgs: 97 active+clean; 2.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 35 MiB/s wr, 1.36k op/s 2024-04-26T18:00:11.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:11 smithi031 ceph-mon[29164]: pgmap v113: 97 pgs: 97 active+clean; 2.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 35 MiB/s wr, 1.36k op/s 2024-04-26T18:00:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:11 smithi097 ceph-mon[32419]: pgmap v113: 97 pgs: 97 active+clean; 2.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 35 MiB/s wr, 1.36k op/s 2024-04-26T18:00:13.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 170.0 s, 3251.6 tps, lat 30.638 ms stddev 32.356 2024-04-26T18:00:13.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:13 smithi186 ceph-mon[32797]: pgmap v114: 97 pgs: 97 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 41 MiB/s wr, 1.72k op/s 2024-04-26T18:00:13.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:13 smithi031 ceph-mon[29164]: pgmap v114: 97 pgs: 97 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 41 MiB/s wr, 1.72k op/s 2024-04-26T18:00:13.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:13 smithi097 ceph-mon[32419]: pgmap v114: 97 pgs: 97 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 41 MiB/s wr, 1.72k op/s 2024-04-26T18:00:15.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:15 smithi186 ceph-mon[32797]: pgmap v115: 97 pgs: 97 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 38 MiB/s wr, 1.40k op/s 2024-04-26T18:00:15.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:15 smithi031 ceph-mon[29164]: pgmap v115: 97 pgs: 97 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 38 MiB/s wr, 1.40k op/s 2024-04-26T18:00:15.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:15 smithi097 ceph-mon[32419]: pgmap v115: 97 pgs: 97 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 38 MiB/s wr, 1.40k op/s 2024-04-26T18:00:16.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.117+0000 7f48b514e640 1 -- 172.21.15.31:0/3647965723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48b010cf00 msgr2=0x7f48b010f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:16.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.117+0000 7f48b514e640 1 --2- 172.21.15.31:0/3647965723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48b010cf00 0x7f48b010f3a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009ca0 tx=0x7f48a802a9e0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.117+0000 7f48b514e640 1 -- 172.21.15.31:0/3647965723 shutdown_connections 2024-04-26T18:00:16.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.117+0000 7f48b514e640 1 --2- 172.21.15.31:0/3647965723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48b010cf00 0x7f48b010f3a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.117+0000 7f48b514e640 1 --2- 172.21.15.31:0/3647965723 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48b01044e0 0x7f48b010c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.117+0000 7f48b514e640 1 --2- 172.21.15.31:0/3647965723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48b0103bc0 0x7f48b0103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.117+0000 7f48b514e640 1 -- 172.21.15.31:0/3647965723 >> 172.21.15.31:0/3647965723 conn(0x7f48b00789b0 msgr2=0x7f48b0078dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:16.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.118+0000 7f48b514e640 1 -- 172.21.15.31:0/3647965723 shutdown_connections 2024-04-26T18:00:16.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.119+0000 7f48b514e640 1 -- 172.21.15.31:0/3647965723 wait complete. 2024-04-26T18:00:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.119+0000 7f48b514e640 1 Processor -- start 2024-04-26T18:00:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.120+0000 7f48b514e640 1 -- start start 2024-04-26T18:00:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.120+0000 7f48b514e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48b0103bc0 0x7f48b019b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.121+0000 7f48b514e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48b01044e0 0x7f48b019b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.121+0000 7f48b514e640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48b010cf00 0x7f48b019fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.121+0000 7f48b514e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f48b0112c00 con 0x7f48b01044e0 2024-04-26T18:00:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.121+0000 7f48aed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48b0103bc0 0x7f48b019b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.121+0000 7f48aed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48b0103bc0 0x7f48b019b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:55534/0 (socket says 172.21.15.31:55534) 2024-04-26T18:00:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.121+0000 7f48aed9d640 1 -- 172.21.15.31:0/4188066425 learned_addr learned my addr 172.21.15.31:0/4188066425 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.121+0000 7f48b514e640 1 -- 172.21.15.31:0/4188066425 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f48b0112a80 con 0x7f48b010cf00 2024-04-26T18:00:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.121+0000 7f48ae59c640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48b01044e0 0x7f48b019b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.121+0000 7f48b514e640 1 -- 172.21.15.31:0/4188066425 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48b0112d80 con 0x7f48b0103bc0 2024-04-26T18:00:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.121+0000 7f48af59e640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48b010cf00 0x7f48b019fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.122+0000 7f48aed9d640 1 -- 172.21.15.31:0/4188066425 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48b010cf00 msgr2=0x7f48b019fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.122+0000 7f48aed9d640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48b010cf00 0x7f48b019fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.122+0000 7f48aed9d640 1 -- 172.21.15.31:0/4188066425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48b01044e0 msgr2=0x7f48b019b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.122+0000 7f48aed9d640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48b01044e0 0x7f48b019b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.122+0000 7f48aed9d640 1 -- 172.21.15.31:0/4188066425 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48b01a02c0 con 0x7f48b0103bc0 2024-04-26T18:00:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.122+0000 7f48ae59c640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48b01044e0 0x7f48b019b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:00:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.122+0000 7f48aed9d640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48b0103bc0 0x7f48b019b2d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f48a000c970 tx=0x7f48a000ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.122+0000 7f4897fff640 1 -- 172.21.15.31:0/4188066425 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a0007bb0 con 0x7f48b0103bc0 2024-04-26T18:00:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.122+0000 7f4897fff640 1 -- 172.21.15.31:0/4188066425 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48a0007d50 con 0x7f48b0103bc0 2024-04-26T18:00:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.122+0000 7f48b514e640 1 -- 172.21.15.31:0/4188066425 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48b01a05b0 con 0x7f48b0103bc0 2024-04-26T18:00:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.122+0000 7f4897fff640 1 -- 172.21.15.31:0/4188066425 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a000f450 con 0x7f48b0103bc0 2024-04-26T18:00:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.123+0000 7f48b514e640 1 -- 172.21.15.31:0/4188066425 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48b01a7e40 con 0x7f48b0103bc0 2024-04-26T18:00:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.126+0000 7f48b7e5f740 1 Processor -- start 2024-04-26T18:00:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.126+0000 7f48b7e5f740 1 -- start start 2024-04-26T18:00:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.126+0000 7f4897fff640 1 -- 172.21.15.31:0/4188066425 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f48a0004f00 con 0x7f48b0103bc0 2024-04-26T18:00:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.126+0000 7f48b7e5f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f8106710 0x5615f8106af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.126+0000 7f48b7e5f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f81113b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.126+0000 7f48b7e5f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f81118f0 0x5615f8113ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.126+0000 7f48b7e5f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5615f8064460 con 0x5615f8106710 2024-04-26T18:00:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.126+0000 7f48b7e5f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5615f801e590 con 0x5615f81118f0 2024-04-26T18:00:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.126+0000 7f48b7e5f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5615f8060420 con 0x5615f8107030 2024-04-26T18:00:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.127+0000 7f48aed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f81113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.127+0000 7f48aed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f81113b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:55548/0 (socket says 172.21.15.31:55548) 2024-04-26T18:00:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.127+0000 7f48aed9d640 1 -- 172.21.15.31:0/668774601 learned_addr learned my addr 172.21.15.31:0/668774601 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.127+0000 7f48ae59c640 1 --2- 172.21.15.31:0/668774601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f81118f0 0x5615f8113ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.127+0000 7f48af59e640 1 --2- 172.21.15.31:0/668774601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f8106710 0x5615f8106af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.127+0000 7f48aed9d640 1 -- 172.21.15.31:0/668774601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f81118f0 msgr2=0x5615f8113ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.127+0000 7f48aed9d640 1 --2- 172.21.15.31:0/668774601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f81118f0 0x5615f8113ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.127+0000 7f48aed9d640 1 -- 172.21.15.31:0/668774601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f8106710 msgr2=0x5615f8106af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.127+0000 7f48aed9d640 1 --2- 172.21.15.31:0/668774601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f8106710 0x5615f8106af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.127+0000 7f48aed9d640 1 -- 172.21.15.31:0/668774601 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615f802b0d0 con 0x5615f8107030 2024-04-26T18:00:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.127+0000 7f48ae59c640 1 --2- 172.21.15.31:0/668774601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f81118f0 0x5615f8113ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:00:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.128+0000 7f4897fff640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f48980a3dc0 0x7f48980a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.128+0000 7f4897fff640 1 -- 172.21.15.31:0/4188066425 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f48a00d0500 con 0x7f48b0103bc0 2024-04-26T18:00:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.129+0000 7f48aed9d640 1 --2- 172.21.15.31:0/668774601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f81113b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f48a00cdbb0 tx=0x7f48a00d6ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=542a5407fd3dbf3e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.129+0000 7f48af59e640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f48980a3dc0 0x7f48980a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.129+0000 7f48957fa640 1 -- 172.21.15.31:0/668774601 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a001d020 con 0x5615f8107030 2024-04-26T18:00:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.129+0000 7f48957fa640 1 -- 172.21.15.31:0/668774601 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48a00055f0 con 0x5615f8107030 2024-04-26T18:00:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.130+0000 7f48957fa640 1 -- 172.21.15.31:0/668774601 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a009bea0 con 0x5615f8107030 2024-04-26T18:00:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.130+0000 7f48b7e5f740 1 -- 172.21.15.31:0/668774601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 msgr2=0x5615f81113b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.130+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/668774601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f81113b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f48a00cdbb0 tx=0x7f48a00d6ae0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.130+0000 7f48af59e640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f48980a3dc0 0x7f48980a6280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f48a8006010 tx=0x7f48a803a040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.130+0000 7f48b7e5f740 1 -- 172.21.15.31:0/668774601 shutdown_connections 2024-04-26T18:00:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.130+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/668774601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f81118f0 0x5615f8113ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.130+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/668774601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f81113b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.130+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/668774601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f8106710 0x5615f8106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.130+0000 7f48b7e5f740 1 -- 172.21.15.31:0/668774601 >> 172.21.15.31:0/668774601 conn(0x5615f8102c20 msgr2=0x5615f8103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.130+0000 7f48b7e5f740 1 -- 172.21.15.31:0/668774601 shutdown_connections 2024-04-26T18:00:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.130+0000 7f48b7e5f740 1 -- 172.21.15.31:0/668774601 wait complete. 2024-04-26T18:00:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.131+0000 7f48b7e5f740 1 -- start start 2024-04-26T18:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.131+0000 7f48b7e5f740 1 Processor -- start 2024-04-26T18:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.132+0000 7f48b7e5f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f8106710 0x5615f810e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.132+0000 7f48b7e5f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f810ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.132+0000 7f48b7e5f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f81118f0 0x5615f810f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.132+0000 7f48b7e5f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5615f8114520 con 0x5615f81118f0 2024-04-26T18:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.132+0000 7f48b7e5f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5615f7febb60 con 0x5615f8106710 2024-04-26T18:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.132+0000 7f48b7e5f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5615f81146a0 con 0x5615f8107030 2024-04-26T18:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.132+0000 7f48b7e5f740 10 client.0 _ll_register_callbacks cb 0x5615f80102b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.132+0000 7f48af59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f81118f0 0x5615f810f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.132+0000 7f48af59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f81118f0 0x5615f810f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42818/0 (socket says 172.21.15.31:42818) 2024-04-26T18:00:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.132+0000 7f48ae59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f810ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.132+0000 7f48af59e640 1 -- 172.21.15.31:0/614148717 learned_addr learned my addr 172.21.15.31:0/614148717 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.132+0000 7f48aed9d640 1 --2- 172.21.15.31:0/614148717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f8106710 0x5615f810e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.133+0000 7f48ae59c640 1 -- 172.21.15.31:0/614148717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f8106710 msgr2=0x5615f810e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.133+0000 7f48ae59c640 1 --2- 172.21.15.31:0/614148717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f8106710 0x5615f810e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.133+0000 7f48ae59c640 1 -- 172.21.15.31:0/614148717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f81118f0 msgr2=0x5615f810f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.133+0000 7f48ae59c640 1 --2- 172.21.15.31:0/614148717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f81118f0 0x5615f810f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.133+0000 7f48ae59c640 1 -- 172.21.15.31:0/614148717 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615f7d76210 con 0x5615f8107030 2024-04-26T18:00:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.133+0000 7f48af59e640 1 --2- 172.21.15.31:0/614148717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f81118f0 0x5615f810f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:00:16.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.133+0000 7f48ae59c640 1 --2- 172.21.15.31:0/614148717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f810ee30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x5615f8103db0 tx=0x7f48a400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cdfb45cb1f792b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:16.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.133+0000 7f4892ff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T18:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.134+0000 7f4892ff5640 1 -- 172.21.15.31:0/614148717 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a40120d0 con 0x5615f8107030 2024-04-26T18:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.134+0000 7f4892ff5640 1 -- 172.21.15.31:0/614148717 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f48a4004020 con 0x5615f8107030 2024-04-26T18:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.134+0000 7f48b7e5f740 1 -- 172.21.15.31:0/614148717 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5615f8072070 con 0x5615f8107030 2024-04-26T18:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.134+0000 7f4892ff5640 1 -- 172.21.15.31:0/614148717 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f48a4012270 con 0x5615f8107030 2024-04-26T18:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.134+0000 7f48b7e5f740 10 client.25327 fetch_fsmap learned FSMap version 32 2024-04-26T18:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.134+0000 7f48b7e5f740 1 -- 172.21.15.31:0/614148717 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5615f7e42620 con 0x5615f8107030 2024-04-26T18:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.134+0000 7f4892ff5640 1 -- 172.21.15.31:0/614148717 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5615f8072070 con 0x5615f8107030 2024-04-26T18:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.134+0000 7f4892ff5640 1 -- 172.21.15.31:0/614148717 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f48a401c470 con 0x5615f8107030 2024-04-26T18:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.134+0000 7f48b7e5f740 10 client.25327 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.134+0000 7f48b7e5f740 10 client.25327 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.135+0000 7f48b7e5f740 20 client.25327 populate_metadata read hostname 'smithi031' 2024-04-26T18:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.135+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/614148717 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5615f8121f70 0x5615f81243f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.135+0000 7f48b7e5f740 4 client.25327 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.135+0000 7f48b7e5f740 1 -- 172.21.15.31:0/614148717 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615f806bee0 con 0x5615f8121f70 2024-04-26T18:00:16.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.135+0000 7f48aed9d640 1 --2- 172.21.15.31:0/614148717 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5615f8121f70 0x5615f81243f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.136+0000 7f48aed9d640 1 --2- 172.21.15.31:0/614148717 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5615f8121f70 0x5615f81243f0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8108c6ab4a353e54 server_cookie=8a29fe04ed2255b6 in_seq=0 out_seq=0 2024-04-26T18:00:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.136+0000 7f4892ff5640 10 client.25327 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.136+0000 7f4892ff5640 1 -- 172.21.15.31:0/614148717 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5615f806bee0 con 0x5615f8121f70 2024-04-26T18:00:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.136+0000 7f4892ff5640 10 client.25327 handle_command_reply: tid=0 2024-04-26T18:00:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.137+0000 7f48b7e5f740 1 client.25327 shutdown 2024-04-26T18:00:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.137+0000 7f48b7e5f740 1 -- 172.21.15.31:0/614148717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 msgr2=0x5615f810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.137+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/614148717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f810ee30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x5615f8103db0 tx=0x7f48a400f790 comp rx=0 tx=0).stop 2024-04-26T18:00:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.137+0000 7f48b7e5f740 1 -- 172.21.15.31:0/614148717 shutdown_connections 2024-04-26T18:00:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.137+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/614148717 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5615f8121f70 0x5615f81243f0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.137+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/614148717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f81118f0 0x5615f810f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.137+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/614148717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f810ee30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.137+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/614148717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f8106710 0x5615f810e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.137+0000 7f48b7e5f740 1 -- 172.21.15.31:0/614148717 >> 172.21.15.31:0/614148717 conn(0x5615f8102c20 msgr2=0x5615f80f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:16.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.137+0000 7f4892ff5640 0 client.25327 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:16.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.137+0000 7f48b7e5f740 1 -- 172.21.15.31:0/614148717 shutdown_connections 2024-04-26T18:00:16.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.137+0000 7f48b7e5f740 1 -- 172.21.15.31:0/614148717 wait complete. 2024-04-26T18:00:16.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.137+0000 7f48b7e5f740 20 client.25327 trim_cache size 0 max 16384 2024-04-26T18:00:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.162+0000 7f48b7e5f740 1 Processor -- start 2024-04-26T18:00:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.162+0000 7f48b7e5f740 1 -- start start 2024-04-26T18:00:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.163+0000 7f48b7e5f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8106710 0x5615f811d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.163+0000 7f48b7e5f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f8107030 0x5615f8109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.163+0000 7f48b7e5f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f81118f0 0x5615f8129d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.163+0000 7f48b7e5f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f48b01aebb0 con 0x5615f8107030 2024-04-26T18:00:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.163+0000 7f48aed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8106710 0x5615f811d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.163+0000 7f48b7e5f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f48b01ac420 con 0x5615f81118f0 2024-04-26T18:00:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.163+0000 7f48aed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8106710 0x5615f811d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:55564/0 (socket says 172.21.15.31:55564) 2024-04-26T18:00:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.163+0000 7f48ae59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f8107030 0x5615f8109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.163+0000 7f48b7e5f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48b01aa2f0 con 0x5615f8106710 2024-04-26T18:00:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.163+0000 7f48aed9d640 1 -- 172.21.15.31:0/3020273988 learned_addr learned my addr 172.21.15.31:0/3020273988 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.163+0000 7f48af59e640 1 --2- 172.21.15.31:0/3020273988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f81118f0 0x5615f8129d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.164+0000 7f48ae59c640 1 -- 172.21.15.31:0/3020273988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8106710 msgr2=0x5615f811d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.164+0000 7f48ae59c640 1 --2- 172.21.15.31:0/3020273988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8106710 0x5615f811d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.164+0000 7f48ae59c640 1 -- 172.21.15.31:0/3020273988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f81118f0 msgr2=0x5615f8129d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.164+0000 7f48ae59c640 1 --2- 172.21.15.31:0/3020273988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f81118f0 0x5615f8129d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.164+0000 7f48ae59c640 1 -- 172.21.15.31:0/3020273988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615f8113ce0 con 0x5615f8107030 2024-04-26T18:00:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.164+0000 7f48ae59c640 1 --2- 172.21.15.31:0/3020273988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f8107030 0x5615f8109120 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f48a401c8c0 tx=0x7f48a4005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5973b7fe12c7a75 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.165+0000 7f4894ff9640 1 -- 172.21.15.31:0/3020273988 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a402a030 con 0x5615f8107030 2024-04-26T18:00:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.165+0000 7f4894ff9640 1 -- 172.21.15.31:0/3020273988 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48a4005580 con 0x5615f8107030 2024-04-26T18:00:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.165+0000 7f4894ff9640 1 -- 172.21.15.31:0/3020273988 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a40035f0 con 0x5615f8107030 2024-04-26T18:00:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.166+0000 7f48b7e5f740 1 -- 172.21.15.31:0/3020273988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f8107030 msgr2=0x5615f8109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.166+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/3020273988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f8107030 0x5615f8109120 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f48a401c8c0 tx=0x7f48a4005970 comp rx=0 tx=0).stop 2024-04-26T18:00:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.166+0000 7f48b7e5f740 1 -- 172.21.15.31:0/3020273988 shutdown_connections 2024-04-26T18:00:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.166+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/3020273988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f81118f0 0x5615f8129d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.166+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/3020273988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f8107030 0x5615f8109120 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.166+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/3020273988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8106710 0x5615f811d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.166+0000 7f48b7e5f740 1 -- 172.21.15.31:0/3020273988 >> 172.21.15.31:0/3020273988 conn(0x5615f8102c20 msgr2=0x5615f8109c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.166+0000 7f48b7e5f740 1 -- 172.21.15.31:0/3020273988 shutdown_connections 2024-04-26T18:00:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.166+0000 7f48b7e5f740 1 -- 172.21.15.31:0/3020273988 wait complete. 2024-04-26T18:00:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.167+0000 7f48b7e5f740 1 -- start start 2024-04-26T18:00:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.167+0000 7f48b7e5f740 1 Processor -- start 2024-04-26T18:00:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.167+0000 7f48b7e5f740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f8106710 0x5615f8119e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.167+0000 7f48b7e5f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f810c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48b7e5f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f81118f0 0x5615f810ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48b7e5f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f48b01aa940 con 0x5615f81118f0 2024-04-26T18:00:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48b7e5f740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f48b01abd50 con 0x5615f8106710 2024-04-26T18:00:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48b7e5f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48b01aafb0 con 0x5615f8107030 2024-04-26T18:00:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48aed9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f8106710 0x5615f8119e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48aed9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f8106710 0x5615f8119e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:49396/0 (socket says 172.21.15.31:49396) 2024-04-26T18:00:16.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48aed9d640 1 -- 172.21.15.31:0/2714343524 learned_addr learned my addr 172.21.15.31:0/2714343524 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:16.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48ae59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f810c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48b7e5f740 10 client.0 _ll_register_callbacks cb 0x7f48a4009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:00:16.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48ae59c640 1 -- 172.21.15.31:0/2714343524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f8106710 msgr2=0x5615f8119e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:16.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48ae59c640 1 --2- 172.21.15.31:0/2714343524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f8106710 0x5615f8119e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48ae59c640 1 -- 172.21.15.31:0/2714343524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f81118f0 msgr2=0x5615f810ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48ae59c640 1 --2- 172.21.15.31:0/2714343524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f81118f0 0x5615f810ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48ae59c640 1 -- 172.21.15.31:0/2714343524 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615f810b0c0 con 0x5615f8107030 2024-04-26T18:00:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.168+0000 7f48aed9d640 1 --2- 172.21.15.31:0/2714343524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f8106710 0x5615f8119e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:00:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.169+0000 7f48ae59c640 1 --2- 172.21.15.31:0/2714343524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f810c030 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f48a401c960 tx=0x7f48a4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3afb5963ab9fb0b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.169+0000 7f4892ff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T18:00:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.169+0000 7f4892ff5640 1 -- 172.21.15.31:0/2714343524 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a402a070 con 0x5615f8107030 2024-04-26T18:00:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.169+0000 7f48b7e5f740 1 -- 172.21.15.31:0/2714343524 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48a4005580 con 0x5615f8107030 2024-04-26T18:00:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.169+0000 7f4892ff5640 1 -- 172.21.15.31:0/2714343524 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f48a4018070 con 0x5615f8107030 2024-04-26T18:00:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.170+0000 7f4892ff5640 1 -- 172.21.15.31:0/2714343524 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f48a4005730 con 0x5615f8107030 2024-04-26T18:00:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.170+0000 7f48b7e5f740 10 client.25333 fetch_fsmap learned FSMap version 32 2024-04-26T18:00:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.170+0000 7f48b7e5f740 1 -- 172.21.15.31:0/2714343524 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5615f8187b30 con 0x5615f8107030 2024-04-26T18:00:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.171+0000 7f4892ff5640 1 -- 172.21.15.31:0/2714343524 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48a4005580 con 0x5615f8107030 2024-04-26T18:00:16.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.171+0000 7f4892ff5640 1 -- 172.21.15.31:0/2714343524 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f48a4025430 con 0x5615f8107030 2024-04-26T18:00:16.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.172+0000 7f48b7e5f740 10 client.25333 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:00:16.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.172+0000 7f48b7e5f740 10 client.25333 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:00:16.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.172+0000 7f48b7e5f740 20 client.25333 populate_metadata read hostname 'smithi031' 2024-04-26T18:00:16.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.172+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/2714343524 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5615f81881f0 0x5615f818a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:16.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.172+0000 7f48b7e5f740 4 client.25333 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:00:16.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.172+0000 7f48b7e5f740 1 -- 172.21.15.31:0/2714343524 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5615f7fe75f0 con 0x5615f81881f0 2024-04-26T18:00:16.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.172+0000 7f48aed9d640 1 --2- 172.21.15.31:0/2714343524 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5615f81881f0 0x5615f818a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:16.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.173+0000 7f48aed9d640 1 --2- 172.21.15.31:0/2714343524 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5615f81881f0 0x5615f818a610 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e263a5b96e9f4cd4 server_cookie=4bf4c57435cdcab in_seq=0 out_seq=0 2024-04-26T18:00:16.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.173+0000 7f4892ff5640 10 client.25333 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:16.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f4892ff5640 1 -- 172.21.15.31:0/2714343524 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5615f7fe75f0 con 0x5615f81881f0 2024-04-26T18:00:16.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f4892ff5640 10 client.25333 handle_command_reply: tid=0 2024-04-26T18:00:16.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f48b7e5f740 1 client.25333 shutdown 2024-04-26T18:00:16.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f48b7e5f740 1 -- 172.21.15.31:0/2714343524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 msgr2=0x5615f810c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:16.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/2714343524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f810c030 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f48a401c960 tx=0x7f48a4000bc0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f48b7e5f740 1 -- 172.21.15.31:0/2714343524 shutdown_connections 2024-04-26T18:00:16.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/2714343524 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5615f81881f0 0x5615f818a610 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/2714343524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615f81118f0 0x5615f810ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/2714343524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5615f8107030 0x5615f810c030 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f48b7e5f740 1 --2- 172.21.15.31:0/2714343524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5615f8106710 0x5615f8119e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f48b7e5f740 1 -- 172.21.15.31:0/2714343524 >> 172.21.15.31:0/2714343524 conn(0x5615f8102c20 msgr2=0x5615f8110120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:16.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f4892ff5640 0 client.25333 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:16.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f48b7e5f740 1 -- 172.21.15.31:0/2714343524 shutdown_connections 2024-04-26T18:00:16.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f48b7e5f740 1 -- 172.21.15.31:0/2714343524 wait complete. 2024-04-26T18:00:16.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.191+0000 7f48b7e5f740 20 client.25333 trim_cache size 0 max 16384 2024-04-26T18:00:16.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:00:16.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:00:16.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:00:16.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:00:16.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.192+0000 7f48b514e640 1 -- 172.21.15.31:0/4188066425 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f48980a3dc0 msgr2=0x7f48980a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:16.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.192+0000 7f48b514e640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f48980a3dc0 0x7f48980a6280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f48a8006010 tx=0x7f48a803a040 comp rx=0 tx=0).stop 2024-04-26T18:00:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.192+0000 7f48b514e640 1 -- 172.21.15.31:0/4188066425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48b0103bc0 msgr2=0x7f48b019b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.192+0000 7f48b514e640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48b0103bc0 0x7f48b019b2d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f48a000c970 tx=0x7f48a000ce30 comp rx=0 tx=0).stop 2024-04-26T18:00:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.192+0000 7f48b514e640 1 -- 172.21.15.31:0/4188066425 shutdown_connections 2024-04-26T18:00:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.192+0000 7f48b514e640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f48b010cf00 0x7f48b019fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.192+0000 7f48b514e640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f48980a3dc0 0x7f48980a6280 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.192+0000 7f48b514e640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48b01044e0 0x7f48b019b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.192+0000 7f48b514e640 1 --2- 172.21.15.31:0/4188066425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48b0103bc0 0x7f48b019b2d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.192+0000 7f48b514e640 1 -- 172.21.15.31:0/4188066425 >> 172.21.15.31:0/4188066425 conn(0x7f48b00789b0 msgr2=0x7f48b0104960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.192+0000 7f48b514e640 1 -- 172.21.15.31:0/4188066425 shutdown_connections 2024-04-26T18:00:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:16.192+0000 7f48b514e640 1 -- 172.21.15.31:0/4188066425 wait complete. 2024-04-26T18:00:17.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:17 smithi186 ceph-mon[32797]: pgmap v116: 97 pgs: 97 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 38 MiB/s wr, 1.50k op/s 2024-04-26T18:00:17.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:17 smithi031 ceph-mon[29164]: pgmap v116: 97 pgs: 97 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 38 MiB/s wr, 1.50k op/s 2024-04-26T18:00:17.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:17 smithi097 ceph-mon[32419]: pgmap v116: 97 pgs: 97 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 38 MiB/s wr, 1.50k op/s 2024-04-26T18:00:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 175.0 s, 3341.4 tps, lat 30.038 ms stddev 33.576 2024-04-26T18:00:19.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:19 smithi186 ceph-mon[32797]: pgmap v117: 97 pgs: 97 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 MiB/s wr, 1.36k op/s 2024-04-26T18:00:19.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:19 smithi031 ceph-mon[29164]: pgmap v117: 97 pgs: 97 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 MiB/s wr, 1.36k op/s 2024-04-26T18:00:19.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:19 smithi097 ceph-mon[32419]: pgmap v117: 97 pgs: 97 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 MiB/s wr, 1.36k op/s 2024-04-26T18:00:21.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:21 smithi186 ceph-mon[32797]: pgmap v118: 97 pgs: 97 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.5 MiB/s wr, 667 op/s 2024-04-26T18:00:21.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:21 smithi031 ceph-mon[29164]: pgmap v118: 97 pgs: 97 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.5 MiB/s wr, 667 op/s 2024-04-26T18:00:21.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:21 smithi097 ceph-mon[32419]: pgmap v118: 97 pgs: 97 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.5 MiB/s wr, 667 op/s 2024-04-26T18:00:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 180.0 s, 3418.2 tps, lat 29.163 ms stddev 30.062 2024-04-26T18:00:23.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:23 smithi186 ceph-mon[32797]: pgmap v119: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 13 MiB/s wr, 1.11k op/s 2024-04-26T18:00:23.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:23 smithi031 ceph-mon[29164]: pgmap v119: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 13 MiB/s wr, 1.11k op/s 2024-04-26T18:00:23.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:23 smithi097 ceph-mon[32419]: pgmap v119: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 13 MiB/s wr, 1.11k op/s 2024-04-26T18:00:25.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:25 smithi186 ceph-mon[32797]: pgmap v120: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 745 op/s 2024-04-26T18:00:25.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:25 smithi031 ceph-mon[29164]: pgmap v120: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 745 op/s 2024-04-26T18:00:25.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:25 smithi097 ceph-mon[32419]: pgmap v120: 97 pgs: 97 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 745 op/s 2024-04-26T18:00:27.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:27 smithi186 ceph-mon[32797]: pgmap v121: 97 pgs: 97 active+clean; 2.2 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 842 op/s 2024-04-26T18:00:27.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:27 smithi031 ceph-mon[29164]: pgmap v121: 97 pgs: 97 active+clean; 2.2 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 842 op/s 2024-04-26T18:00:27.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:27 smithi097 ceph-mon[32419]: pgmap v121: 97 pgs: 97 active+clean; 2.2 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 842 op/s 2024-04-26T18:00:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 185.0 s, 3256.2 tps, lat 30.716 ms stddev 29.778 2024-04-26T18:00:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.323+0000 7f0df0f5f640 1 -- 172.21.15.31:0/1953276984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dec10abf0 msgr2=0x7f0dec10afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.323+0000 7f0df0f5f640 1 --2- 172.21.15.31:0/1953276984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dec10abf0 0x7f0dec10afd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc009900 tx=0x7f0ddc02a3f0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.324+0000 7f0df0f5f640 1 -- 172.21.15.31:0/1953276984 shutdown_connections 2024-04-26T18:00:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.324+0000 7f0df0f5f640 1 --2- 172.21.15.31:0/1953276984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec1054f0 0x7f0dec1119b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.324+0000 7f0df0f5f640 1 --2- 172.21.15.31:0/1953276984 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0dec104b30 0x7f0dec104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.324+0000 7f0df0f5f640 1 --2- 172.21.15.31:0/1953276984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dec10abf0 0x7f0dec10afd0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.324+0000 7f0df0f5f640 1 -- 172.21.15.31:0/1953276984 >> 172.21.15.31:0/1953276984 conn(0x7f0dec100620 msgr2=0x7f0dec102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.324+0000 7f0df0f5f640 1 -- 172.21.15.31:0/1953276984 shutdown_connections 2024-04-26T18:00:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.324+0000 7f0df0f5f640 1 -- 172.21.15.31:0/1953276984 wait complete. 2024-04-26T18:00:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.325+0000 7f0df0f5f640 1 Processor -- start 2024-04-26T18:00:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.325+0000 7f0df0f5f640 1 -- start start 2024-04-26T18:00:29.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.326+0000 7f0df0f5f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec104b30 0x7f0dec1102b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.326+0000 7f0df0f5f640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0dec1054f0 0x7f0dec1107f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.326+0000 7f0df0f5f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dec10abf0 0x7f0dec1ab220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.326+0000 7f0dea59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec104b30 0x7f0dec1102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.326+0000 7f0df0f5f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec077f10 con 0x7f0dec10abf0 2024-04-26T18:00:29.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.326+0000 7f0df0f5f640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec077d90 con 0x7f0dec1054f0 2024-04-26T18:00:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.326+0000 7f0df0f5f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec078090 con 0x7f0dec104b30 2024-04-26T18:00:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.326+0000 7f0dea59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec104b30 0x7f0dec1102b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:44180/0 (socket says 172.21.15.31:44180) 2024-04-26T18:00:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.326+0000 7f0dea59c640 1 -- 172.21.15.31:0/2738921137 learned_addr learned my addr 172.21.15.31:0/2738921137 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.326+0000 7f0dead9d640 1 --2- 172.21.15.31:0/2738921137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dec10abf0 0x7f0dec1ab220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.326+0000 7f0de9d9b640 1 --2- 172.21.15.31:0/2738921137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0dec1054f0 0x7f0dec1107f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.326+0000 7f0dea59c640 1 -- 172.21.15.31:0/2738921137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0dec1054f0 msgr2=0x7f0dec1107f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.326+0000 7f0dea59c640 1 --2- 172.21.15.31:0/2738921137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0dec1054f0 0x7f0dec1107f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.327+0000 7f0dea59c640 1 -- 172.21.15.31:0/2738921137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dec10abf0 msgr2=0x7f0dec1ab220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.327+0000 7f0dea59c640 1 --2- 172.21.15.31:0/2738921137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dec10abf0 0x7f0dec1ab220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.327+0000 7f0dea59c640 1 -- 172.21.15.31:0/2738921137 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dec1ab860 con 0x7f0dec104b30 2024-04-26T18:00:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.327+0000 7f0dea59c640 1 --2- 172.21.15.31:0/2738921137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec104b30 0x7f0dec1102b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc02a9e0 tx=0x7f0ddc003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.327+0000 7f0ddb7fe640 1 -- 172.21.15.31:0/2738921137 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc03f070 con 0x7f0dec104b30 2024-04-26T18:00:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.327+0000 7f0df0f5f640 1 -- 172.21.15.31:0/2738921137 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dec1abaf0 con 0x7f0dec104b30 2024-04-26T18:00:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.327+0000 7f0ddb7fe640 1 -- 172.21.15.31:0/2738921137 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ddc03bb20 con 0x7f0dec104b30 2024-04-26T18:00:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.327+0000 7f0ddb7fe640 1 -- 172.21.15.31:0/2738921137 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc035030 con 0x7f0dec104b30 2024-04-26T18:00:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.327+0000 7f0df0f5f640 1 -- 172.21.15.31:0/2738921137 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dec1ac030 con 0x7f0dec104b30 2024-04-26T18:00:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.330+0000 7f0ddb7fe640 1 -- 172.21.15.31:0/2738921137 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0ddc042070 con 0x7f0dec104b30 2024-04-26T18:00:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.330+0000 7f0df3c7c740 1 Processor -- start 2024-04-26T18:00:29.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.330+0000 7f0df3c7c740 1 -- start start 2024-04-26T18:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.330+0000 7f0ddb7fe640 1 --2- 172.21.15.31:0/2738921137 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f0dd40a3df0 0x7f0dd40a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0df3c7c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0ddb7fe640 1 -- 172.21.15.31:0/2738921137 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f0ddc0f0dc0 con 0x7f0dec104b30 2024-04-26T18:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0df3c7c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0df3c7c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1db4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0df3c7c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a0ba134460 con 0x55a0ba1d90e0 2024-04-26T18:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0df3c7c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55a0ba0ee590 con 0x55a0ba1cdf00 2024-04-26T18:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0df3c7c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a0ba130420 con 0x55a0ba1ce820 2024-04-26T18:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0dea59c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0dead9d640 1 --2- 172.21.15.31:0/2738921137 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f0dd40a3df0 0x7f0dd40a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0dea59c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1ce2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:54702/0 (socket says 172.21.15.31:54702) 2024-04-26T18:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0dea59c640 1 -- 172.21.15.31:0/4259719455 learned_addr learned my addr 172.21.15.31:0/4259719455 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0dead9d640 1 --2- 172.21.15.31:0/4259719455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1db4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0de9d9b640 1 --2- 172.21.15.31:0/4259719455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0dea59c640 1 -- 172.21.15.31:0/4259719455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1ce820 msgr2=0x55a0ba1d8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0dea59c640 1 --2- 172.21.15.31:0/4259719455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0dea59c640 1 -- 172.21.15.31:0/4259719455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 msgr2=0x55a0ba1db4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0dea59c640 1 --2- 172.21.15.31:0/4259719455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1db4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.331+0000 7f0dea59c640 1 -- 172.21.15.31:0/4259719455 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0ba0fb0d0 con 0x55a0ba1cdf00 2024-04-26T18:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.332+0000 7f0dea59c640 1 --2- 172.21.15.31:0/4259719455 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1ce2e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0b92c0 tx=0x7f0ddc031610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c4e7873d65392d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.332+0000 7f0dd8ff9640 1 -- 172.21.15.31:0/4259719455 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc049070 con 0x55a0ba1cdf00 2024-04-26T18:00:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.332+0000 7f0dead9d640 1 --2- 172.21.15.31:0/2738921137 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f0dd40a3df0 0x7f0dd40a62b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x55a0ba1cb830 tx=0x7f0de4009b70 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.332+0000 7f0dd8ff9640 1 -- 172.21.15.31:0/4259719455 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ddc0c0020 con 0x55a0ba1cdf00 2024-04-26T18:00:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.332+0000 7f0dd8ff9640 1 -- 172.21.15.31:0/4259719455 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc0b94d0 con 0x55a0ba1cdf00 2024-04-26T18:00:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.333+0000 7f0df3c7c740 1 -- 172.21.15.31:0/4259719455 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 msgr2=0x55a0ba1ce2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.333+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/4259719455 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1ce2e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0b92c0 tx=0x7f0ddc031610 comp rx=0 tx=0).stop 2024-04-26T18:00:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.333+0000 7f0df3c7c740 1 -- 172.21.15.31:0/4259719455 shutdown_connections 2024-04-26T18:00:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.333+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/4259719455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1db4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.333+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/4259719455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.333+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/4259719455 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1ce2e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.333+0000 7f0df3c7c740 1 -- 172.21.15.31:0/4259719455 >> 172.21.15.31:0/4259719455 conn(0x55a0ba1ca410 msgr2=0x55a0ba1ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.333+0000 7f0df3c7c740 1 -- 172.21.15.31:0/4259719455 shutdown_connections 2024-04-26T18:00:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.333+0000 7f0df3c7c740 1 -- 172.21.15.31:0/4259719455 wait complete. 2024-04-26T18:00:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.334+0000 7f0df3c7c740 1 -- start start 2024-04-26T18:00:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.334+0000 7f0df3c7c740 1 Processor -- start 2024-04-26T18:00:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.334+0000 7f0df3c7c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1d6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.334+0000 7f0df3c7c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.334+0000 7f0df3c7c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1d6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.334+0000 7f0de9d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.334+0000 7f0de9d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:44202/0 (socket says 172.21.15.31:44202) 2024-04-26T18:00:29.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.335+0000 7f0dead9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1d6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.335+0000 7f0de9d9b640 1 -- 172.21.15.31:0/2600353090 learned_addr learned my addr 172.21.15.31:0/2600353090 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.335+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2600353090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a0ba1dbad0 con 0x55a0ba1d90e0 2024-04-26T18:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.335+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2600353090 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55a0ba0bbb60 con 0x55a0ba1cdf00 2024-04-26T18:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.335+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2600353090 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a0ba1dbc50 con 0x55a0ba1ce820 2024-04-26T18:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.335+0000 7f0df3c7c740 10 client.0 _ll_register_callbacks cb 0x55a0ba0e02b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.335+0000 7f0de9d9b640 1 -- 172.21.15.31:0/2600353090 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 msgr2=0x55a0ba1d6100 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T18:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.335+0000 7f0de9d9b640 1 --2- 172.21.15.31:0/2600353090 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1d6100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.335+0000 7f0de9d9b640 1 -- 172.21.15.31:0/2600353090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 msgr2=0x55a0ba1d6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.335+0000 7f0de9d9b640 1 --2- 172.21.15.31:0/2600353090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1d6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.335+0000 7f0de9d9b640 1 -- 172.21.15.31:0/2600353090 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0b9e46210 con 0x55a0ba1ce820 2024-04-26T18:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.336+0000 7f0de9d9b640 1 --2- 172.21.15.31:0/2600353090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d6640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0de0012700 tx=0x7f0de0012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9365e565c015a45 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.336+0000 7f0dd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T18:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.336+0000 7f0dd1ffb640 1 -- 172.21.15.31:0/2600353090 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de0004050 con 0x55a0ba1ce820 2024-04-26T18:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.336+0000 7f0dd1ffb640 1 -- 172.21.15.31:0/2600353090 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f0de00041f0 con 0x55a0ba1ce820 2024-04-26T18:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.336+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2600353090 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ddc0c0020 con 0x55a0ba1ce820 2024-04-26T18:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.337+0000 7f0dd1ffb640 1 -- 172.21.15.31:0/2600353090 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0de00043f0 con 0x55a0ba1ce820 2024-04-26T18:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.337+0000 7f0df3c7c740 10 client.25345 fetch_fsmap learned FSMap version 32 2024-04-26T18:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.337+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2600353090 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a0b9f12620 con 0x55a0ba1ce820 2024-04-26T18:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.337+0000 7f0dd1ffb640 1 -- 172.21.15.31:0/2600353090 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ddc0c0020 con 0x55a0ba1ce820 2024-04-26T18:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.338+0000 7f0dd1ffb640 1 -- 172.21.15.31:0/2600353090 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f0de0003ab0 con 0x55a0ba1ce820 2024-04-26T18:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.338+0000 7f0df3c7c740 10 client.25345 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.338+0000 7f0df3c7c740 10 client.25345 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.338+0000 7f0df3c7c740 20 client.25345 populate_metadata read hostname 'smithi031' 2024-04-26T18:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.338+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2600353090 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a0ba1e9060 0x55a0ba1eb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.338+0000 7f0df3c7c740 4 client.25345 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.338+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2600353090 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0ba142070 con 0x55a0ba1e9060 2024-04-26T18:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.338+0000 7f0dea59c640 1 --2- 172.21.15.31:0/2600353090 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a0ba1e9060 0x55a0ba1eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.338+0000 7f0dd1ffb640 10 client.25345 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.338+0000 7f0dea59c640 1 --2- 172.21.15.31:0/2600353090 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a0ba1e9060 0x55a0ba1eb450 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=198cb33462600689 server_cookie=73954951cdade682 in_seq=0 out_seq=0 2024-04-26T18:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.339+0000 7f0dd1ffb640 1 -- 172.21.15.31:0/2600353090 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a0ba142070 con 0x55a0ba1e9060 2024-04-26T18:00:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.339+0000 7f0dd1ffb640 10 client.25345 handle_command_reply: tid=0 2024-04-26T18:00:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.339+0000 7f0df3c7c740 1 client.25345 shutdown 2024-04-26T18:00:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.339+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2600353090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1ce820 msgr2=0x55a0ba1d6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.339+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2600353090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d6640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0de0012700 tx=0x7f0de0012bc0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.340+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2600353090 shutdown_connections 2024-04-26T18:00:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.340+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2600353090 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a0ba1e9060 0x55a0ba1eb450 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.340+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2600353090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1d6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.340+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2600353090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d6640 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.340+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2600353090 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1d6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.340+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2600353090 >> 172.21.15.31:0/2600353090 conn(0x55a0ba1ca410 msgr2=0x55a0ba1d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:29.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.340+0000 7f0dd1ffb640 0 client.25345 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:29.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.340+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2600353090 shutdown_connections 2024-04-26T18:00:29.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.340+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2600353090 wait complete. 2024-04-26T18:00:29.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.340+0000 7f0df3c7c740 20 client.25345 trim_cache size 0 max 16384 2024-04-26T18:00:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.362+0000 7f0df3c7c740 1 Processor -- start 2024-04-26T18:00:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.362+0000 7f0df3c7c740 1 -- start start 2024-04-26T18:00:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.362+0000 7f0df3c7c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1d88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0df3c7c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1ce820 0x55a0ba1c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0df3c7c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1e4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0df3c7c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1b7420 con 0x55a0ba1ce820 2024-04-26T18:00:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0de9d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1ce820 0x55a0ba1c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0df3c7c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1b4c90 con 0x55a0ba1cdf00 2024-04-26T18:00:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0df3c7c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1b2ae0 con 0x55a0ba1d90e0 2024-04-26T18:00:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0de9d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1ce820 0x55a0ba1c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60040/0 (socket says 172.21.15.31:60040) 2024-04-26T18:00:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0de9d9b640 1 -- 172.21.15.31:0/2364514758 learned_addr learned my addr 172.21.15.31:0/2364514758 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0dea59c640 1 --2- 172.21.15.31:0/2364514758 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1d88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0dead9d640 1 --2- 172.21.15.31:0/2364514758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1e4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0de9d9b640 1 -- 172.21.15.31:0/2364514758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1d90e0 msgr2=0x55a0ba1e4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0de9d9b640 1 --2- 172.21.15.31:0/2364514758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1e4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0de9d9b640 1 -- 172.21.15.31:0/2364514758 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 msgr2=0x55a0ba1d88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0de9d9b640 1 --2- 172.21.15.31:0/2364514758 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1d88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0de9d9b640 1 -- 172.21.15.31:0/2364514758 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0ba1e7bb0 con 0x55a0ba1ce820 2024-04-26T18:00:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0dea59c640 1 --2- 172.21.15.31:0/2364514758 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1d88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:00:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0dead9d640 1 --2- 172.21.15.31:0/2364514758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1e4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:00:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0de9d9b640 1 --2- 172.21.15.31:0/2364514758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1ce820 0x55a0ba1c97f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0de0012420 tx=0x7f0de0005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8d4cff8542d485 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.363+0000 7f0dd3fff640 1 -- 172.21.15.31:0/2364514758 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de002c070 con 0x55a0ba1ce820 2024-04-26T18:00:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.364+0000 7f0dd3fff640 1 -- 172.21.15.31:0/2364514758 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0de0003fd0 con 0x55a0ba1ce820 2024-04-26T18:00:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.364+0000 7f0dd3fff640 1 -- 172.21.15.31:0/2364514758 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de000fb80 con 0x55a0ba1ce820 2024-04-26T18:00:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.364+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2364514758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1ce820 msgr2=0x55a0ba1c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.364+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2364514758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1ce820 0x55a0ba1c97f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0de0012420 tx=0x7f0de0005d30 comp rx=0 tx=0).stop 2024-04-26T18:00:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.365+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2364514758 shutdown_connections 2024-04-26T18:00:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.365+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2364514758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1e4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.365+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2364514758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1ce820 0x55a0ba1c97f0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.365+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2364514758 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1d88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.365+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2364514758 >> 172.21.15.31:0/2364514758 conn(0x55a0ba1ca410 msgr2=0x55a0ba1d81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.366+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2364514758 shutdown_connections 2024-04-26T18:00:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.366+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2364514758 wait complete. 2024-04-26T18:00:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.367+0000 7f0df3c7c740 1 -- start start 2024-04-26T18:00:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.367+0000 7f0df3c7c740 1 Processor -- start 2024-04-26T18:00:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0df3c7c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1d3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0df3c7c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0df3c7c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0df3c7c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1b3150 con 0x55a0ba1d90e0 2024-04-26T18:00:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0df3c7c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1b45c0 con 0x55a0ba1ce820 2024-04-26T18:00:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0df3c7c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec1b3820 con 0x55a0ba1cdf00 2024-04-26T18:00:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0de9d9b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0df3c7c740 10 client.0 _ll_register_callbacks cb 0x7f0de0009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:00:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0dead9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0dead9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60052/0 (socket says 172.21.15.31:60052) 2024-04-26T18:00:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0dead9d640 1 -- 172.21.15.31:0/2618687073 learned_addr learned my addr 172.21.15.31:0/2618687073 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0dea59c640 1 --2- 172.21.15.31:0/2618687073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1d3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0de9d9b640 1 -- 172.21.15.31:0/2618687073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1cdf00 msgr2=0x55a0ba1d3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0de9d9b640 1 --2- 172.21.15.31:0/2618687073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1d3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0de9d9b640 1 -- 172.21.15.31:0/2618687073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 msgr2=0x55a0ba1d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0de9d9b640 1 --2- 172.21.15.31:0/2618687073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0de9d9b640 1 -- 172.21.15.31:0/2618687073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0ba24e910 con 0x55a0ba1ce820 2024-04-26T18:00:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.368+0000 7f0dead9d640 1 --2- 172.21.15.31:0/2618687073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:00:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0de9d9b640 1 --2- 172.21.15.31:0/2618687073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d3650 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0de0003fa0 tx=0x7f0de0009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b2ccf53690d5ed1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0dd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T18:00:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0dd1ffb640 1 -- 172.21.15.31:0/2618687073 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de002c040 con 0x55a0ba1ce820 2024-04-26T18:00:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0dd1ffb640 1 -- 172.21.15.31:0/2618687073 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f0de0005040 con 0x55a0ba1ce820 2024-04-26T18:00:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2618687073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0de0003fd0 con 0x55a0ba1ce820 2024-04-26T18:00:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0dd1ffb640 1 -- 172.21.15.31:0/2618687073 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0de0005260 con 0x55a0ba1ce820 2024-04-26T18:00:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0df3c7c740 10 client.25424 fetch_fsmap learned FSMap version 32 2024-04-26T18:00:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2618687073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a0ba24fb90 con 0x55a0ba1ce820 2024-04-26T18:00:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0dd1ffb640 1 -- 172.21.15.31:0/2618687073 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0de0003fd0 con 0x55a0ba1ce820 2024-04-26T18:00:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0dd1ffb640 1 -- 172.21.15.31:0/2618687073 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f0de0003680 con 0x55a0ba1ce820 2024-04-26T18:00:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0df3c7c740 10 client.25424 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:00:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0df3c7c740 10 client.25424 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:00:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0df3c7c740 20 client.25424 populate_metadata read hostname 'smithi031' 2024-04-26T18:00:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2618687073 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a0ba250280 0x55a0ba2526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0df3c7c740 4 client.25424 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:00:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.369+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2618687073 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a0ba13bee0 con 0x55a0ba250280 2024-04-26T18:00:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.370+0000 7f0dea59c640 1 --2- 172.21.15.31:0/2618687073 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a0ba250280 0x55a0ba2526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.370+0000 7f0dd1ffb640 10 client.25424 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:29.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.370+0000 7f0dea59c640 1 --2- 172.21.15.31:0/2618687073 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a0ba250280 0x55a0ba2526a0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb4c3eb048218fe5 server_cookie=398d41226b2a39ce in_seq=0 out_seq=0 2024-04-26T18:00:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0dd1ffb640 1 -- 172.21.15.31:0/2618687073 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a0ba13bee0 con 0x55a0ba250280 2024-04-26T18:00:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0dd1ffb640 10 client.25424 handle_command_reply: tid=0 2024-04-26T18:00:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0df3c7c740 1 client.25424 shutdown 2024-04-26T18:00:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2618687073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1ce820 msgr2=0x55a0ba1d3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2618687073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d3650 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0de0003fa0 tx=0x7f0de0009240 comp rx=0 tx=0).stop 2024-04-26T18:00:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2618687073 shutdown_connections 2024-04-26T18:00:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2618687073 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a0ba250280 0x55a0ba2526a0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2618687073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0ba1d90e0 0x55a0ba1d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0dd1ffb640 0 client.25424 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2618687073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a0ba1ce820 0x55a0ba1d3650 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0df3c7c740 1 --2- 172.21.15.31:0/2618687073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0ba1cdf00 0x55a0ba1d3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2618687073 >> 172.21.15.31:0/2618687073 conn(0x55a0ba1ca410 msgr2=0x55a0ba1e38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2618687073 shutdown_connections 2024-04-26T18:00:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0df3c7c740 1 -- 172.21.15.31:0/2618687073 wait complete. 2024-04-26T18:00:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.381+0000 7f0df3c7c740 20 client.25424 trim_cache size 0 max 16384 2024-04-26T18:00:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:00:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:00:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:00:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:00:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.382+0000 7f0df0f5f640 1 -- 172.21.15.31:0/2738921137 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f0dd40a3df0 msgr2=0x7f0dd40a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.382+0000 7f0df0f5f640 1 --2- 172.21.15.31:0/2738921137 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f0dd40a3df0 0x7f0dd40a62b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x55a0ba1cb830 tx=0x7f0de4009b70 comp rx=0 tx=0).stop 2024-04-26T18:00:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.383+0000 7f0df0f5f640 1 -- 172.21.15.31:0/2738921137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec104b30 msgr2=0x7f0dec1102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.383+0000 7f0df0f5f640 1 --2- 172.21.15.31:0/2738921137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec104b30 0x7f0dec1102b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc02a9e0 tx=0x7f0ddc003c10 comp rx=0 tx=0).stop 2024-04-26T18:00:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.384+0000 7f0df0f5f640 1 -- 172.21.15.31:0/2738921137 shutdown_connections 2024-04-26T18:00:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.384+0000 7f0df0f5f640 1 --2- 172.21.15.31:0/2738921137 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f0dd40a3df0 0x7f0dd40a62b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.384+0000 7f0df0f5f640 1 --2- 172.21.15.31:0/2738921137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0dec10abf0 0x7f0dec1ab220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.384+0000 7f0df0f5f640 1 --2- 172.21.15.31:0/2738921137 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0dec1054f0 0x7f0dec1107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.384+0000 7f0df0f5f640 1 --2- 172.21.15.31:0/2738921137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0dec104b30 0x7f0dec1102b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:29.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.384+0000 7f0df0f5f640 1 -- 172.21.15.31:0/2738921137 >> 172.21.15.31:0/2738921137 conn(0x7f0dec100620 msgr2=0x7f0dec109d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:29.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.385+0000 7f0df0f5f640 1 -- 172.21.15.31:0/2738921137 shutdown_connections 2024-04-26T18:00:29.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:29.385+0000 7f0df0f5f640 1 -- 172.21.15.31:0/2738921137 wait complete. 2024-04-26T18:00:29.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:29 smithi186 ceph-mon[32797]: pgmap v122: 97 pgs: 97 active+clean; 2.2 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 953 op/s 2024-04-26T18:00:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:29 smithi031 ceph-mon[29164]: pgmap v122: 97 pgs: 97 active+clean; 2.2 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 953 op/s 2024-04-26T18:00:29.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:29 smithi097 ceph-mon[32419]: pgmap v122: 97 pgs: 97 active+clean; 2.2 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 953 op/s 2024-04-26T18:00:31.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:31 smithi186 ceph-mon[32797]: pgmap v123: 97 pgs: 97 active+clean; 2.2 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.9 MiB/s wr, 743 op/s 2024-04-26T18:00:31.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:31 smithi031 ceph-mon[29164]: pgmap v123: 97 pgs: 97 active+clean; 2.2 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.9 MiB/s wr, 743 op/s 2024-04-26T18:00:31.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:31 smithi097 ceph-mon[32419]: pgmap v123: 97 pgs: 97 active+clean; 2.2 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.9 MiB/s wr, 743 op/s 2024-04-26T18:00:33.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 190.0 s, 2927.4 tps, lat 33.848 ms stddev 36.975 2024-04-26T18:00:33.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:33 smithi186 ceph-mon[32797]: pgmap v124: 97 pgs: 97 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 13 MiB/s wr, 1.04k op/s 2024-04-26T18:00:33.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:33 smithi031 ceph-mon[29164]: pgmap v124: 97 pgs: 97 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 13 MiB/s wr, 1.04k op/s 2024-04-26T18:00:33.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:33 smithi097 ceph-mon[32419]: pgmap v124: 97 pgs: 97 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 13 MiB/s wr, 1.04k op/s 2024-04-26T18:00:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:35 smithi186 ceph-mon[32797]: pgmap v125: 97 pgs: 97 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.3 MiB/s wr, 597 op/s 2024-04-26T18:00:35.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:35 smithi031 ceph-mon[29164]: pgmap v125: 97 pgs: 97 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.3 MiB/s wr, 597 op/s 2024-04-26T18:00:35.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:35 smithi097 ceph-mon[32419]: pgmap v125: 97 pgs: 97 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.3 MiB/s wr, 597 op/s 2024-04-26T18:00:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:37 smithi186 ceph-mon[32797]: pgmap v126: 97 pgs: 97 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.3 MiB/s wr, 695 op/s 2024-04-26T18:00:37.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:37 smithi031 ceph-mon[29164]: pgmap v126: 97 pgs: 97 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.3 MiB/s wr, 695 op/s 2024-04-26T18:00:37.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:37 smithi097 ceph-mon[32419]: pgmap v126: 97 pgs: 97 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.3 MiB/s wr, 695 op/s 2024-04-26T18:00:37.808 INFO:tasks.quiescer.fs.[cephfs]:Total duration (39.0) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-26T18:00:37.808 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.5 2024-04-26T18:00:37.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T18:00:38.095 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 195.0 s, 3034.7 tps, lat 33.248 ms stddev 35.508 2024-04-26T18:00:38.182 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:00:38.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:38 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:00:38.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:38 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:00:38.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:38 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:00:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:39 smithi186 ceph-mon[32797]: pgmap v127: 97 pgs: 97 active+clean; 2.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 11 MiB/s wr, 809 op/s 2024-04-26T18:00:39.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:39 smithi031 ceph-mon[29164]: pgmap v127: 97 pgs: 97 active+clean; 2.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 11 MiB/s wr, 809 op/s 2024-04-26T18:00:39.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:39 smithi097 ceph-mon[32419]: pgmap v127: 97 pgs: 97 active+clean; 2.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 11 MiB/s wr, 809 op/s 2024-04-26T18:00:40.427 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:40 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:00:40.427 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:40 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:00:40.427 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:40 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:00:40.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:40 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:00:40.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:40 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:00:40.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:40 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:00:40.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.580+0000 7f4528e0c700 1 -- 172.21.15.31:0/1568950753 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f452410a680 msgr2=0x7f452410aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:40.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.580+0000 7f4528e0c700 1 --2- 172.21.15.31:0/1568950753 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f452410a680 0x7f452410aa60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f450c009920 tx=0x7f450c009c30 comp rx=0 tx=0).stop 2024-04-26T18:00:40.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.581+0000 7f4528e0c700 1 -- 172.21.15.31:0/1568950753 shutdown_connections 2024-04-26T18:00:40.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.581+0000 7f4528e0c700 1 --2- 172.21.15.31:0/1568950753 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f452410a680 0x7f452410aa60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:40.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.581+0000 7f4528e0c700 1 --2- 172.21.15.31:0/1568950753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410f010 0x7f452410f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:40.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.581+0000 7f4528e0c700 1 --2- 172.21.15.31:0/1568950753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410afa0 0x7f452410ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:40.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.581+0000 7f4528e0c700 1 -- 172.21.15.31:0/1568950753 >> 172.21.15.31:0/1568950753 conn(0x7f452406e590 msgr2=0x7f452406e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:40.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.581+0000 7f4528e0c700 1 -- 172.21.15.31:0/1568950753 shutdown_connections 2024-04-26T18:00:40.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.581+0000 7f4528e0c700 1 -- 172.21.15.31:0/1568950753 wait complete. 2024-04-26T18:00:40.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.582+0000 7f4528e0c700 1 Processor -- start 2024-04-26T18:00:40.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.582+0000 7f4528e0c700 1 -- start start 2024-04-26T18:00:40.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.583+0000 7f4528e0c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410a680 0x7f4524118430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:40.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.583+0000 7f4528e0c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410afa0 0x7f4524118970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:40.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.583+0000 7f4528e0c700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f452410f010 0x7f4524118eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:40.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.583+0000 7f4528e0c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f452411be80 con 0x7f452410afa0 2024-04-26T18:00:40.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.583+0000 7f4528e0c700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f452411bd00 con 0x7f452410f010 2024-04-26T18:00:40.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.583+0000 7f4528e0c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f452411c000 con 0x7f452410a680 2024-04-26T18:00:40.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.583+0000 7f452259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410a680 0x7f4524118430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:40.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.583+0000 7f452259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410a680 0x7f4524118430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47310/0 (socket says 172.21.15.31:47310) 2024-04-26T18:00:40.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.583+0000 7f452259c700 1 -- 172.21.15.31:0/3543842988 learned_addr learned my addr 172.21.15.31:0/3543842988 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:40.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.583+0000 7f4522d9d700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f452410f010 0x7f4524118eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.583+0000 7f451bfff700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410afa0 0x7f4524118970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.584+0000 7f452259c700 1 -- 172.21.15.31:0/3543842988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f452410f010 msgr2=0x7f4524118eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.584+0000 7f452259c700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f452410f010 0x7f4524118eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.584+0000 7f452259c700 1 -- 172.21.15.31:0/3543842988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410afa0 msgr2=0x7f4524118970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.584+0000 7f452259c700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410afa0 0x7f4524118970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.584+0000 7f452259c700 1 -- 172.21.15.31:0/3543842988 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45241b4da0 con 0x7f452410a680 2024-04-26T18:00:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.584+0000 7f451bfff700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410afa0 0x7f4524118970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:00:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.584+0000 7f452259c700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410a680 0x7f4524118430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f450c000c00 tx=0x7f450c00f990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.584+0000 7f451b7fe700 1 -- 172.21.15.31:0/3543842988 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450c027070 con 0x7f452410a680 2024-04-26T18:00:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.584+0000 7f451b7fe700 1 -- 172.21.15.31:0/3543842988 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f450c0055f0 con 0x7f452410a680 2024-04-26T18:00:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.584+0000 7f451b7fe700 1 -- 172.21.15.31:0/3543842988 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450c0049a0 con 0x7f452410a680 2024-04-26T18:00:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.584+0000 7f4528e0c700 1 -- 172.21.15.31:0/3543842988 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45241b4fd0 con 0x7f452410a680 2024-04-26T18:00:40.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.584+0000 7f4528e0c700 1 -- 172.21.15.31:0/3543842988 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45241b5370 con 0x7f452410a680 2024-04-26T18:00:40.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.586+0000 7f4528e0c700 1 -- 172.21.15.31:0/3543842988 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f452410b640 con 0x7f452410a680 2024-04-26T18:00:40.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.588+0000 7f451b7fe700 1 -- 172.21.15.31:0/3543842988 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f450c02a070 con 0x7f452410a680 2024-04-26T18:00:40.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.589+0000 7f451b7fe700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f45100a3e30 0x7f45100a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:40.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.589+0000 7f451bfff700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f45100a3e30 0x7f45100a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:40.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.589+0000 7f451b7fe700 1 -- 172.21.15.31:0/3543842988 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f450c0d1050 con 0x7f452410a680 2024-04-26T18:00:40.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.589+0000 7f451bfff700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f45100a3e30 0x7f45100a62f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4524119bf0 tx=0x7f4514009310 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:40.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:40.594+0000 7f451b7fe700 1 -- 172.21.15.31:0/3543842988 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f450c05eff0 con 0x7f452410a680 2024-04-26T18:00:40.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:40 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:00:40.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:40 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:00:40.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:40 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:00:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:41.012+0000 7f4528e0c700 1 -- 172.21.15.31:0/3543842988 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f452406a110 con 0x7f452410a680 2024-04-26T18:00:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:41.013+0000 7f451b7fe700 1 -- 172.21.15.31:0/3543842988 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11877 (secure 0 0 0) 0x7f450c013030 con 0x7f452410a680 2024-04-26T18:00:41.014 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:00:41.014 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T18:00:06.847155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24446,24457,24473]},"id":1}]} 2024-04-26T18:00:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:41.016+0000 7f4528e0c700 1 -- 172.21.15.31:0/3543842988 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f45100a3e30 msgr2=0x7f45100a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:41.016+0000 7f4528e0c700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f45100a3e30 0x7f45100a62f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4524119bf0 tx=0x7f4514009310 comp rx=0 tx=0).stop 2024-04-26T18:00:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:41.016+0000 7f4528e0c700 1 -- 172.21.15.31:0/3543842988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410a680 msgr2=0x7f4524118430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:41.016+0000 7f4528e0c700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410a680 0x7f4524118430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f450c000c00 tx=0x7f450c00f990 comp rx=0 tx=0).stop 2024-04-26T18:00:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:41.016+0000 7f4528e0c700 1 -- 172.21.15.31:0/3543842988 shutdown_connections 2024-04-26T18:00:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:41.016+0000 7f4528e0c700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f45100a3e30 0x7f45100a62f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:41.016+0000 7f4528e0c700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f452410f010 0x7f4524118eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:41.016+0000 7f4528e0c700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410a680 0x7f4524118430 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:41.016+0000 7f4528e0c700 1 --2- 172.21.15.31:0/3543842988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410afa0 0x7f4524118970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:41.017+0000 7f4528e0c700 1 -- 172.21.15.31:0/3543842988 >> 172.21.15.31:0/3543842988 conn(0x7f452406e590 msgr2=0x7f4524073fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:41.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:41.017+0000 7f4528e0c700 1 -- 172.21.15.31:0/3543842988 shutdown_connections 2024-04-26T18:00:41.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:41.017+0000 7f4528e0c700 1 -- 172.21.15.31:0/3543842988 wait complete. 2024-04-26T18:00:41.018 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 32 2024-04-26T18:00:41.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:41 smithi031 ceph-mon[29164]: pgmap v128: 97 pgs: 97 active+clean; 2.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.1 MiB/s wr, 603 op/s 2024-04-26T18:00:41.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:41 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3543842988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:00:41.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:41 smithi097 ceph-mon[32419]: pgmap v128: 97 pgs: 97 active+clean; 2.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.1 MiB/s wr, 603 op/s 2024-04-26T18:00:41.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:41 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3543842988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:00:41.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:41 smithi186 ceph-mon[32797]: pgmap v128: 97 pgs: 97 active+clean; 2.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.1 MiB/s wr, 603 op/s 2024-04-26T18:00:41.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:41 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3543842988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:00:41.837 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24473 quiesce db / --timeout 60 --expiration 79.5 --await-for 42.0' 2024-04-26T18:00:41.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24473 quiesce db / --timeout 60 --expiration 79.5 --await-for 42.0 2024-04-26T18:00:42.189 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:00:42.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.523+0000 7f0cb0914640 1 -- 172.21.15.31:0/3857001719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cac107840 msgr2=0x7f0cac107c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:42.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.523+0000 7f0cb0914640 1 --2- 172.21.15.31:0/3857001719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cac107840 0x7f0cac107c20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c009cd0 tx=0x7f0c9c02a8e0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.523+0000 7f0cb0914640 1 -- 172.21.15.31:0/3857001719 shutdown_connections 2024-04-26T18:00:42.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.523+0000 7f0cb0914640 1 --2- 172.21.15.31:0/3857001719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cac102240 0x7f0cac10e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.523+0000 7f0cb0914640 1 --2- 172.21.15.31:0/3857001719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0cac101880 0x7f0cac101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.523+0000 7f0cb0914640 1 --2- 172.21.15.31:0/3857001719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cac107840 0x7f0cac107c20 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.523+0000 7f0cb0914640 1 -- 172.21.15.31:0/3857001719 >> 172.21.15.31:0/3857001719 conn(0x7f0cac078980 msgr2=0x7f0cac078d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:42.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.524+0000 7f0cb0914640 1 -- 172.21.15.31:0/3857001719 shutdown_connections 2024-04-26T18:00:42.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.524+0000 7f0cb0914640 1 -- 172.21.15.31:0/3857001719 wait complete. 2024-04-26T18:00:42.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.525+0000 7f0cb0914640 1 Processor -- start 2024-04-26T18:00:42.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.525+0000 7f0cb0914640 1 -- start start 2024-04-26T18:00:42.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.525+0000 7f0cb0914640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cac101880 0x7f0cac196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.525+0000 7f0cb0914640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0cac102240 0x7f0cac197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.525+0000 7f0cb0914640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cac107840 0x7f0cac19b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.525+0000 7f0cb0914640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac1109f0 con 0x7f0cac107840 2024-04-26T18:00:42.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.525+0000 7f0cb0914640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac110870 con 0x7f0cac102240 2024-04-26T18:00:42.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.525+0000 7f0cb0914640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac110b70 con 0x7f0cac101880 2024-04-26T18:00:42.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0caad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cac107840 0x7f0cac19b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0caa59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cac101880 0x7f0cac196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0caad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cac107840 0x7f0cac19b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44486/0 (socket says 172.21.15.31:44486) 2024-04-26T18:00:42.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0caad9d640 1 -- 172.21.15.31:0/2881929 learned_addr learned my addr 172.21.15.31:0/2881929 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:42.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0ca9d9b640 1 --2- 172.21.15.31:0/2881929 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0cac102240 0x7f0cac197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0caad9d640 1 -- 172.21.15.31:0/2881929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cac101880 msgr2=0x7f0cac196e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:42.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0caad9d640 1 --2- 172.21.15.31:0/2881929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cac101880 0x7f0cac196e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0caad9d640 1 -- 172.21.15.31:0/2881929 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0cac102240 msgr2=0x7f0cac197350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0caad9d640 1 --2- 172.21.15.31:0/2881929 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0cac102240 0x7f0cac197350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0caad9d640 1 -- 172.21.15.31:0/2881929 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cac19be00 con 0x7f0cac107840 2024-04-26T18:00:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0ca9d9b640 1 --2- 172.21.15.31:0/2881929 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0cac102240 0x7f0cac197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:00:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0caad9d640 1 --2- 172.21.15.31:0/2881929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cac107840 0x7f0cac19b6d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0ca400c970 tx=0x7f0ca400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0c9b7fe640 1 -- 172.21.15.31:0/2881929 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4007bb0 con 0x7f0cac107840 2024-04-26T18:00:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0cb0914640 1 -- 172.21.15.31:0/2881929 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cac19c0f0 con 0x7f0cac107840 2024-04-26T18:00:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0cb0914640 1 -- 172.21.15.31:0/2881929 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cac1a7d90 con 0x7f0cac107840 2024-04-26T18:00:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0c9b7fe640 1 -- 172.21.15.31:0/2881929 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ca4007d50 con 0x7f0cac107840 2024-04-26T18:00:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.526+0000 7f0c9b7fe640 1 -- 172.21.15.31:0/2881929 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca400f450 con 0x7f0cac107840 2024-04-26T18:00:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.529+0000 7f0cb362d740 1 Processor -- start 2024-04-26T18:00:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.531+0000 7f0c9b7fe640 1 -- 172.21.15.31:0/2881929 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0ca400f5f0 con 0x7f0cac107840 2024-04-26T18:00:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0c9b7fe640 1 --2- 172.21.15.31:0/2881929 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f0c940a3e40 0x7f0c940a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0c9b7fe640 1 -- 172.21.15.31:0/2881929 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f0ca40d0a60 con 0x7f0cac107840 2024-04-26T18:00:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0cb362d740 1 -- start start 2024-04-26T18:00:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0cb362d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8233710 0x55bdc8233af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0cb362d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8234030 0x55bdc823e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0cb362d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc823e8f0 0x55bdc8240ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0cb362d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bdc8191460 con 0x55bdc823e8f0 2024-04-26T18:00:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0cb362d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55bdc814b590 con 0x55bdc8234030 2024-04-26T18:00:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0cb362d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bdc818d420 con 0x55bdc8233710 2024-04-26T18:00:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0ca9d9b640 1 --2- 172.21.15.31:0/2881929 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f0c940a3e40 0x7f0c940a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0caa59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8233710 0x55bdc8233af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0caa59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8233710 0x55bdc8233af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47348/0 (socket says 172.21.15.31:47348) 2024-04-26T18:00:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0caa59c640 1 -- 172.21.15.31:0/449944123 learned_addr learned my addr 172.21.15.31:0/449944123 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0caad9d640 1 --2- 172.21.15.31:0/449944123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc823e8f0 0x55bdc8240ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.532+0000 7f0ca9d9b640 1 --2- 172.21.15.31:0/449944123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8234030 0x55bdc823e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.533+0000 7f0caa59c640 1 -- 172.21.15.31:0/449944123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8234030 msgr2=0x55bdc823e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.533+0000 7f0caa59c640 1 --2- 172.21.15.31:0/449944123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8234030 0x55bdc823e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.533+0000 7f0caa59c640 1 -- 172.21.15.31:0/449944123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc823e8f0 msgr2=0x55bdc8240ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.533+0000 7f0caa59c640 1 --2- 172.21.15.31:0/449944123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc823e8f0 0x55bdc8240ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.533+0000 7f0caa59c640 1 -- 172.21.15.31:0/449944123 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdc81580d0 con 0x55bdc8233710 2024-04-26T18:00:42.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.533+0000 7f0ca9d9b640 1 --2- 172.21.15.31:0/449944123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8234030 0x55bdc823e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:00:42.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.533+0000 7f0caa59c640 1 --2- 172.21.15.31:0/449944123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8233710 0x55bdc8233af0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c02ae00 tx=0x7f0c9c0098b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dccdd0e32571462 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.534+0000 7f0c98ff9640 1 -- 172.21.15.31:0/449944123 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c049070 con 0x55bdc8233710 2024-04-26T18:00:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.534+0000 7f0c98ff9640 1 -- 172.21.15.31:0/449944123 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c9c036040 con 0x55bdc8233710 2024-04-26T18:00:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.534+0000 7f0c98ff9640 1 -- 172.21.15.31:0/449944123 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c031c50 con 0x55bdc8233710 2024-04-26T18:00:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.535+0000 7f0cb362d740 1 -- 172.21.15.31:0/449944123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8233710 msgr2=0x55bdc8233af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.535+0000 7f0ca9d9b640 1 --2- 172.21.15.31:0/2881929 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f0c940a3e40 0x7f0c940a6300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x55bdc8234cb0 tx=0x7f0ca0009560 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.535+0000 7f0cb362d740 1 --2- 172.21.15.31:0/449944123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8233710 0x55bdc8233af0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c02ae00 tx=0x7f0c9c0098b0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.535+0000 7f0cb362d740 1 -- 172.21.15.31:0/449944123 shutdown_connections 2024-04-26T18:00:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.535+0000 7f0cb362d740 1 --2- 172.21.15.31:0/449944123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc823e8f0 0x55bdc8240ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.535+0000 7f0cb362d740 1 --2- 172.21.15.31:0/449944123 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8234030 0x55bdc823e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.535+0000 7f0cb362d740 1 --2- 172.21.15.31:0/449944123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8233710 0x55bdc8233af0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.535+0000 7f0cb362d740 1 -- 172.21.15.31:0/449944123 >> 172.21.15.31:0/449944123 conn(0x55bdc822fc20 msgr2=0x55bdc8230000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.535+0000 7f0cb362d740 1 -- 172.21.15.31:0/449944123 shutdown_connections 2024-04-26T18:00:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.535+0000 7f0cb362d740 1 -- 172.21.15.31:0/449944123 wait complete. 2024-04-26T18:00:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.535+0000 7f0cb362d740 1 -- start start 2024-04-26T18:00:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.535+0000 7f0cb362d740 1 Processor -- start 2024-04-26T18:00:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.535+0000 7f0cb362d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8233710 0x55bdc823b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.535+0000 7f0cb362d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc8234030 0x55bdc823bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0cb362d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc823e8f0 0x55bdc823c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0ca9d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc8234030 0x55bdc823bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0caa59c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8233710 0x55bdc823b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0caa59c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8233710 0x55bdc823b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:36528/0 (socket says 172.21.15.31:36528) 2024-04-26T18:00:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0caa59c640 1 -- 172.21.15.31:0/967875502 learned_addr learned my addr 172.21.15.31:0/967875502 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0cb362d740 1 -- 172.21.15.31:0/967875502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bdc8241510 con 0x55bdc8234030 2024-04-26T18:00:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0cb362d740 1 -- 172.21.15.31:0/967875502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55bdc8118b60 con 0x55bdc8233710 2024-04-26T18:00:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0caad9d640 1 --2- 172.21.15.31:0/967875502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc823e8f0 0x55bdc823c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0cb362d740 1 -- 172.21.15.31:0/967875502 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bdc8241690 con 0x55bdc823e8f0 2024-04-26T18:00:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0cb362d740 10 client.0 _ll_register_callbacks cb 0x55bdc813d2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:00:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0caa59c640 1 -- 172.21.15.31:0/967875502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc823e8f0 msgr2=0x55bdc823c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0caa59c640 1 --2- 172.21.15.31:0/967875502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc823e8f0 0x55bdc823c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0caa59c640 1 -- 172.21.15.31:0/967875502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc8234030 msgr2=0x55bdc823bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0caa59c640 1 --2- 172.21.15.31:0/967875502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc8234030 0x55bdc823bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0caa59c640 1 -- 172.21.15.31:0/967875502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdc7ea3210 con 0x55bdc8233710 2024-04-26T18:00:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0ca9d9b640 1 --2- 172.21.15.31:0/967875502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc8234030 0x55bdc823bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:00:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0cb362d740 1 -- 172.21.15.31:0/967875502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c9c036040 con 0x55bdc8233710 2024-04-26T18:00:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0caa59c640 1 --2- 172.21.15.31:0/967875502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8233710 0x55bdc823b770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c047040 tx=0x7f0c9c031c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa3106ae4899b40f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.536+0000 7f0c91ffb640 10 client.25442 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T18:00:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.537+0000 7f0c91ffb640 1 -- 172.21.15.31:0/967875502 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c054060 con 0x55bdc8233710 2024-04-26T18:00:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.537+0000 7f0c91ffb640 1 -- 172.21.15.31:0/967875502 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f0c9c02e040 con 0x55bdc8233710 2024-04-26T18:00:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.537+0000 7f0c91ffb640 1 -- 172.21.15.31:0/967875502 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0c9c049030 con 0x55bdc8233710 2024-04-26T18:00:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.537+0000 7f0cb362d740 10 client.25442 fetch_fsmap learned FSMap version 32 2024-04-26T18:00:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.537+0000 7f0cb362d740 1 -- 172.21.15.31:0/967875502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55bdc7f6f620 con 0x55bdc8233710 2024-04-26T18:00:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.537+0000 7f0c91ffb640 1 -- 172.21.15.31:0/967875502 <== mon.2 v2:172.21.15.97:3300/0 4 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f0c9c030070 con 0x55bdc8233710 2024-04-26T18:00:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.537+0000 7f0cb362d740 10 client.25442 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:00:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.537+0000 7f0cb362d740 10 client.25442 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:00:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.537+0000 7f0cb362d740 20 client.25442 populate_metadata read hostname 'smithi031' 2024-04-26T18:00:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.537+0000 7f0cb362d740 1 --2- 172.21.15.31:0/967875502 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bdc824e6f0 0x55bdc8250ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.537+0000 7f0cb362d740 4 client.25442 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:00:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.537+0000 7f0cb362d740 1 -- 172.21.15.31:0/967875502 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdc819f070 con 0x55bdc824e6f0 2024-04-26T18:00:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.538+0000 7f0ca9d9b640 1 --2- 172.21.15.31:0/967875502 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bdc824e6f0 0x55bdc8250ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.539+0000 7f0ca9d9b640 1 --2- 172.21.15.31:0/967875502 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bdc824e6f0 0x55bdc8250ae0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67d3cf5a6656f268 server_cookie=bc5d179715fe26b3 in_seq=0 out_seq=0 2024-04-26T18:00:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.539+0000 7f0c91ffb640 10 client.25442 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.539+0000 7f0c91ffb640 1 -- 172.21.15.31:0/967875502 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bdc819f070 con 0x55bdc824e6f0 2024-04-26T18:00:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.539+0000 7f0c91ffb640 10 client.25442 handle_command_reply: tid=0 2024-04-26T18:00:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.539+0000 7f0cb362d740 1 client.25442 shutdown 2024-04-26T18:00:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.540+0000 7f0cb362d740 1 -- 172.21.15.31:0/967875502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8233710 msgr2=0x55bdc823b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.540+0000 7f0cb362d740 1 --2- 172.21.15.31:0/967875502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8233710 0x55bdc823b770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c047040 tx=0x7f0c9c031c20 comp rx=0 tx=0).stop 2024-04-26T18:00:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.540+0000 7f0cb362d740 1 -- 172.21.15.31:0/967875502 shutdown_connections 2024-04-26T18:00:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.540+0000 7f0cb362d740 1 --2- 172.21.15.31:0/967875502 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bdc824e6f0 0x55bdc8250ae0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.540+0000 7f0c91ffb640 0 client.25442 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.540+0000 7f0cb362d740 1 --2- 172.21.15.31:0/967875502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc823e8f0 0x55bdc823c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.540+0000 7f0cb362d740 1 --2- 172.21.15.31:0/967875502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc8234030 0x55bdc823bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.540+0000 7f0cb362d740 1 --2- 172.21.15.31:0/967875502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8233710 0x55bdc823b770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.540+0000 7f0cb362d740 1 -- 172.21.15.31:0/967875502 >> 172.21.15.31:0/967875502 conn(0x55bdc822fc20 msgr2=0x55bdc82265b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.540+0000 7f0cb362d740 1 -- 172.21.15.31:0/967875502 shutdown_connections 2024-04-26T18:00:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.540+0000 7f0cb362d740 1 -- 172.21.15.31:0/967875502 wait complete. 2024-04-26T18:00:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.540+0000 7f0cb362d740 20 client.25442 trim_cache size 0 max 16384 2024-04-26T18:00:42.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.564+0000 7f0cb362d740 1 Processor -- start 2024-04-26T18:00:42.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.564+0000 7f0cb362d740 1 -- start start 2024-04-26T18:00:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.564+0000 7f0cb362d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8233710 0x55bdc824a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.564+0000 7f0cb362d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc8234030 0x55bdc823da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.564+0000 7f0cb362d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc823e8f0 0x55bdc82360d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.564+0000 7f0cb362d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c010ee0 con 0x55bdc8234030 2024-04-26T18:00:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.564+0000 7f0cb362d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c00e750 con 0x55bdc823e8f0 2024-04-26T18:00:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.564+0000 7f0cb362d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c00c5a0 con 0x55bdc8233710 2024-04-26T18:00:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.564+0000 7f0caad9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc823e8f0 0x55bdc82360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.564+0000 7f0caad9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc823e8f0 0x55bdc82360d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:36534/0 (socket says 172.21.15.31:36534) 2024-04-26T18:00:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.564+0000 7f0caad9d640 1 -- 172.21.15.31:0/2146963085 learned_addr learned my addr 172.21.15.31:0/2146963085 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.565+0000 7f0caad9d640 1 -- 172.21.15.31:0/2146963085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8233710 msgr2=0x55bdc824a400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T18:00:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.565+0000 7f0caad9d640 1 --2- 172.21.15.31:0/2146963085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8233710 0x55bdc824a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.565+0000 7f0caad9d640 1 -- 172.21.15.31:0/2146963085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc8234030 msgr2=0x55bdc823da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.565+0000 7f0caad9d640 1 --2- 172.21.15.31:0/2146963085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc8234030 0x55bdc823da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.565+0000 7f0caad9d640 1 -- 172.21.15.31:0/2146963085 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdc822fa50 con 0x55bdc823e8f0 2024-04-26T18:00:42.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.565+0000 7f0caad9d640 1 --2- 172.21.15.31:0/2146963085 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc823e8f0 0x55bdc82360d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4005940 tx=0x7f0ca4087040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b412dc523434ad40 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:42.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.566+0000 7f0c93fff640 1 -- 172.21.15.31:0/2146963085 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca401d040 con 0x55bdc823e8f0 2024-04-26T18:00:42.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.566+0000 7f0c93fff640 1 -- 172.21.15.31:0/2146963085 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ca40a5660 con 0x55bdc823e8f0 2024-04-26T18:00:42.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.566+0000 7f0c93fff640 1 -- 172.21.15.31:0/2146963085 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca40a51d0 con 0x55bdc823e8f0 2024-04-26T18:00:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.567+0000 7f0cb362d740 1 -- 172.21.15.31:0/2146963085 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc823e8f0 msgr2=0x55bdc82360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.567+0000 7f0cb362d740 1 --2- 172.21.15.31:0/2146963085 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc823e8f0 0x55bdc82360d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4005940 tx=0x7f0ca4087040 comp rx=0 tx=0).stop 2024-04-26T18:00:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.567+0000 7f0cb362d740 1 -- 172.21.15.31:0/2146963085 shutdown_connections 2024-04-26T18:00:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.567+0000 7f0cb362d740 1 --2- 172.21.15.31:0/2146963085 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc823e8f0 0x55bdc82360d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.567+0000 7f0cb362d740 1 --2- 172.21.15.31:0/2146963085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc8234030 0x55bdc823da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.567+0000 7f0cb362d740 1 --2- 172.21.15.31:0/2146963085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8233710 0x55bdc824a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.567+0000 7f0cb362d740 1 -- 172.21.15.31:0/2146963085 >> 172.21.15.31:0/2146963085 conn(0x55bdc822fc20 msgr2=0x55bdc823a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:42.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.568+0000 7f0cb362d740 1 -- 172.21.15.31:0/2146963085 shutdown_connections 2024-04-26T18:00:42.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.568+0000 7f0cb362d740 1 -- 172.21.15.31:0/2146963085 wait complete. 2024-04-26T18:00:42.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.568+0000 7f0cb362d740 1 -- start start 2024-04-26T18:00:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.568+0000 7f0cb362d740 1 Processor -- start 2024-04-26T18:00:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.569+0000 7f0cb362d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8233710 0x55bdc8246910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.569+0000 7f0cb362d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8234030 0x55bdc8246e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.569+0000 7f0cb362d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc823e8f0 0x55bdc82438e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.569+0000 7f0cb362d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c00cc10 con 0x55bdc823e8f0 2024-04-26T18:00:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.569+0000 7f0cb362d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c00e080 con 0x55bdc8233710 2024-04-26T18:00:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.569+0000 7f0cb362d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c00d2e0 con 0x55bdc8234030 2024-04-26T18:00:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.569+0000 7f0cb362d740 10 client.0 _ll_register_callbacks cb 0x7f0c9c047070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:00:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.569+0000 7f0caa59c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8233710 0x55bdc8246910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.569+0000 7f0caa59c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8233710 0x55bdc8246910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:36544/0 (socket says 172.21.15.31:36544) 2024-04-26T18:00:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.569+0000 7f0caa59c640 1 -- 172.21.15.31:0/152593461 learned_addr learned my addr 172.21.15.31:0/152593461 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.569+0000 7f0ca9d9b640 1 --2- 172.21.15.31:0/152593461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8234030 0x55bdc8246e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.570+0000 7f0caa59c640 1 -- 172.21.15.31:0/152593461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8234030 msgr2=0x55bdc8246e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.570+0000 7f0caa59c640 1 --2- 172.21.15.31:0/152593461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8234030 0x55bdc8246e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.570+0000 7f0caa59c640 1 -- 172.21.15.31:0/152593461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc823e8f0 msgr2=0x55bdc82438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.570+0000 7f0caa59c640 1 --2- 172.21.15.31:0/152593461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc823e8f0 0x55bdc82438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.570+0000 7f0caa59c640 1 -- 172.21.15.31:0/152593461 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdc8244110 con 0x55bdc8233710 2024-04-26T18:00:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.570+0000 7f0caa59c640 1 --2- 172.21.15.31:0/152593461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8233710 0x55bdc8246910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c000f80 tx=0x7f0c9c004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb70a5b3535d8dd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.570+0000 7f0c91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T18:00:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.571+0000 7f0c91ffb640 1 -- 172.21.15.31:0/152593461 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c004700 con 0x55bdc8233710 2024-04-26T18:00:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.571+0000 7f0cb362d740 1 -- 172.21.15.31:0/152593461 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ca40a5660 con 0x55bdc8233710 2024-04-26T18:00:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.571+0000 7f0c91ffb640 1 -- 172.21.15.31:0/152593461 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f0c9c041040 con 0x55bdc8233710 2024-04-26T18:00:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.571+0000 7f0c91ffb640 1 -- 172.21.15.31:0/152593461 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0c9c0305a0 con 0x55bdc8233710 2024-04-26T18:00:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.572+0000 7f0cb362d740 10 client.25448 fetch_fsmap learned FSMap version 32 2024-04-26T18:00:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.572+0000 7f0cb362d740 1 -- 172.21.15.31:0/152593461 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bdc82b5520 con 0x55bdc8233710 2024-04-26T18:00:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.572+0000 7f0c91ffb640 1 -- 172.21.15.31:0/152593461 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ca40a5660 con 0x55bdc8233710 2024-04-26T18:00:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.572+0000 7f0c91ffb640 1 -- 172.21.15.31:0/152593461 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f0c9c030040 con 0x55bdc8233710 2024-04-26T18:00:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.572+0000 7f0cb362d740 10 client.25448 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:00:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.572+0000 7f0cb362d740 10 client.25448 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:00:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.573+0000 7f0cb362d740 20 client.25448 populate_metadata read hostname 'smithi031' 2024-04-26T18:00:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.573+0000 7f0cb362d740 1 --2- 172.21.15.31:0/152593461 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bdc824e6f0 0x55bdc82b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.573+0000 7f0cb362d740 4 client.25448 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:00:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.573+0000 7f0cb362d740 1 -- 172.21.15.31:0/152593461 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bdc8198ee0 con 0x55bdc824e6f0 2024-04-26T18:00:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.573+0000 7f0ca9d9b640 1 --2- 172.21.15.31:0/152593461 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bdc824e6f0 0x55bdc82b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.574+0000 7f0ca9d9b640 1 --2- 172.21.15.31:0/152593461 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bdc824e6f0 0x55bdc82b7c20 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=791f1ba16e2365ab server_cookie=2a4443b0a65f912c in_seq=0 out_seq=0 2024-04-26T18:00:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.574+0000 7f0c91ffb640 10 client.25448 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.592+0000 7f0c91ffb640 1 -- 172.21.15.31:0/152593461 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bdc8198ee0 con 0x55bdc824e6f0 2024-04-26T18:00:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.592+0000 7f0c91ffb640 10 client.25448 handle_command_reply: tid=0 2024-04-26T18:00:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.592+0000 7f0cb362d740 1 client.25448 shutdown 2024-04-26T18:00:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.594+0000 7f0cb362d740 1 -- 172.21.15.31:0/152593461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8233710 msgr2=0x55bdc8246910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.594+0000 7f0cb362d740 1 --2- 172.21.15.31:0/152593461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8233710 0x55bdc8246910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c000f80 tx=0x7f0c9c004a80 comp rx=0 tx=0).stop 2024-04-26T18:00:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.594+0000 7f0cb362d740 1 -- 172.21.15.31:0/152593461 shutdown_connections 2024-04-26T18:00:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.594+0000 7f0cb362d740 1 --2- 172.21.15.31:0/152593461 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bdc824e6f0 0x55bdc82b7c20 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.594+0000 7f0cb362d740 1 --2- 172.21.15.31:0/152593461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdc823e8f0 0x55bdc82438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.594+0000 7f0cb362d740 1 --2- 172.21.15.31:0/152593461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bdc8234030 0x55bdc8246e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.594+0000 7f0cb362d740 1 --2- 172.21.15.31:0/152593461 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bdc8233710 0x55bdc8246910 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.594+0000 7f0cb362d740 1 -- 172.21.15.31:0/152593461 >> 172.21.15.31:0/152593461 conn(0x55bdc822fc20 msgr2=0x55bdc8239f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.594+0000 7f0c91ffb640 0 client.25448 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.594+0000 7f0cb362d740 1 -- 172.21.15.31:0/152593461 shutdown_connections 2024-04-26T18:00:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.594+0000 7f0cb362d740 1 -- 172.21.15.31:0/152593461 wait complete. 2024-04-26T18:00:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.594+0000 7f0cb362d740 20 client.25448 trim_cache size 0 max 16384 2024-04-26T18:00:42.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:00:42.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:00:42.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:00:42.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:00:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.595+0000 7f0cb0914640 1 -- 172.21.15.31:0/2881929 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f0c940a3e40 msgr2=0x7f0c940a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.595+0000 7f0cb0914640 1 --2- 172.21.15.31:0/2881929 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f0c940a3e40 0x7f0c940a6300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x55bdc8234cb0 tx=0x7f0ca0009560 comp rx=0 tx=0).stop 2024-04-26T18:00:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.595+0000 7f0cb0914640 1 -- 172.21.15.31:0/2881929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cac107840 msgr2=0x7f0cac19b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.595+0000 7f0cb0914640 1 --2- 172.21.15.31:0/2881929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cac107840 0x7f0cac19b6d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0ca400c970 tx=0x7f0ca400ce30 comp rx=0 tx=0).stop 2024-04-26T18:00:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.595+0000 7f0cb0914640 1 -- 172.21.15.31:0/2881929 shutdown_connections 2024-04-26T18:00:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.595+0000 7f0cb0914640 1 --2- 172.21.15.31:0/2881929 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f0c940a3e40 0x7f0c940a6300 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.595+0000 7f0cb0914640 1 --2- 172.21.15.31:0/2881929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cac107840 0x7f0cac19b6d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.595+0000 7f0cb0914640 1 --2- 172.21.15.31:0/2881929 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0cac102240 0x7f0cac197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.595+0000 7f0cb0914640 1 --2- 172.21.15.31:0/2881929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cac101880 0x7f0cac196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.595+0000 7f0cb0914640 1 -- 172.21.15.31:0/2881929 >> 172.21.15.31:0/2881929 conn(0x7f0cac078980 msgr2=0x7f0cac104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.595+0000 7f0cb0914640 1 -- 172.21.15.31:0/2881929 shutdown_connections 2024-04-26T18:00:42.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:42.595+0000 7f0cb0914640 1 -- 172.21.15.31:0/2881929 wait complete. 2024-04-26T18:00:43.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 200.0 s, 3313.6 tps, lat 30.101 ms stddev 32.614 2024-04-26T18:00:43.549 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:43 smithi031 ceph-mon[29164]: pgmap v129: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 MiB/s wr, 902 op/s 2024-04-26T18:00:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:43 smithi097 ceph-mon[32419]: pgmap v129: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 MiB/s wr, 902 op/s 2024-04-26T18:00:43.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:43 smithi186 ceph-mon[32797]: pgmap v129: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 MiB/s wr, 902 op/s 2024-04-26T18:00:44.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.430+0000 7ff200675700 1 -- 172.21.15.31:0/204836482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1f8106020 msgr2=0x7ff1f8112530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:44.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.430+0000 7ff200675700 1 --2- 172.21.15.31:0/204836482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1f8106020 0x7ff1f8112530 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40098f0 tx=0x7ff1f4009c00 comp rx=0 tx=0).stop 2024-04-26T18:00:44.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.431+0000 7ff200675700 1 -- 172.21.15.31:0/204836482 shutdown_connections 2024-04-26T18:00:44.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.431+0000 7ff200675700 1 --2- 172.21.15.31:0/204836482 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1f8106020 0x7ff1f8112530 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.431+0000 7ff200675700 1 --2- 172.21.15.31:0/204836482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f810b660 0x7ff1f810ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.431+0000 7ff200675700 1 --2- 172.21.15.31:0/204836482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1f8105660 0x7ff1f8105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.431+0000 7ff200675700 1 -- 172.21.15.31:0/204836482 >> 172.21.15.31:0/204836482 conn(0x7ff1f8101150 msgr2=0x7ff1f8103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:44.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.431+0000 7ff200675700 1 -- 172.21.15.31:0/204836482 shutdown_connections 2024-04-26T18:00:44.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.431+0000 7ff200675700 1 -- 172.21.15.31:0/204836482 wait complete. 2024-04-26T18:00:44.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.431+0000 7ff200675700 1 Processor -- start 2024-04-26T18:00:44.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.431+0000 7ff200675700 1 -- start start 2024-04-26T18:00:44.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff200675700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1f8105660 0x7ff1f81a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff200675700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f8106020 0x7ff1f81a1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff200675700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1f810b660 0x7ff1f819ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff200675700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f81158f0 con 0x7ff1f810b660 2024-04-26T18:00:44.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff200675700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f8115770 con 0x7ff1f8105660 2024-04-26T18:00:44.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff200675700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f8115a70 con 0x7ff1f8106020 2024-04-26T18:00:44.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff1fe411700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1f8105660 0x7ff1f81a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff1fec12700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1f810b660 0x7ff1f819ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff1fe411700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1f8105660 0x7ff1f81a18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:36564/0 (socket says 172.21.15.31:36564) 2024-04-26T18:00:44.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff1fe411700 1 -- 172.21.15.31:0/431387623 learned_addr learned my addr 172.21.15.31:0/431387623 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:44.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff1fec12700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1f810b660 0x7ff1f819ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44512/0 (socket says 172.21.15.31:44512) 2024-04-26T18:00:44.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff1fdc10700 1 --2- 172.21.15.31:0/431387623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f8106020 0x7ff1f81a1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff1fe411700 1 -- 172.21.15.31:0/431387623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f8106020 msgr2=0x7ff1f81a1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:44.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff1fe411700 1 --2- 172.21.15.31:0/431387623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f8106020 0x7ff1f81a1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff1fe411700 1 -- 172.21.15.31:0/431387623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1f810b660 msgr2=0x7ff1f819ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:44.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff1fe411700 1 --2- 172.21.15.31:0/431387623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1f810b660 0x7ff1f819ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.432+0000 7ff1fe411700 1 -- 172.21.15.31:0/431387623 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f819c1f0 con 0x7ff1f8105660 2024-04-26T18:00:44.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.433+0000 7ff1fe411700 1 --2- 172.21.15.31:0/431387623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1f8105660 0x7ff1f81a18d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800e9a0 tx=0x7ff1e800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:44.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.433+0000 7ff1ef7fe700 1 -- 172.21.15.31:0/431387623 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e800cbd0 con 0x7ff1f8105660 2024-04-26T18:00:44.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.433+0000 7ff1ef7fe700 1 -- 172.21.15.31:0/431387623 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1e800cd70 con 0x7ff1f8105660 2024-04-26T18:00:44.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.433+0000 7ff1ef7fe700 1 -- 172.21.15.31:0/431387623 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8018680 con 0x7ff1f8105660 2024-04-26T18:00:44.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.433+0000 7ff200675700 1 -- 172.21.15.31:0/431387623 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1f819c4e0 con 0x7ff1f8105660 2024-04-26T18:00:44.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.433+0000 7ff200675700 1 -- 172.21.15.31:0/431387623 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1f81a87a0 con 0x7ff1f8105660 2024-04-26T18:00:44.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.436+0000 7ff1ef7fe700 1 -- 172.21.15.31:0/431387623 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff1e8010430 con 0x7ff1f8105660 2024-04-26T18:00:44.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.437+0000 7ff1ef7fe700 1 --2- 172.21.15.31:0/431387623 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ff1e40a3e50 0x7ff1e40a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.438+0000 7ff1fdc10700 1 --2- 172.21.15.31:0/431387623 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ff1e40a3e50 0x7ff1e40a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.438+0000 7ff1ef7fe700 1 -- 172.21.15.31:0/431387623 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7ff1e80d0970 con 0x7ff1f8105660 2024-04-26T18:00:44.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.438+0000 7ff1fdc10700 1 --2- 172.21.15.31:0/431387623 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ff1e40a3e50 0x7ff1e40a6310 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff1f810ac20 tx=0x7ff1f000b040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:44.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.438+0000 7ff20af07b80 1 Processor -- start 2024-04-26T18:00:44.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff20af07b80 1 -- start start 2024-04-26T18:00:44.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff20af07b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2230 0x55d2d1ba2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff20af07b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1ba2b50 0x55d2d1bad4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff20af07b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1bada00 0x55d2d1bafdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff20af07b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d2d1af58d0 con 0x55d2d1ba2b50 2024-04-26T18:00:44.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff20af07b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55d2d1a94770 con 0x55d2d1ba2230 2024-04-26T18:00:44.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff20af07b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d2d1af1890 con 0x55d2d1bada00 2024-04-26T18:00:44.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff1fe411700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2230 0x55d2d1ba2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff1fdc10700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1ba2b50 0x55d2d1bad4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff1fdc10700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1ba2b50 0x55d2d1bad4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44524/0 (socket says 172.21.15.31:44524) 2024-04-26T18:00:44.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff1fdc10700 1 -- 172.21.15.31:0/1031635704 learned_addr learned my addr 172.21.15.31:0/1031635704 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:44.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff1fe411700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2230 0x55d2d1ba2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:36574/0 (socket says 172.21.15.31:36574) 2024-04-26T18:00:44.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff1fec12700 1 --2- 172.21.15.31:0/1031635704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1bada00 0x55d2d1bafdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff1fdc10700 1 -- 172.21.15.31:0/1031635704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1bada00 msgr2=0x55d2d1bafdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:44.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff1fdc10700 1 --2- 172.21.15.31:0/1031635704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1bada00 0x55d2d1bafdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff1fdc10700 1 -- 172.21.15.31:0/1031635704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2230 msgr2=0x55d2d1ba2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:44.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff1fdc10700 1 --2- 172.21.15.31:0/1031635704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2230 0x55d2d1ba2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.439+0000 7ff1fdc10700 1 -- 172.21.15.31:0/1031635704 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2d1a42a60 con 0x55d2d1ba2b50 2024-04-26T18:00:44.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.440+0000 7ff1fdc10700 1 --2- 172.21.15.31:0/1031635704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1ba2b50 0x55d2d1bad4c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00091e0 tx=0x7ff1f0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2487613d55e4fb66 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:44.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.440+0000 7ff1ecff9700 1 -- 172.21.15.31:0/1031635704 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f001c050 con 0x55d2d1ba2b50 2024-04-26T18:00:44.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.441+0000 7ff1ecff9700 1 -- 172.21.15.31:0/1031635704 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1f0026070 con 0x55d2d1ba2b50 2024-04-26T18:00:44.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.442+0000 7ff20af07b80 1 -- 172.21.15.31:0/1031635704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1ba2b50 msgr2=0x55d2d1bad4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:44.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.442+0000 7ff20af07b80 1 --2- 172.21.15.31:0/1031635704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1ba2b50 0x55d2d1bad4c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00091e0 tx=0x7ff1f0013000 comp rx=0 tx=0).stop 2024-04-26T18:00:44.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.442+0000 7ff20af07b80 1 -- 172.21.15.31:0/1031635704 shutdown_connections 2024-04-26T18:00:44.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.442+0000 7ff20af07b80 1 --2- 172.21.15.31:0/1031635704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2230 0x55d2d1ba2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.442+0000 7ff20af07b80 1 --2- 172.21.15.31:0/1031635704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1bada00 0x55d2d1bafdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.442+0000 7ff20af07b80 1 --2- 172.21.15.31:0/1031635704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1ba2b50 0x55d2d1bad4c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.442+0000 7ff20af07b80 1 -- 172.21.15.31:0/1031635704 >> 172.21.15.31:0/1031635704 conn(0x55d2d1acfed0 msgr2=0x55d2d1b9e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:44.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.443+0000 7ff20af07b80 1 -- 172.21.15.31:0/1031635704 shutdown_connections 2024-04-26T18:00:44.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.443+0000 7ff20af07b80 1 -- 172.21.15.31:0/1031635704 wait complete. 2024-04-26T18:00:44.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.443+0000 7ff20af07b80 1 -- start start 2024-04-26T18:00:44.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.443+0000 7ff20af07b80 1 Processor -- start 2024-04-26T18:00:44.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.443+0000 7ff20af07b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2230 0x55d2d1bad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.443+0000 7ff20af07b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2b50 0x55d2d1baa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.443+0000 7ff20af07b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1bada00 0x55d2d1baa920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.443+0000 7ff20af07b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d2d1bb0670 con 0x55d2d1bada00 2024-04-26T18:00:44.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.443+0000 7ff20af07b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55d2d1a5bae0 con 0x55d2d1ba2230 2024-04-26T18:00:44.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.443+0000 7ff20af07b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d2d1bb07f0 con 0x55d2d1ba2b50 2024-04-26T18:00:44.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.443+0000 7ff20af07b80 10 client.0 _ll_register_callbacks cb 0x55d2d1a89710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:00:44.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1fe411700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2230 0x55d2d1bad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1fe411700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2230 0x55d2d1bad0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:36586/0 (socket says 172.21.15.31:36586) 2024-04-26T18:00:44.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1fe411700 1 -- 172.21.15.31:0/1569366700 learned_addr learned my addr 172.21.15.31:0/1569366700 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:44.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1fec12700 1 --2- 172.21.15.31:0/1569366700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1bada00 0x55d2d1baa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1fdc10700 1 --2- 172.21.15.31:0/1569366700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2b50 0x55d2d1baa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1fe411700 1 -- 172.21.15.31:0/1569366700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2b50 msgr2=0x55d2d1baa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:44.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1fe411700 1 --2- 172.21.15.31:0/1569366700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2b50 0x55d2d1baa3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1fe411700 1 -- 172.21.15.31:0/1569366700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1bada00 msgr2=0x55d2d1baa920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:44.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1fe411700 1 --2- 172.21.15.31:0/1569366700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1bada00 0x55d2d1baa920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1fe411700 1 -- 172.21.15.31:0/1569366700 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2d1baaf60 con 0x55d2d1ba2230 2024-04-26T18:00:44.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1fe411700 1 --2- 172.21.15.31:0/1569366700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2230 0x55d2d1bad0e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff1e80ce710 tx=0x7ff1e809c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b71770906b634444 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1e1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1e1ffb700 1 -- 172.21.15.31:0/1569366700 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e80a0d10 con 0x55d2d1ba2230 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff20af07b80 1 -- 172.21.15.31:0/1569366700 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1f0026070 con 0x55d2d1ba2230 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1e1ffb700 1 -- 172.21.15.31:0/1569366700 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7ff1e80870b0 con 0x55d2d1ba2230 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff1e1ffb700 1 -- 172.21.15.31:0/1569366700 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff1e80a55d0 con 0x55d2d1ba2230 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff20af07b80 10 client.25460 fetch_fsmap learned FSMap version 32 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.444+0000 7ff20af07b80 1 -- 172.21.15.31:0/1569366700 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d2d1bbb160 con 0x55d2d1ba2230 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.445+0000 7ff1e1ffb700 1 -- 172.21.15.31:0/1569366700 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1f0026070 con 0x55d2d1ba2230 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.445+0000 7ff1e1ffb700 1 -- 172.21.15.31:0/1569366700 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ff1e80bbb20 con 0x55d2d1ba2230 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.445+0000 7ff20af07b80 10 client.25460 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.445+0000 7ff20af07b80 10 client.25460 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.445+0000 7ff20af07b80 20 client.25460 populate_metadata read hostname 'smithi031' 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.445+0000 7ff20af07b80 1 --2- 172.21.15.31:0/1569366700 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55d2d1bbd7a0 0x55d2d1bbfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.445+0000 7ff20af07b80 4 client.25460 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:00:44.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.445+0000 7ff20af07b80 1 -- 172.21.15.31:0/1569366700 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2d1b032b0 con 0x55d2d1bbd7a0 2024-04-26T18:00:44.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.446+0000 7ff1fdc10700 1 --2- 172.21.15.31:0/1569366700 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55d2d1bbd7a0 0x55d2d1bbfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.447+0000 7ff1fdc10700 1 --2- 172.21.15.31:0/1569366700 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55d2d1bbd7a0 0x55d2d1bbfbf0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6185ee63b9e7c2b server_cookie=348c71528e96b28c in_seq=0 out_seq=0 2024-04-26T18:00:44.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.447+0000 7ff1e1ffb700 10 client.25460 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:44.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.448+0000 7ff1e1ffb700 1 -- 172.21.15.31:0/1569366700 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d2d1b032b0 con 0x55d2d1bbd7a0 2024-04-26T18:00:44.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.448+0000 7ff1e1ffb700 10 client.25460 handle_command_reply: tid=0 2024-04-26T18:00:44.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.448+0000 7ff20af07b80 1 client.25460 shutdown 2024-04-26T18:00:44.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.448+0000 7ff20af07b80 1 -- 172.21.15.31:0/1569366700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2230 msgr2=0x55d2d1bad0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:44.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.448+0000 7ff20af07b80 1 --2- 172.21.15.31:0/1569366700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2230 0x55d2d1bad0e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff1e80ce710 tx=0x7ff1e809c340 comp rx=0 tx=0).stop 2024-04-26T18:00:44.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.448+0000 7ff20af07b80 1 -- 172.21.15.31:0/1569366700 shutdown_connections 2024-04-26T18:00:44.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.448+0000 7ff20af07b80 1 --2- 172.21.15.31:0/1569366700 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55d2d1bbd7a0 0x55d2d1bbfbf0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.448+0000 7ff20af07b80 1 --2- 172.21.15.31:0/1569366700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2230 0x55d2d1bad0e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.448+0000 7ff20af07b80 1 --2- 172.21.15.31:0/1569366700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1bada00 0x55d2d1baa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.448+0000 7ff20af07b80 1 --2- 172.21.15.31:0/1569366700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2b50 0x55d2d1baa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.448+0000 7ff20af07b80 1 -- 172.21.15.31:0/1569366700 >> 172.21.15.31:0/1569366700 conn(0x55d2d1acfed0 msgr2=0x55d2d1b9e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:44.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.448+0000 7ff1e1ffb700 0 client.25460 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:44.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.448+0000 7ff20af07b80 1 -- 172.21.15.31:0/1569366700 shutdown_connections 2024-04-26T18:00:44.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.448+0000 7ff20af07b80 1 -- 172.21.15.31:0/1569366700 wait complete. 2024-04-26T18:00:44.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.449+0000 7ff20af07b80 20 client.25460 trim_cache size 0 max 16384 2024-04-26T18:00:44.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.486+0000 7ff20af07b80 1 Processor -- start 2024-04-26T18:00:44.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.486+0000 7ff20af07b80 1 -- start start 2024-04-26T18:00:44.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.487+0000 7ff20af07b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1ba2230 0x55d2d1ba9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.487+0000 7ff20af07b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2b50 0x55d2d1ba5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.487+0000 7ff20af07b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1bada00 0x55d2d1ba4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.487+0000 7ff1fe411700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1ba2230 0x55d2d1ba9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.487+0000 7ff1fe411700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1ba2230 0x55d2d1ba9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44538/0 (socket says 172.21.15.31:44538) 2024-04-26T18:00:44.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.487+0000 7ff20af07b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f81b2720 con 0x55d2d1ba2230 2024-04-26T18:00:44.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.487+0000 7ff1fe411700 1 -- 172.21.15.31:0/2662988667 learned_addr learned my addr 172.21.15.31:0/2662988667 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:44.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.487+0000 7ff1fdc10700 1 --2- 172.21.15.31:0/2662988667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2b50 0x55d2d1ba5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.487+0000 7ff1fec12700 1 --2- 172.21.15.31:0/2662988667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1bada00 0x55d2d1ba4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.487+0000 7ff20af07b80 1 -- 172.21.15.31:0/2662988667 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f81aff50 con 0x55d2d1bada00 2024-04-26T18:00:44.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.487+0000 7ff20af07b80 1 -- 172.21.15.31:0/2662988667 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f81addc0 con 0x55d2d1ba2b50 2024-04-26T18:00:44.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.488+0000 7ff1fe411700 1 -- 172.21.15.31:0/2662988667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2b50 msgr2=0x55d2d1ba5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:44.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.488+0000 7ff1fe411700 1 --2- 172.21.15.31:0/2662988667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2b50 0x55d2d1ba5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.488+0000 7ff1fe411700 1 -- 172.21.15.31:0/2662988667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1bada00 msgr2=0x55d2d1ba4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:44.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.488+0000 7ff1fe411700 1 --2- 172.21.15.31:0/2662988667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1bada00 0x55d2d1ba4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.488+0000 7ff1fe411700 1 -- 172.21.15.31:0/2662988667 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2d1bbaf70 con 0x55d2d1ba2230 2024-04-26T18:00:44.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.488+0000 7ff1fe411700 1 --2- 172.21.15.31:0/2662988667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1ba2230 0x55d2d1ba9380 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8098dc0 tx=0x7ff1e80ce1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c057abbe7872ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:44.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.488+0000 7ff1e3fff700 1 -- 172.21.15.31:0/2662988667 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8065070 con 0x55d2d1ba2230 2024-04-26T18:00:44.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.488+0000 7ff1e3fff700 1 -- 172.21.15.31:0/2662988667 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1e80986b0 con 0x55d2d1ba2230 2024-04-26T18:00:44.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.488+0000 7ff1e3fff700 1 -- 172.21.15.31:0/2662988667 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8010980 con 0x55d2d1ba2230 2024-04-26T18:00:44.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.489+0000 7ff20af07b80 1 -- 172.21.15.31:0/2662988667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1ba2230 msgr2=0x55d2d1ba9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:44.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.489+0000 7ff20af07b80 1 --2- 172.21.15.31:0/2662988667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1ba2230 0x55d2d1ba9380 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8098dc0 tx=0x7ff1e80ce1c0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.490+0000 7ff20af07b80 1 -- 172.21.15.31:0/2662988667 shutdown_connections 2024-04-26T18:00:44.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.490+0000 7ff20af07b80 1 --2- 172.21.15.31:0/2662988667 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1bada00 0x55d2d1ba4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.490+0000 7ff20af07b80 1 --2- 172.21.15.31:0/2662988667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1ba2230 0x55d2d1ba9380 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.490+0000 7ff20af07b80 1 --2- 172.21.15.31:0/2662988667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2b50 0x55d2d1ba5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.490+0000 7ff20af07b80 1 -- 172.21.15.31:0/2662988667 >> 172.21.15.31:0/2662988667 conn(0x55d2d1acfed0 msgr2=0x55d2d1ba4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:44.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.491+0000 7ff20af07b80 1 -- 172.21.15.31:0/2662988667 shutdown_connections 2024-04-26T18:00:44.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.491+0000 7ff20af07b80 1 -- 172.21.15.31:0/2662988667 wait complete. 2024-04-26T18:00:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.492+0000 7ff20af07b80 1 -- start start 2024-04-26T18:00:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.492+0000 7ff20af07b80 1 Processor -- start 2024-04-26T18:00:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.492+0000 7ff20af07b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2230 0x55d2d1bb5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.492+0000 7ff20af07b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2b50 0x55d2d1bb6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.492+0000 7ff20af07b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1bada00 0x55d2d1bb2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.492+0000 7ff20af07b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f81ae410 con 0x55d2d1bada00 2024-04-26T18:00:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.492+0000 7ff20af07b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f81af880 con 0x55d2d1ba2b50 2024-04-26T18:00:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.492+0000 7ff20af07b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f81aeae0 con 0x55d2d1ba2230 2024-04-26T18:00:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.492+0000 7ff20af07b80 10 client.0 _ll_register_callbacks cb 0x55d2d1a89710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:00:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.492+0000 7ff1fec12700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1bada00 0x55d2d1bb2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.493+0000 7ff1fec12700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1bada00 0x55d2d1bb2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44540/0 (socket says 172.21.15.31:44540) 2024-04-26T18:00:44.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.493+0000 7ff1fec12700 1 -- 172.21.15.31:0/3610794353 learned_addr learned my addr 172.21.15.31:0/3610794353 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:44.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.493+0000 7ff1fe411700 1 --2- 172.21.15.31:0/3610794353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2230 0x55d2d1bb5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.493+0000 7ff1fdc10700 1 --2- 172.21.15.31:0/3610794353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2b50 0x55d2d1bb6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.493+0000 7ff1fec12700 1 -- 172.21.15.31:0/3610794353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2230 msgr2=0x55d2d1bb5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:44.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.493+0000 7ff1fec12700 1 --2- 172.21.15.31:0/3610794353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2230 0x55d2d1bb5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.493+0000 7ff1fec12700 1 -- 172.21.15.31:0/3610794353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2b50 msgr2=0x55d2d1bb6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:44.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.493+0000 7ff1fec12700 1 --2- 172.21.15.31:0/3610794353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2b50 0x55d2d1bb6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.493+0000 7ff1fec12700 1 -- 172.21.15.31:0/3610794353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2d1bb3090 con 0x55d2d1bada00 2024-04-26T18:00:44.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.493+0000 7ff1fec12700 1 --2- 172.21.15.31:0/3610794353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1bada00 0x55d2d1bb2aa0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40098f0 tx=0x7ff1f400f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e90406346ad26e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:44.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.493+0000 7ff1e1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:00:44.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.493+0000 7ff1e1ffb700 1 -- 172.21.15.31:0/3610794353 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4005250 con 0x55d2d1bada00 2024-04-26T18:00:44.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.493+0000 7ff20af07b80 1 -- 172.21.15.31:0/3610794353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1e80986b0 con 0x55d2d1bada00 2024-04-26T18:00:44.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.493+0000 7ff1e1ffb700 1 -- 172.21.15.31:0/3610794353 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7ff1f40053f0 con 0x55d2d1bada00 2024-04-26T18:00:44.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.494+0000 7ff1e1ffb700 1 -- 172.21.15.31:0/3610794353 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff1f4033050 con 0x55d2d1bada00 2024-04-26T18:00:44.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.494+0000 7ff20af07b80 10 client.15771 fetch_fsmap learned FSMap version 32 2024-04-26T18:00:44.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.494+0000 7ff20af07b80 1 -- 172.21.15.31:0/3610794353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d2d1c24650 con 0x55d2d1bada00 2024-04-26T18:00:44.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.494+0000 7ff1e1ffb700 1 -- 172.21.15.31:0/3610794353 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1e80986b0 con 0x55d2d1bada00 2024-04-26T18:00:44.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.495+0000 7ff1e1ffb700 1 -- 172.21.15.31:0/3610794353 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ff1f400b5f0 con 0x55d2d1bada00 2024-04-26T18:00:44.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.495+0000 7ff20af07b80 10 client.15771 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:00:44.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.495+0000 7ff20af07b80 10 client.15771 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T18:00:44.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.495+0000 7ff20af07b80 20 client.15771 populate_metadata read hostname 'smithi031' 2024-04-26T18:00:44.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.495+0000 7ff20af07b80 1 --2- 172.21.15.31:0/3610794353 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55d2d1c24eb0 0x55d2d1c272d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:44.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.495+0000 7ff20af07b80 4 client.15771 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.5, "await_for": 42.0}] 2024-04-26T18:00:44.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.495+0000 7ff20af07b80 1 -- 172.21.15.31:0/3610794353 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.5, "await_for": 42.0}) -- 0x55d2d1afd350 con 0x55d2d1c24eb0 2024-04-26T18:00:44.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.496+0000 7ff1fe411700 1 --2- 172.21.15.31:0/3610794353 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55d2d1c24eb0 0x55d2d1c272d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:44.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.496+0000 7ff1e1ffb700 10 client.15771 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:44.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.496+0000 7ff1fe411700 1 --2- 172.21.15.31:0/3610794353 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55d2d1c24eb0 0x55d2d1c272d0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3932fa5b9eda3091 server_cookie=2195668528a65731 in_seq=0 out_seq=0 2024-04-26T18:00:44.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.688+0000 7ff1e1ffb700 1 -- 172.21.15.31:0/3610794353 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55d2d1afd350 con 0x55d2d1c24eb0 2024-04-26T18:00:44.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.688+0000 7ff1e1ffb700 10 client.15771 handle_command_reply: tid=0 2024-04-26T18:00:44.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.688+0000 7ff20af07b80 1 client.15771 shutdown 2024-04-26T18:00:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.689+0000 7ff20af07b80 1 -- 172.21.15.31:0/3610794353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1bada00 msgr2=0x55d2d1bb2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.689+0000 7ff20af07b80 1 --2- 172.21.15.31:0/3610794353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1bada00 0x55d2d1bb2aa0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40098f0 tx=0x7ff1f400f8a0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.689+0000 7ff20af07b80 1 -- 172.21.15.31:0/3610794353 shutdown_connections 2024-04-26T18:00:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.689+0000 7ff20af07b80 1 --2- 172.21.15.31:0/3610794353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2d1bada00 0x55d2d1bb2aa0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.689+0000 7ff20af07b80 1 --2- 172.21.15.31:0/3610794353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2d1ba2230 0x55d2d1bb5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.689+0000 7ff20af07b80 1 --2- 172.21.15.31:0/3610794353 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55d2d1c24eb0 0x55d2d1c272d0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.689+0000 7ff20af07b80 1 --2- 172.21.15.31:0/3610794353 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55d2d1ba2b50 0x55d2d1bb6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.689+0000 7ff1e1ffb700 0 client.15771 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.689+0000 7ff20af07b80 1 -- 172.21.15.31:0/3610794353 >> 172.21.15.31:0/3610794353 conn(0x55d2d1acfed0 msgr2=0x55d2d1ba4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.689+0000 7ff20af07b80 1 -- 172.21.15.31:0/3610794353 shutdown_connections 2024-04-26T18:00:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.689+0000 7ff20af07b80 1 -- 172.21.15.31:0/3610794353 wait complete. 2024-04-26T18:00:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.689+0000 7ff20af07b80 20 client.15771 trim_cache size 0 max 16384 2024-04-26T18:00:44.691 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T18:00:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 32, 2024-04-26T18:00:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "leader": 24473, 2024-04-26T18:00:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "set_version": 10, 2024-04-26T18:00:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "sets": { 2024-04-26T18:00:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "7562d4d8": { 2024-04-26T18:00:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "version": 10, 2024-04-26T18:00:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "age_ref": 0.0, 2024-04-26T18:00:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T18:00:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "QUIESCED", 2024-04-26T18:00:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T18:00:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:00:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "timeout": 60.0, 2024-04-26T18:00:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "expiration": 79.5, 2024-04-26T18:00:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "members": { 2024-04-26T18:00:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "file:/": { 2024-04-26T18:00:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "excluded": false, 2024-04-26T18:00:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T18:00:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "QUIESCED", 2024-04-26T18:00:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T18:00:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:00:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:00:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:00:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:00:44.693 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:00:44.693 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T18:00:44.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.690+0000 7ff200675700 1 -- 172.21.15.31:0/431387623 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ff1e40a3e50 msgr2=0x7ff1e40a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:44.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.690+0000 7ff200675700 1 --2- 172.21.15.31:0/431387623 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ff1e40a3e50 0x7ff1e40a6310 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff1f810ac20 tx=0x7ff1f000b040 comp rx=0 tx=0).stop 2024-04-26T18:00:44.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.690+0000 7ff200675700 1 -- 172.21.15.31:0/431387623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1f8105660 msgr2=0x7ff1f81a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:44.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.690+0000 7ff200675700 1 --2- 172.21.15.31:0/431387623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1f8105660 0x7ff1f81a18d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800e9a0 tx=0x7ff1e800ecb0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.690+0000 7ff200675700 1 -- 172.21.15.31:0/431387623 shutdown_connections 2024-04-26T18:00:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.690+0000 7ff200675700 1 --2- 172.21.15.31:0/431387623 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ff1e40a3e50 0x7ff1e40a6310 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.690+0000 7ff200675700 1 --2- 172.21.15.31:0/431387623 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1f8105660 0x7ff1f81a18d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.690+0000 7ff200675700 1 --2- 172.21.15.31:0/431387623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1f810b660 0x7ff1f819ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.690+0000 7ff200675700 1 --2- 172.21.15.31:0/431387623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f8106020 0x7ff1f81a1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.690+0000 7ff200675700 1 -- 172.21.15.31:0/431387623 >> 172.21.15.31:0/431387623 conn(0x7ff1f8101150 msgr2=0x7ff1f81016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.690+0000 7ff200675700 1 -- 172.21.15.31:0/431387623 shutdown_connections 2024-04-26T18:00:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:00:44.690+0000 7ff200675700 1 -- 172.21.15.31:0/431387623 wait complete. 2024-04-26T18:00:45.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:45 smithi031 ceph-mon[29164]: pgmap v130: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.2 MiB/s wr, 607 op/s 2024-04-26T18:00:45.486 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '7562d4d8', quiesce took 7.7 seconds. Will release after: 19.5 2024-04-26T18:00:45.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:45 smithi097 ceph-mon[32419]: pgmap v130: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.2 MiB/s wr, 607 op/s 2024-04-26T18:00:45.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:45 smithi186 ceph-mon[32797]: pgmap v130: 97 pgs: 97 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.2 MiB/s wr, 607 op/s 2024-04-26T18:00:47.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:47 smithi031 ceph-mon[29164]: pgmap v131: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.6 MiB/s wr, 702 op/s 2024-04-26T18:00:47.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:47 smithi097 ceph-mon[32419]: pgmap v131: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.6 MiB/s wr, 702 op/s 2024-04-26T18:00:47.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:47 smithi186 ceph-mon[32797]: pgmap v131: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.6 MiB/s wr, 702 op/s 2024-04-26T18:00:48.098 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 205.0 s, 907.9 tps, lat 31.220 ms stddev 31.727 2024-04-26T18:00:49.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:49 smithi031 ceph-mon[29164]: pgmap v132: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 9.7 MiB/s wr, 749 op/s 2024-04-26T18:00:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:49 smithi097 ceph-mon[32419]: pgmap v132: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 9.7 MiB/s wr, 749 op/s 2024-04-26T18:00:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:49 smithi186 ceph-mon[32797]: pgmap v132: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 9.7 MiB/s wr, 749 op/s 2024-04-26T18:00:51.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:51 smithi031 ceph-mon[29164]: pgmap v133: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.5 MiB/s wr, 539 op/s 2024-04-26T18:00:51.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:51 smithi097 ceph-mon[32419]: pgmap v133: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.5 MiB/s wr, 539 op/s 2024-04-26T18:00:51.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:51 smithi186 ceph-mon[32797]: pgmap v133: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.5 MiB/s wr, 539 op/s 2024-04-26T18:00:53.098 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 210.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:00:53.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:53 smithi031 ceph-mon[29164]: pgmap v134: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.2 MiB/s wr, 585 op/s 2024-04-26T18:00:53.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:53 smithi097 ceph-mon[32419]: pgmap v134: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.2 MiB/s wr, 585 op/s 2024-04-26T18:00:53.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:53 smithi186 ceph-mon[32797]: pgmap v134: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.2 MiB/s wr, 585 op/s 2024-04-26T18:00:55.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:54 smithi186 ceph-mon[32797]: pgmap v135: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 286 op/s 2024-04-26T18:00:55.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:54 smithi031 ceph-mon[29164]: pgmap v135: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 286 op/s 2024-04-26T18:00:55.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:54 smithi097 ceph-mon[32419]: pgmap v135: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 286 op/s 2024-04-26T18:00:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.702+0000 7f7bde314640 1 -- 172.21.15.31:0/4252812085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bd81044e0 msgr2=0x7f7bd810c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.702+0000 7f7bde314640 1 --2- 172.21.15.31:0/4252812085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bd81044e0 0x7f7bd810c9c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009900 tx=0x7f7bcc02a510 comp rx=0 tx=0).stop 2024-04-26T18:00:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.702+0000 7f7bde314640 1 -- 172.21.15.31:0/4252812085 shutdown_connections 2024-04-26T18:00:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.702+0000 7f7bde314640 1 --2- 172.21.15.31:0/4252812085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bd810cf00 0x7f7bd810f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.702+0000 7f7bde314640 1 --2- 172.21.15.31:0/4252812085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bd81044e0 0x7f7bd810c9c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.702+0000 7f7bde314640 1 --2- 172.21.15.31:0/4252812085 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd8103bc0 0x7f7bd8103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.702+0000 7f7bde314640 1 -- 172.21.15.31:0/4252812085 >> 172.21.15.31:0/4252812085 conn(0x7f7bd80789b0 msgr2=0x7f7bd8078dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.702+0000 7f7bde314640 1 -- 172.21.15.31:0/4252812085 shutdown_connections 2024-04-26T18:00:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.702+0000 7f7bde314640 1 -- 172.21.15.31:0/4252812085 wait complete. 2024-04-26T18:00:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.703+0000 7f7bde314640 1 Processor -- start 2024-04-26T18:00:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.703+0000 7f7bde314640 1 -- start start 2024-04-26T18:00:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.703+0000 7f7bde314640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bd8103bc0 0x7f7bd8199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.703+0000 7f7bde314640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bd81044e0 0x7f7bd81996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.703+0000 7f7bde314640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd810cf00 0x7f7bd819d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.703+0000 7f7bde314640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd8112bb0 con 0x7f7bd8103bc0 2024-04-26T18:00:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.703+0000 7f7bde314640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd8112a30 con 0x7f7bd810cf00 2024-04-26T18:00:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.703+0000 7f7bde314640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd8112d30 con 0x7f7bd81044e0 2024-04-26T18:00:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.703+0000 7f7bd7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bd8103bc0 0x7f7bd8199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.703+0000 7f7bd7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bd8103bc0 0x7f7bd8199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60512/0 (socket says 172.21.15.31:60512) 2024-04-26T18:00:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bd7fff640 1 -- 172.21.15.31:0/1404652252 learned_addr learned my addr 172.21.15.31:0/1404652252 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bdc8b1640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd810cf00 0x7f7bd819d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bd77fe640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bd81044e0 0x7f7bd81996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bd7fff640 1 -- 172.21.15.31:0/1404652252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bd81044e0 msgr2=0x7f7bd81996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bd7fff640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bd81044e0 0x7f7bd81996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bd7fff640 1 -- 172.21.15.31:0/1404652252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd810cf00 msgr2=0x7f7bd819d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bd7fff640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd810cf00 0x7f7bd819d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bd7fff640 1 -- 172.21.15.31:0/1404652252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd819e110 con 0x7f7bd8103bc0 2024-04-26T18:00:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bdc8b1640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd810cf00 0x7f7bd819d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:00:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bd7fff640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bd8103bc0 0x7f7bd8199170 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800d950 tx=0x7f7bc800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bd57fa640 1 -- 172.21.15.31:0/1404652252 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8017070 con 0x7f7bd8103bc0 2024-04-26T18:00:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bd57fa640 1 -- 172.21.15.31:0/1404652252 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bc8009c30 con 0x7f7bd8103bc0 2024-04-26T18:00:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bde314640 1 -- 172.21.15.31:0/1404652252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bd819e400 con 0x7f7bd8103bc0 2024-04-26T18:00:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bd57fa640 1 -- 172.21.15.31:0/1404652252 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc800f620 con 0x7f7bd8103bc0 2024-04-26T18:00:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.704+0000 7f7bde314640 1 -- 172.21.15.31:0/1404652252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bd81a5c90 con 0x7f7bd8103bc0 2024-04-26T18:00:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.707+0000 7f7be102c740 1 Processor -- start 2024-04-26T18:00:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.707+0000 7f7bd57fa640 1 -- 172.21.15.31:0/1404652252 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7bc8010460 con 0x7f7bd8103bc0 2024-04-26T18:00:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.707+0000 7f7be102c740 1 -- start start 2024-04-26T18:00:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7be102c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3c98f00 0x563de3c992e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7be102c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 0x563de3ca3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7be102c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3ca40e0 0x563de3ca64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7be102c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563de3bff460 con 0x563de3ca40e0 2024-04-26T18:00:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7be102c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x563de3bb9590 con 0x563de3c99820 2024-04-26T18:00:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7be102c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563de3bfb420 con 0x563de3c98f00 2024-04-26T18:00:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7bd77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3ca40e0 0x563de3ca64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7bd77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3ca40e0 0x563de3ca64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60528/0 (socket says 172.21.15.31:60528) 2024-04-26T18:00:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7bd77fe640 1 -- 172.21.15.31:0/477701198 learned_addr learned my addr 172.21.15.31:0/477701198 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7bd7fff640 1 --2- 172.21.15.31:0/477701198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 0x563de3ca3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7bdc8b1640 1 --2- 172.21.15.31:0/477701198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3c98f00 0x563de3c992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7bd77fe640 1 -- 172.21.15.31:0/477701198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3c98f00 msgr2=0x563de3c992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7bd77fe640 1 --2- 172.21.15.31:0/477701198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3c98f00 0x563de3c992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7bd77fe640 1 -- 172.21.15.31:0/477701198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 msgr2=0x563de3ca3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7bd77fe640 1 --2- 172.21.15.31:0/477701198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 0x563de3ca3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7bd77fe640 1 -- 172.21.15.31:0/477701198 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563de3bc60d0 con 0x563de3ca40e0 2024-04-26T18:00:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7bd57fa640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7bc00a3dc0 0x7f7bc00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7bdc8b1640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7bc00a3dc0 0x7f7bc00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7bd77fe640 1 --2- 172.21.15.31:0/477701198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3ca40e0 0x563de3ca64d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009900 tx=0x7f7bcc032c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3d6368ff9eda83 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.708+0000 7f7bd57fa640 1 -- 172.21.15.31:0/1404652252 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7bc80d04c0 con 0x7f7bd8103bc0 2024-04-26T18:00:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.709+0000 7f7bbe7fc640 1 -- 172.21.15.31:0/477701198 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc03f070 con 0x563de3ca40e0 2024-04-26T18:00:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.709+0000 7f7bbe7fc640 1 -- 172.21.15.31:0/477701198 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bcc0316a0 con 0x563de3ca40e0 2024-04-26T18:00:55.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.709+0000 7f7bbe7fc640 1 -- 172.21.15.31:0/477701198 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc035070 con 0x563de3ca40e0 2024-04-26T18:00:55.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.709+0000 7f7be102c740 1 -- 172.21.15.31:0/477701198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3ca40e0 msgr2=0x563de3ca64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:55.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.709+0000 7f7bdc8b1640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7bc00a3dc0 0x7f7bc00a6280 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0005fd0 tx=0x7f7bd00098d0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:55.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.709+0000 7f7be102c740 1 --2- 172.21.15.31:0/477701198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3ca40e0 0x563de3ca64d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009900 tx=0x7f7bcc032c40 comp rx=0 tx=0).stop 2024-04-26T18:00:55.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.710+0000 7f7be102c740 1 -- 172.21.15.31:0/477701198 shutdown_connections 2024-04-26T18:00:55.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.710+0000 7f7be102c740 1 --2- 172.21.15.31:0/477701198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3ca40e0 0x563de3ca64d0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.710+0000 7f7be102c740 1 --2- 172.21.15.31:0/477701198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 0x563de3ca3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.710+0000 7f7be102c740 1 --2- 172.21.15.31:0/477701198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3c98f00 0x563de3c992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.710+0000 7f7be102c740 1 -- 172.21.15.31:0/477701198 >> 172.21.15.31:0/477701198 conn(0x563de3c95410 msgr2=0x563de3c957f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:55.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.710+0000 7f7be102c740 1 -- 172.21.15.31:0/477701198 shutdown_connections 2024-04-26T18:00:55.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.710+0000 7f7be102c740 1 -- 172.21.15.31:0/477701198 wait complete. 2024-04-26T18:00:55.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.710+0000 7f7be102c740 1 -- start start 2024-04-26T18:00:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.710+0000 7f7be102c740 1 Processor -- start 2024-04-26T18:00:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7be102c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3c98f00 0x563de3ca10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7be102c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 0x563de3ca15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7be102c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3ca40e0 0x563de3ca1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7be102c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563de3ca6cb0 con 0x563de3ca40e0 2024-04-26T18:00:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7be102c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x563de3b86b60 con 0x563de3c99820 2024-04-26T18:00:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7be102c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563de3ca6e30 con 0x563de3c98f00 2024-04-26T18:00:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7be102c740 10 client.0 _ll_register_callbacks cb 0x563de3bab2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:00:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7bdc8b1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3ca40e0 0x563de3ca1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7bd7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3c98f00 0x563de3ca10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7bdc8b1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3ca40e0 0x563de3ca1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60542/0 (socket says 172.21.15.31:60542) 2024-04-26T18:00:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7bdc8b1640 1 -- 172.21.15.31:0/153953417 learned_addr learned my addr 172.21.15.31:0/153953417 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7bd77fe640 1 --2- 172.21.15.31:0/153953417 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 0x563de3ca15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7bdc8b1640 1 -- 172.21.15.31:0/153953417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3c98f00 msgr2=0x563de3ca10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7bdc8b1640 1 --2- 172.21.15.31:0/153953417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3c98f00 0x563de3ca10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7bdc8b1640 1 -- 172.21.15.31:0/153953417 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 msgr2=0x563de3ca15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7bdc8b1640 1 --2- 172.21.15.31:0/153953417 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 0x563de3ca15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7bdc8b1640 1 -- 172.21.15.31:0/153953417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563de3911210 con 0x563de3ca40e0 2024-04-26T18:00:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7bd77fe640 1 --2- 172.21.15.31:0/153953417 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 0x563de3ca15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:00:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7bdc8b1640 1 --2- 172.21.15.31:0/153953417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3ca40e0 0x563de3ca1b30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0009b30 tx=0x7f7bd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf9421bfae19d334 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7bb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:00:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.711+0000 7f7bb77fe640 1 -- 172.21.15.31:0/153953417 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd001e820 con 0x563de3ca40e0 2024-04-26T18:00:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.712+0000 7f7bb77fe640 1 -- 172.21.15.31:0/153953417 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7bd001e9c0 con 0x563de3ca40e0 2024-04-26T18:00:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.712+0000 7f7be102c740 1 -- 172.21.15.31:0/153953417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bcc0316a0 con 0x563de3ca40e0 2024-04-26T18:00:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.712+0000 7f7bb77fe640 1 -- 172.21.15.31:0/153953417 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7bd001ebc0 con 0x563de3ca40e0 2024-04-26T18:00:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.712+0000 7f7be102c740 10 client.15783 fetch_fsmap learned FSMap version 32 2024-04-26T18:00:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.712+0000 7f7be102c740 1 -- 172.21.15.31:0/153953417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563de39dd620 con 0x563de3ca40e0 2024-04-26T18:00:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.712+0000 7f7bb77fe640 1 -- 172.21.15.31:0/153953417 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bcc0316a0 con 0x563de3ca40e0 2024-04-26T18:00:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.712+0000 7f7bb77fe640 1 -- 172.21.15.31:0/153953417 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f7bd000f040 con 0x563de3ca40e0 2024-04-26T18:00:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.712+0000 7f7be102c740 10 client.15783 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:00:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.712+0000 7f7be102c740 10 client.15783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:00:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.712+0000 7f7be102c740 20 client.15783 populate_metadata read hostname 'smithi031' 2024-04-26T18:00:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.712+0000 7f7be102c740 1 --2- 172.21.15.31:0/153953417 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563de3cb4270 0x563de3cb66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.712+0000 7f7be102c740 4 client.15783 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:00:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.712+0000 7f7be102c740 1 -- 172.21.15.31:0/153953417 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563de3c0d070 con 0x563de3cb4270 2024-04-26T18:00:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.713+0000 7f7bd7fff640 1 --2- 172.21.15.31:0/153953417 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563de3cb4270 0x563de3cb66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.713+0000 7f7bd7fff640 1 --2- 172.21.15.31:0/153953417 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563de3cb4270 0x563de3cb66c0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9535f8963fd1439a server_cookie=7bb02d69682be41d in_seq=0 out_seq=0 2024-04-26T18:00:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.713+0000 7f7bb77fe640 10 client.15783 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7bb77fe640 1 -- 172.21.15.31:0/153953417 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563de3c0d070 con 0x563de3cb4270 2024-04-26T18:00:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7bb77fe640 10 client.15783 handle_command_reply: tid=0 2024-04-26T18:00:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7be102c740 1 client.15783 shutdown 2024-04-26T18:00:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7be102c740 1 -- 172.21.15.31:0/153953417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3ca40e0 msgr2=0x563de3ca1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7be102c740 1 --2- 172.21.15.31:0/153953417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3ca40e0 0x563de3ca1b30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0009b30 tx=0x7f7bd0004a40 comp rx=0 tx=0).stop 2024-04-26T18:00:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7be102c740 1 -- 172.21.15.31:0/153953417 shutdown_connections 2024-04-26T18:00:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7be102c740 1 --2- 172.21.15.31:0/153953417 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563de3cb4270 0x563de3cb66c0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7be102c740 1 --2- 172.21.15.31:0/153953417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3ca40e0 0x563de3ca1b30 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7bb77fe640 0 client.15783 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7be102c740 1 --2- 172.21.15.31:0/153953417 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 0x563de3ca15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7be102c740 1 --2- 172.21.15.31:0/153953417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3c98f00 0x563de3ca10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7be102c740 1 -- 172.21.15.31:0/153953417 >> 172.21.15.31:0/153953417 conn(0x563de3c95410 msgr2=0x563de3c9b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7be102c740 1 -- 172.21.15.31:0/153953417 shutdown_connections 2024-04-26T18:00:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7be102c740 1 -- 172.21.15.31:0/153953417 wait complete. 2024-04-26T18:00:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.714+0000 7f7be102c740 20 client.15783 trim_cache size 0 max 16384 2024-04-26T18:00:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.729+0000 7f7be102c740 1 Processor -- start 2024-04-26T18:00:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.729+0000 7f7be102c740 1 -- start start 2024-04-26T18:00:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.729+0000 7f7be102c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3c98f00 0x563de3cafb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.729+0000 7f7be102c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 0x563de3c9b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.729+0000 7f7be102c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3ca40e0 0x563de3cbbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.729+0000 7f7be102c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd81b0c90 con 0x563de3c98f00 2024-04-26T18:00:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.729+0000 7f7be102c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd81ae500 con 0x563de3c99820 2024-04-26T18:00:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.729+0000 7f7be102c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd81ac350 con 0x563de3ca40e0 2024-04-26T18:00:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.730+0000 7f7bd7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3c98f00 0x563de3cafb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.730+0000 7f7bd7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3c98f00 0x563de3cafb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60556/0 (socket says 172.21.15.31:60556) 2024-04-26T18:00:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.730+0000 7f7bd7fff640 1 -- 172.21.15.31:0/3076146033 learned_addr learned my addr 172.21.15.31:0/3076146033 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.730+0000 7f7bdc8b1640 1 --2- 172.21.15.31:0/3076146033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3ca40e0 0x563de3cbbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.730+0000 7f7bd77fe640 1 --2- 172.21.15.31:0/3076146033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 0x563de3c9b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.730+0000 7f7bd7fff640 1 -- 172.21.15.31:0/3076146033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3ca40e0 msgr2=0x563de3cbbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.730+0000 7f7bd7fff640 1 --2- 172.21.15.31:0/3076146033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3ca40e0 0x563de3cbbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.730+0000 7f7bd7fff640 1 -- 172.21.15.31:0/3076146033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 msgr2=0x563de3c9b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.730+0000 7f7bd7fff640 1 --2- 172.21.15.31:0/3076146033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 0x563de3c9b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.730+0000 7f7bd7fff640 1 -- 172.21.15.31:0/3076146033 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563de3c95180 con 0x563de3c98f00 2024-04-26T18:00:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.730+0000 7f7bd7fff640 1 --2- 172.21.15.31:0/3076146033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3c98f00 0x563de3cafb70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7bc80ce260 tx=0x7f7bc80d3f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74575405513ed9ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.730+0000 7f7bbdffb640 1 -- 172.21.15.31:0/3076146033 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8020040 con 0x563de3c98f00 2024-04-26T18:00:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.730+0000 7f7bbdffb640 1 -- 172.21.15.31:0/3076146033 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bc809cbb0 con 0x563de3c98f00 2024-04-26T18:00:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.730+0000 7f7bbdffb640 1 -- 172.21.15.31:0/3076146033 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc80cdee0 con 0x563de3c98f00 2024-04-26T18:00:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.731+0000 7f7be102c740 1 -- 172.21.15.31:0/3076146033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3c98f00 msgr2=0x563de3cafb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.731+0000 7f7be102c740 1 --2- 172.21.15.31:0/3076146033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3c98f00 0x563de3cafb70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7bc80ce260 tx=0x7f7bc80d3f90 comp rx=0 tx=0).stop 2024-04-26T18:00:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.731+0000 7f7be102c740 1 -- 172.21.15.31:0/3076146033 shutdown_connections 2024-04-26T18:00:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.731+0000 7f7be102c740 1 --2- 172.21.15.31:0/3076146033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3ca40e0 0x563de3cbbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.731+0000 7f7be102c740 1 --2- 172.21.15.31:0/3076146033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c99820 0x563de3c9b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.731+0000 7f7be102c740 1 --2- 172.21.15.31:0/3076146033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3c98f00 0x563de3cafb70 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.731+0000 7f7be102c740 1 -- 172.21.15.31:0/3076146033 >> 172.21.15.31:0/3076146033 conn(0x563de3c95410 msgr2=0x563de3c945b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.731+0000 7f7be102c740 1 -- 172.21.15.31:0/3076146033 shutdown_connections 2024-04-26T18:00:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.731+0000 7f7be102c740 1 -- 172.21.15.31:0/3076146033 wait complete. 2024-04-26T18:00:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.731+0000 7f7be102c740 1 -- start start 2024-04-26T18:00:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.731+0000 7f7be102c740 1 Processor -- start 2024-04-26T18:00:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7be102c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c98f00 0x563de3c9e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7be102c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3c99820 0x563de3c9ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7be102c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3ca40e0 0x563de3d1dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7be102c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd81ac9c0 con 0x563de3c99820 2024-04-26T18:00:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7be102c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd81ade30 con 0x563de3c98f00 2024-04-26T18:00:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7be102c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd81ad090 con 0x563de3ca40e0 2024-04-26T18:00:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7be102c740 10 client.0 _ll_register_callbacks cb 0x7f7bd0009680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:00:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7bd77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3c99820 0x563de3c9ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7bd77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3c99820 0x563de3c9ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60560/0 (socket says 172.21.15.31:60560) 2024-04-26T18:00:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7bd77fe640 1 -- 172.21.15.31:0/2302703287 learned_addr learned my addr 172.21.15.31:0/2302703287 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:00:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7bd7fff640 1 --2- 172.21.15.31:0/2302703287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c98f00 0x563de3c9e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7bdc8b1640 1 --2- 172.21.15.31:0/2302703287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3ca40e0 0x563de3d1dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7bd77fe640 1 -- 172.21.15.31:0/2302703287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3ca40e0 msgr2=0x563de3d1dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7bd77fe640 1 --2- 172.21.15.31:0/2302703287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3ca40e0 0x563de3d1dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7bd77fe640 1 -- 172.21.15.31:0/2302703287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c98f00 msgr2=0x563de3c9e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:00:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7bd77fe640 1 --2- 172.21.15.31:0/2302703287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c98f00 0x563de3c9e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7bd77fe640 1 -- 172.21.15.31:0/2302703287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563de3d1e190 con 0x563de3c99820 2024-04-26T18:00:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7bd77fe640 1 --2- 172.21.15.31:0/2302703287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3c99820 0x563de3c9ea10 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x563de3c965a0 tx=0x7f7bcc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b047fdb2cba4b0c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:00:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7bb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:00:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.732+0000 7f7bb77fe640 1 -- 172.21.15.31:0/2302703287 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc04b020 con 0x563de3c99820 2024-04-26T18:00:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.733+0000 7f7bb77fe640 1 -- 172.21.15.31:0/2302703287 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7bcc0037f0 con 0x563de3c99820 2024-04-26T18:00:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.733+0000 7f7be102c740 1 -- 172.21.15.31:0/2302703287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bc809cbb0 con 0x563de3c99820 2024-04-26T18:00:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.733+0000 7f7bb77fe640 1 -- 172.21.15.31:0/2302703287 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7bcc056050 con 0x563de3c99820 2024-04-26T18:00:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.733+0000 7f7be102c740 10 client.15789 fetch_fsmap learned FSMap version 32 2024-04-26T18:00:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.733+0000 7f7be102c740 1 -- 172.21.15.31:0/2302703287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563de3d1f4d0 con 0x563de3c99820 2024-04-26T18:00:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.733+0000 7f7bb77fe640 1 -- 172.21.15.31:0/2302703287 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bc809cbb0 con 0x563de3c99820 2024-04-26T18:00:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.733+0000 7f7bb77fe640 1 -- 172.21.15.31:0/2302703287 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f7bcc03b770 con 0x563de3c99820 2024-04-26T18:00:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.733+0000 7f7be102c740 10 client.15789 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:00:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.733+0000 7f7be102c740 10 client.15789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:00:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.733+0000 7f7be102c740 20 client.15789 populate_metadata read hostname 'smithi031' 2024-04-26T18:00:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.733+0000 7f7be102c740 1 --2- 172.21.15.31:0/2302703287 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563de3cb4270 0x563de3d21880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:00:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.733+0000 7f7be102c740 4 client.15789 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:00:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.733+0000 7f7be102c740 1 -- 172.21.15.31:0/2302703287 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563de3c06ee0 con 0x563de3cb4270 2024-04-26T18:00:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.733+0000 7f7bd7fff640 1 --2- 172.21.15.31:0/2302703287 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563de3cb4270 0x563de3d21880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:00:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.734+0000 7f7bd7fff640 1 --2- 172.21.15.31:0/2302703287 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563de3cb4270 0x563de3d21880 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98feb9679283f6e5 server_cookie=8c84af80842bd853 in_seq=0 out_seq=0 2024-04-26T18:00:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.734+0000 7f7bb77fe640 10 client.15789 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.749+0000 7f7bb77fe640 1 -- 172.21.15.31:0/2302703287 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563de3c06ee0 con 0x563de3cb4270 2024-04-26T18:00:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.749+0000 7f7bb77fe640 10 client.15789 handle_command_reply: tid=0 2024-04-26T18:00:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.749+0000 7f7be102c740 1 client.15789 shutdown 2024-04-26T18:00:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7be102c740 1 -- 172.21.15.31:0/2302703287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3c99820 msgr2=0x563de3c9ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7be102c740 1 --2- 172.21.15.31:0/2302703287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3c99820 0x563de3c9ea10 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x563de3c965a0 tx=0x7f7bcc004270 comp rx=0 tx=0).stop 2024-04-26T18:00:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7be102c740 1 -- 172.21.15.31:0/2302703287 shutdown_connections 2024-04-26T18:00:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7be102c740 1 --2- 172.21.15.31:0/2302703287 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x563de3cb4270 0x563de3d21880 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7be102c740 1 --2- 172.21.15.31:0/2302703287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de3ca40e0 0x563de3d1dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7bb77fe640 0 client.15789 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:00:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7be102c740 1 --2- 172.21.15.31:0/2302703287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de3c99820 0x563de3c9ea10 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7be102c740 1 --2- 172.21.15.31:0/2302703287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563de3c98f00 0x563de3c9e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7be102c740 1 -- 172.21.15.31:0/2302703287 >> 172.21.15.31:0/2302703287 conn(0x563de3c95410 msgr2=0x563de3ca1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7be102c740 1 -- 172.21.15.31:0/2302703287 shutdown_connections 2024-04-26T18:00:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7be102c740 1 -- 172.21.15.31:0/2302703287 wait complete. 2024-04-26T18:00:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7be102c740 20 client.15789 trim_cache size 0 max 16384 2024-04-26T18:00:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:00:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:00:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:00:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:00:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7bde314640 1 -- 172.21.15.31:0/1404652252 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7bc00a3dc0 msgr2=0x7f7bc00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7bde314640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7bc00a3dc0 0x7f7bc00a6280 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0005fd0 tx=0x7f7bd00098d0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7bde314640 1 -- 172.21.15.31:0/1404652252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bd8103bc0 msgr2=0x7f7bd8199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:00:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.750+0000 7f7bde314640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bd8103bc0 0x7f7bd8199170 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800d950 tx=0x7f7bc800de10 comp rx=0 tx=0).stop 2024-04-26T18:00:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.751+0000 7f7bde314640 1 -- 172.21.15.31:0/1404652252 shutdown_connections 2024-04-26T18:00:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.751+0000 7f7bde314640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd810cf00 0x7f7bd819d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.751+0000 7f7bde314640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bd81044e0 0x7f7bd81996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.751+0000 7f7bde314640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7bc00a3dc0 0x7f7bc00a6280 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.751+0000 7f7bde314640 1 --2- 172.21.15.31:0/1404652252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bd8103bc0 0x7f7bd8199170 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:00:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.751+0000 7f7bde314640 1 -- 172.21.15.31:0/1404652252 >> 172.21.15.31:0/1404652252 conn(0x7f7bd80789b0 msgr2=0x7f7bd80ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:00:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.751+0000 7f7bde314640 1 -- 172.21.15.31:0/1404652252 shutdown_connections 2024-04-26T18:00:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:00:55.751+0000 7f7bde314640 1 -- 172.21.15.31:0/1404652252 wait complete. 2024-04-26T18:00:57.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:56 smithi186 ceph-mon[32797]: pgmap v136: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 4.5 MiB/s wr, 290 op/s 2024-04-26T18:00:57.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:56 smithi031 ceph-mon[29164]: pgmap v136: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 4.5 MiB/s wr, 290 op/s 2024-04-26T18:00:57.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:56 smithi097 ceph-mon[32419]: pgmap v136: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 4.5 MiB/s wr, 290 op/s 2024-04-26T18:00:58.098 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 215.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:00:59.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:00:58 smithi186 ceph-mon[32797]: pgmap v137: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.3 MiB/s wr, 199 op/s 2024-04-26T18:00:59.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:00:58 smithi031 ceph-mon[29164]: pgmap v137: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.3 MiB/s wr, 199 op/s 2024-04-26T18:00:59.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:00:58 smithi097 ceph-mon[32419]: pgmap v137: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.3 MiB/s wr, 199 op/s 2024-04-26T18:01:01.256 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:00 smithi031 ceph-mon[29164]: pgmap v138: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.1 MiB/s wr, 54 op/s 2024-04-26T18:01:01.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:00 smithi186 ceph-mon[32797]: pgmap v138: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.1 MiB/s wr, 54 op/s 2024-04-26T18:01:01.354 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:00 smithi097 ceph-mon[32419]: pgmap v138: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.1 MiB/s wr, 54 op/s 2024-04-26T18:01:03.098 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 220.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:01:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:02 smithi186 ceph-mon[32797]: pgmap v139: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.1 MiB/s wr, 73 op/s 2024-04-26T18:01:03.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:02 smithi031 ceph-mon[29164]: pgmap v139: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.1 MiB/s wr, 73 op/s 2024-04-26T18:01:03.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:02 smithi097 ceph-mon[32419]: pgmap v139: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.1 MiB/s wr, 73 op/s 2024-04-26T18:01:04.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T18:01:05.187 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:04 smithi097 ceph-mon[32419]: pgmap v140: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 417 KiB/s wr, 27 op/s 2024-04-26T18:01:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:01:05.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:04 smithi186 ceph-mon[32797]: pgmap v140: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 417 KiB/s wr, 27 op/s 2024-04-26T18:01:05.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:04 smithi031 ceph-mon[29164]: pgmap v140: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 417 KiB/s wr, 27 op/s 2024-04-26T18:01:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.045+0000 7f95c29d0700 1 -- 172.21.15.31:0/3252665614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95bc06c110 msgr2=0x7f95bc06c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.045+0000 7f95c29d0700 1 --2- 172.21.15.31:0/3252665614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95bc06c110 0x7f95bc06c590 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f95b0009990 tx=0x7f95b0009ca0 comp rx=0 tx=0).stop 2024-04-26T18:01:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.045+0000 7f95c29d0700 1 -- 172.21.15.31:0/3252665614 shutdown_connections 2024-04-26T18:01:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.045+0000 7f95c29d0700 1 --2- 172.21.15.31:0/3252665614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95bc108380 0x7f95bc06bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.045+0000 7f95c29d0700 1 --2- 172.21.15.31:0/3252665614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95bc105e90 0x7f95bc1062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.045+0000 7f95c29d0700 1 --2- 172.21.15.31:0/3252665614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95bc06c110 0x7f95bc06c590 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.045+0000 7f95c29d0700 1 -- 172.21.15.31:0/3252665614 >> 172.21.15.31:0/3252665614 conn(0x7f95bc0793c0 msgr2=0x7f95bc0797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:07.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.045+0000 7f95c29d0700 1 -- 172.21.15.31:0/3252665614 shutdown_connections 2024-04-26T18:01:07.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.045+0000 7f95c29d0700 1 -- 172.21.15.31:0/3252665614 wait complete. 2024-04-26T18:01:07.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.045+0000 7f95c29d0700 1 Processor -- start 2024-04-26T18:01:07.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.045+0000 7f95c29d0700 1 -- start start 2024-04-26T18:01:07.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95c29d0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95bc06c110 0x7f95bc1a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:07.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95c29d0700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95bc105e90 0x7f95bc1a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:07.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95c29d0700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95bc108380 0x7f95bc1a4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:07.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95c29d0700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc119c80 con 0x7f95bc105e90 2024-04-26T18:01:07.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95c29d0700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc119b00 con 0x7f95bc108380 2024-04-26T18:01:07.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95c29d0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc119e00 con 0x7f95bc06c110 2024-04-26T18:01:07.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95bb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95bc105e90 0x7f95bc1a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:07.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95bb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95bc105e90 0x7f95bc1a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44620/0 (socket says 172.21.15.31:44620) 2024-04-26T18:01:07.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95bb7fe700 1 -- 172.21.15.31:0/4230410344 learned_addr learned my addr 172.21.15.31:0/4230410344 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:07.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95c0f6d700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95bc108380 0x7f95bc1a4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95bbfff700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95bc06c110 0x7f95bc1a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95bb7fe700 1 -- 172.21.15.31:0/4230410344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95bc06c110 msgr2=0x7f95bc1a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95bb7fe700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95bc06c110 0x7f95bc1a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95bb7fe700 1 -- 172.21.15.31:0/4230410344 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95bc108380 msgr2=0x7f95bc1a4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95bb7fe700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95bc108380 0x7f95bc1a4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95bb7fe700 1 -- 172.21.15.31:0/4230410344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95bc1a5140 con 0x7f95bc105e90 2024-04-26T18:01:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95c0f6d700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95bc108380 0x7f95bc1a4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95bb7fe700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95bc105e90 0x7f95bc1a0670 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f95b000bf20 tx=0x7f95b000f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95b97fa700 1 -- 172.21.15.31:0/4230410344 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b0027070 con 0x7f95bc105e90 2024-04-26T18:01:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.046+0000 7f95c29d0700 1 -- 172.21.15.31:0/4230410344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95bc1a53d0 con 0x7f95bc105e90 2024-04-26T18:01:07.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.047+0000 7f95b97fa700 1 -- 172.21.15.31:0/4230410344 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95b0022470 con 0x7f95bc105e90 2024-04-26T18:01:07.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.047+0000 7f95b97fa700 1 -- 172.21.15.31:0/4230410344 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b0017630 con 0x7f95bc105e90 2024-04-26T18:01:07.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.047+0000 7f95c29d0700 1 -- 172.21.15.31:0/4230410344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95bc1accb0 con 0x7f95bc105e90 2024-04-26T18:01:07.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.047+0000 7f95c29d0700 1 -- 172.21.15.31:0/4230410344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95bc06a110 con 0x7f95bc105e90 2024-04-26T18:01:07.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.049+0000 7f95b97fa700 1 -- 172.21.15.31:0/4230410344 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f95b002a070 con 0x7f95bc105e90 2024-04-26T18:01:07.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.049+0000 7f95b97fa700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f95a40a3e30 0x7f95a40a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:07.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.049+0000 7f95bbfff700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f95a40a3e30 0x7f95a40a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:07.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.049+0000 7f95b97fa700 1 -- 172.21.15.31:0/4230410344 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f95b00d1820 con 0x7f95bc105e90 2024-04-26T18:01:07.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.050+0000 7f95bbfff700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f95a40a3e30 0x7f95a40a62f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f95ac005fd0 tx=0x7f95ac00a560 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:07.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.053+0000 7f95b97fa700 1 -- 172.21.15.31:0/4230410344 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f95b0013070 con 0x7f95bc105e90 2024-04-26T18:01:07.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:06 smithi186 ceph-mon[32797]: pgmap v141: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 418 KiB/s wr, 29 op/s 2024-04-26T18:01:07.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.307+0000 7f95c29d0700 1 -- 172.21.15.31:0/4230410344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95bc1a1490 con 0x7f95bc105e90 2024-04-26T18:01:07.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:06 smithi031 ceph-mon[29164]: pgmap v141: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 418 KiB/s wr, 29 op/s 2024-04-26T18:01:07.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.308+0000 7f95b97fa700 1 -- 172.21.15.31:0/4230410344 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11877 (secure 0 0 0) 0x7f95bc1a1490 con 0x7f95bc105e90 2024-04-26T18:01:07.308 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:01:07.309 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T18:00:06.847155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24446,24457,24473]},"id":1}]} 2024-04-26T18:01:07.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.310+0000 7f95c29d0700 1 -- 172.21.15.31:0/4230410344 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f95a40a3e30 msgr2=0x7f95a40a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:07.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.310+0000 7f95c29d0700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f95a40a3e30 0x7f95a40a62f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f95ac005fd0 tx=0x7f95ac00a560 comp rx=0 tx=0).stop 2024-04-26T18:01:07.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.310+0000 7f95c29d0700 1 -- 172.21.15.31:0/4230410344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95bc105e90 msgr2=0x7f95bc1a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:07.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.310+0000 7f95c29d0700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95bc105e90 0x7f95bc1a0670 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f95b000bf20 tx=0x7f95b000f510 comp rx=0 tx=0).stop 2024-04-26T18:01:07.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.311+0000 7f95c29d0700 1 -- 172.21.15.31:0/4230410344 shutdown_connections 2024-04-26T18:01:07.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.311+0000 7f95c29d0700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f95a40a3e30 0x7f95a40a62f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:07.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.311+0000 7f95c29d0700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95bc108380 0x7f95bc1a4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:07.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.311+0000 7f95c29d0700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95bc06c110 0x7f95bc1a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:07.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.311+0000 7f95c29d0700 1 --2- 172.21.15.31:0/4230410344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95bc105e90 0x7f95bc1a0670 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:07.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.311+0000 7f95c29d0700 1 -- 172.21.15.31:0/4230410344 >> 172.21.15.31:0/4230410344 conn(0x7f95bc0793c0 msgr2=0x7f95bc104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:07.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.311+0000 7f95c29d0700 1 -- 172.21.15.31:0/4230410344 shutdown_connections 2024-04-26T18:01:07.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:07.311+0000 7f95c29d0700 1 -- 172.21.15.31:0/4230410344 wait complete. 2024-04-26T18:01:07.312 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 32 2024-04-26T18:01:07.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:06 smithi097 ceph-mon[32419]: pgmap v141: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 418 KiB/s wr, 29 op/s 2024-04-26T18:01:08.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:07 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/4230410344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:01:08.099 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 225.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:01:08.140 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24473 quiesce db --set-id 7562d4d8 --release --await' 2024-04-26T18:01:08.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24473 quiesce db --set-id 7562d4d8 --release --await 2024-04-26T18:01:08.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:07 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/4230410344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:01:08.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:07 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/4230410344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:01:08.393 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:01:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.839+0000 7fe1b2647640 1 -- 172.21.15.31:0/2474925654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1ac10f1f0 msgr2=0x7fe1ac1115e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.839+0000 7fe1b2647640 1 --2- 172.21.15.31:0/2474925654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1ac10f1f0 0x7fe1ac1115e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4009ca0 tx=0x7fe1a402a950 comp rx=0 tx=0).stop 2024-04-26T18:01:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.839+0000 7fe1b2647640 1 -- 172.21.15.31:0/2474925654 shutdown_connections 2024-04-26T18:01:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.839+0000 7fe1b2647640 1 --2- 172.21.15.31:0/2474925654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1ac10f1f0 0x7fe1ac1115e0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.840+0000 7fe1b2647640 1 --2- 172.21.15.31:0/2474925654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1ac075600 0x7fe1ac10ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.840+0000 7fe1b2647640 1 --2- 172.21.15.31:0/2474925654 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1ac074ce0 0x7fe1ac0750c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.840+0000 7fe1b2647640 1 -- 172.21.15.31:0/2474925654 >> 172.21.15.31:0/2474925654 conn(0x7fe1ac0fe400 msgr2=0x7fe1ac100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.840+0000 7fe1b2647640 1 -- 172.21.15.31:0/2474925654 shutdown_connections 2024-04-26T18:01:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.840+0000 7fe1b2647640 1 -- 172.21.15.31:0/2474925654 wait complete. 2024-04-26T18:01:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.840+0000 7fe1b2647640 1 Processor -- start 2024-04-26T18:01:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.840+0000 7fe1b2647640 1 -- start start 2024-04-26T18:01:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b2647640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1ac074ce0 0x7fe1ac19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b2647640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1ac075600 0x7fe1ac19b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b2647640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1ac10f1f0 0x7fe1ac19fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b2647640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac114e00 con 0x7fe1ac10f1f0 2024-04-26T18:01:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b2647640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac114c80 con 0x7fe1ac075600 2024-04-26T18:01:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b2647640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac114f80 con 0x7fe1ac074ce0 2024-04-26T18:01:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b0be4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1ac10f1f0 0x7fe1ac19fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b0be4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1ac10f1f0 0x7fe1ac19fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44648/0 (socket says 172.21.15.31:44648) 2024-04-26T18:01:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b0be4640 1 -- 172.21.15.31:0/2221196680 learned_addr learned my addr 172.21.15.31:0/2221196680 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1abfff640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1ac074ce0 0x7fe1ac19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1ac075600 0x7fe1ac19b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b0be4640 1 -- 172.21.15.31:0/2221196680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1ac074ce0 msgr2=0x7fe1ac19b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b0be4640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1ac074ce0 0x7fe1ac19b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b0be4640 1 -- 172.21.15.31:0/2221196680 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1ac075600 msgr2=0x7fe1ac19b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b0be4640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1ac075600 0x7fe1ac19b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b0be4640 1 -- 172.21.15.31:0/2221196680 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1ac1a02a0 con 0x7fe1ac10f1f0 2024-04-26T18:01:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1ac075600 0x7fe1ac19b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b0be4640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1ac10f1f0 0x7fe1ac19fb70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4006010 tx=0x7fe1a402f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1a97fa640 1 -- 172.21.15.31:0/2221196680 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4034040 con 0x7fe1ac10f1f0 2024-04-26T18:01:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b2647640 1 -- 172.21.15.31:0/2221196680 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1ac1a0530 con 0x7fe1ac10f1f0 2024-04-26T18:01:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1a97fa640 1 -- 172.21.15.31:0/2221196680 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1a4003e00 con 0x7fe1ac10f1f0 2024-04-26T18:01:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1a97fa640 1 -- 172.21.15.31:0/2221196680 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4045480 con 0x7fe1ac10f1f0 2024-04-26T18:01:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.841+0000 7fe1b2647640 1 -- 172.21.15.31:0/2221196680 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1ac1a7e20 con 0x7fe1ac10f1f0 2024-04-26T18:01:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1a97fa640 1 -- 172.21.15.31:0/2221196680 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe1a4037020 con 0x7fe1ac10f1f0 2024-04-26T18:01:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1b4ffd740 1 Processor -- start 2024-04-26T18:01:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1b4ffd740 1 -- start start 2024-04-26T18:01:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1b4ffd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55816584ff00 0x5581658502e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1b4ffd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558165850820 0x55816585aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1a97fa640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe1940a3dc0 0x7fe1940a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1b4ffd740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816585b0e0 0x55816585d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1b4ffd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5581657b6460 con 0x55816584ff00 2024-04-26T18:01:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1b4ffd740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x558165770590 con 0x55816585b0e0 2024-04-26T18:01:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1b4ffd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5581657b2420 con 0x558165850820 2024-04-26T18:01:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1ab7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55816584ff00 0x5581658502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1ab7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55816584ff00 0x5581658502e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44660/0 (socket says 172.21.15.31:44660) 2024-04-26T18:01:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1ab7fe640 1 -- 172.21.15.31:0/1936482280 learned_addr learned my addr 172.21.15.31:0/1936482280 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe1940a3dc0 0x7fe1940a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1a97fa640 1 -- 172.21.15.31:0/2221196680 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fe1a40f17d0 con 0x7fe1ac10f1f0 2024-04-26T18:01:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.844+0000 7fe1abfff640 1 --2- 172.21.15.31:0/1936482280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558165850820 0x55816585aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.845+0000 7fe1b0be4640 1 --2- 172.21.15.31:0/1936482280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816585b0e0 0x55816585d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.845+0000 7fe1ab7fe640 1 -- 172.21.15.31:0/1936482280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558165850820 msgr2=0x55816585aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.845+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/1936482280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558165850820 0x55816585aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.845+0000 7fe1ab7fe640 1 -- 172.21.15.31:0/1936482280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816585b0e0 msgr2=0x55816585d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.845+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/1936482280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816585b0e0 0x55816585d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.845+0000 7fe1ab7fe640 1 -- 172.21.15.31:0/1936482280 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55816577d0d0 con 0x55816584ff00 2024-04-26T18:01:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.845+0000 7fe1abfff640 1 --2- 172.21.15.31:0/1936482280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558165850820 0x55816585aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.845+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/1936482280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55816584ff00 0x5581658502e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0014810 tx=0x7fe1a0014cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1d3ef62bafcb14 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.845+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe1940a3dc0 0x7fe1940a6280 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0016740 tx=0x7fe1a001a000 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.845+0000 7fe1927fc640 1 -- 172.21.15.31:0/1936482280 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0024070 con 0x55816584ff00 2024-04-26T18:01:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.845+0000 7fe1927fc640 1 -- 172.21.15.31:0/1936482280 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1a000a820 con 0x55816584ff00 2024-04-26T18:01:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.845+0000 7fe1927fc640 1 -- 172.21.15.31:0/1936482280 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a001f650 con 0x55816584ff00 2024-04-26T18:01:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.846+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/1936482280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55816584ff00 msgr2=0x5581658502e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.846+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/1936482280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55816584ff00 0x5581658502e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0014810 tx=0x7fe1a0014cd0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.846+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/1936482280 shutdown_connections 2024-04-26T18:01:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.846+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/1936482280 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816585b0e0 0x55816585d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.846+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/1936482280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558165850820 0x55816585aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.846+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/1936482280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55816584ff00 0x5581658502e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.846+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/1936482280 >> 172.21.15.31:0/1936482280 conn(0x55816584c410 msgr2=0x55816584c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.846+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/1936482280 shutdown_connections 2024-04-26T18:01:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.846+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/1936482280 wait complete. 2024-04-26T18:01:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.846+0000 7fe1b4ffd740 1 -- start start 2024-04-26T18:01:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.846+0000 7fe1b4ffd740 1 Processor -- start 2024-04-26T18:01:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1b4ffd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816584ff00 0x5581658580c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1b4ffd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558165850820 0x558165858600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1b4ffd740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816585b0e0 0x558165858b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1b4ffd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55816585dcb0 con 0x558165850820 2024-04-26T18:01:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1b4ffd740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55816573db60 con 0x55816585b0e0 2024-04-26T18:01:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1b4ffd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55816585de30 con 0x55816584ff00 2024-04-26T18:01:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1b4ffd740 10 client.0 _ll_register_callbacks cb 0x5581657622b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:01:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1ab7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558165850820 0x558165858600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1ab7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558165850820 0x558165858600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44672/0 (socket says 172.21.15.31:44672) 2024-04-26T18:01:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1ab7fe640 1 -- 172.21.15.31:0/3421289148 learned_addr learned my addr 172.21.15.31:0/3421289148 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1abfff640 1 --2- 172.21.15.31:0/3421289148 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816584ff00 0x5581658580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1b0be4640 1 --2- 172.21.15.31:0/3421289148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816585b0e0 0x558165858b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1ab7fe640 1 -- 172.21.15.31:0/3421289148 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816584ff00 msgr2=0x5581658580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/3421289148 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816584ff00 0x5581658580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1ab7fe640 1 -- 172.21.15.31:0/3421289148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816585b0e0 msgr2=0x558165858b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/3421289148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816585b0e0 0x558165858b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1ab7fe640 1 -- 172.21.15.31:0/3421289148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581654c8210 con 0x558165850820 2024-04-26T18:01:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1b0be4640 1 --2- 172.21.15.31:0/3421289148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816585b0e0 0x558165858b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/3421289148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558165850820 0x558165858600 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0004810 tx=0x7fe1a0018a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9501ad1f6e158dfe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.847+0000 7fe18b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:01:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe18b7fe640 1 -- 172.21.15.31:0/3421289148 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a002d060 con 0x558165850820 2024-04-26T18:01:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe18b7fe640 1 -- 172.21.15.31:0/3421289148 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fe1a0028420 con 0x558165850820 2024-04-26T18:01:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3421289148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5581657c4070 con 0x558165850820 2024-04-26T18:01:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe18b7fe640 1 -- 172.21.15.31:0/3421289148 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe1a0024040 con 0x558165850820 2024-04-26T18:01:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe1b4ffd740 10 client.15807 fetch_fsmap learned FSMap version 32 2024-04-26T18:01:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3421289148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558165594620 con 0x558165850820 2024-04-26T18:01:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe18b7fe640 1 -- 172.21.15.31:0/3421289148 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5581657c4070 con 0x558165850820 2024-04-26T18:01:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe18b7fe640 1 -- 172.21.15.31:0/3421289148 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fe1a001e030 con 0x558165850820 2024-04-26T18:01:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe1b4ffd740 10 client.15807 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:01:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe1b4ffd740 10 client.15807 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:01:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe1b4ffd740 20 client.15807 populate_metadata read hostname 'smithi031' 2024-04-26T18:01:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/3421289148 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55816586b210 0x55816586d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe1b4ffd740 4 client.15807 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:01:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3421289148 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581657bdee0 con 0x55816586b210 2024-04-26T18:01:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.848+0000 7fe1abfff640 1 --2- 172.21.15.31:0/3421289148 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55816586b210 0x55816586d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.849+0000 7fe18b7fe640 10 client.15807 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.849+0000 7fe1abfff640 1 --2- 172.21.15.31:0/3421289148 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55816586b210 0x55816586d600 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c511c46c8ea9a77f server_cookie=a3ee9df5fdd14074 in_seq=0 out_seq=0 2024-04-26T18:01:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe18b7fe640 1 -- 172.21.15.31:0/3421289148 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5581657bdee0 con 0x55816586b210 2024-04-26T18:01:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe18b7fe640 10 client.15807 handle_command_reply: tid=0 2024-04-26T18:01:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe1b4ffd740 1 client.15807 shutdown 2024-04-26T18:01:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3421289148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558165850820 msgr2=0x558165858600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/3421289148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558165850820 0x558165858600 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0004810 tx=0x7fe1a0018a50 comp rx=0 tx=0).stop 2024-04-26T18:01:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3421289148 shutdown_connections 2024-04-26T18:01:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/3421289148 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55816586b210 0x55816586d600 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/3421289148 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816585b0e0 0x558165858b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/3421289148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558165850820 0x558165858600 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/3421289148 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816584ff00 0x5581658580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3421289148 >> 172.21.15.31:0/3421289148 conn(0x55816584c410 msgr2=0x558165852140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe18b7fe640 0 client.15807 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3421289148 shutdown_connections 2024-04-26T18:01:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3421289148 wait complete. 2024-04-26T18:01:08.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.850+0000 7fe1b4ffd740 20 client.15807 trim_cache size 0 max 16384 2024-04-26T18:01:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1b4ffd740 1 Processor -- start 2024-04-26T18:01:08.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1b4ffd740 1 -- start start 2024-04-26T18:01:08.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1b4ffd740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816584ff00 0x558165866b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1b4ffd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558165850820 0x5581658528c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1b4ffd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816585b0e0 0x55816584b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1b4ffd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac1b2e30 con 0x558165850820 2024-04-26T18:01:08.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1b4ffd740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac1b06a0 con 0x55816584ff00 2024-04-26T18:01:08.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1b4ffd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac1ae4f0 con 0x55816585b0e0 2024-04-26T18:01:08.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1ab7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558165850820 0x5581658528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1ab7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558165850820 0x5581658528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44684/0 (socket says 172.21.15.31:44684) 2024-04-26T18:01:08.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1ab7fe640 1 -- 172.21.15.31:0/4143885078 learned_addr learned my addr 172.21.15.31:0/4143885078 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:08.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1abfff640 1 --2- 172.21.15.31:0/4143885078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816584ff00 0x558165866b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1ab7fe640 1 -- 172.21.15.31:0/4143885078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816585b0e0 msgr2=0x55816584b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1b0be4640 1 --2- 172.21.15.31:0/4143885078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816585b0e0 0x55816584b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/4143885078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816585b0e0 0x55816584b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1ab7fe640 1 -- 172.21.15.31:0/4143885078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816584ff00 msgr2=0x558165866b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/4143885078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816584ff00 0x558165866b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1ab7fe640 1 -- 172.21.15.31:0/4143885078 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558165856610 con 0x558165850820 2024-04-26T18:01:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1abfff640 1 --2- 172.21.15.31:0/4143885078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816584ff00 0x558165866b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:01:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.865+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/4143885078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558165850820 0x5581658528c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe1a001fde0 tx=0x7fe1a0004660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3169f6db84701b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.866+0000 7fe191ffb640 1 -- 172.21.15.31:0/4143885078 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a002d070 con 0x558165850820 2024-04-26T18:01:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.866+0000 7fe191ffb640 1 -- 172.21.15.31:0/4143885078 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1a0018ac0 con 0x558165850820 2024-04-26T18:01:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.866+0000 7fe191ffb640 1 -- 172.21.15.31:0/4143885078 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a00326c0 con 0x558165850820 2024-04-26T18:01:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.866+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/4143885078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558165850820 msgr2=0x5581658528c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.866+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/4143885078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558165850820 0x5581658528c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe1a001fde0 tx=0x7fe1a0004660 comp rx=0 tx=0).stop 2024-04-26T18:01:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.866+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/4143885078 shutdown_connections 2024-04-26T18:01:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.866+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/4143885078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816585b0e0 0x55816584b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.866+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/4143885078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558165850820 0x5581658528c0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.866+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/4143885078 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55816584ff00 0x558165866b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.866+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/4143885078 >> 172.21.15.31:0/4143885078 conn(0x55816584c410 msgr2=0x55816584ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.866+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/4143885078 shutdown_connections 2024-04-26T18:01:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.866+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/4143885078 wait complete. 2024-04-26T18:01:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1b4ffd740 1 -- start start 2024-04-26T18:01:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1b4ffd740 1 Processor -- start 2024-04-26T18:01:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1b4ffd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55816584ff00 0x558165863450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1b4ffd740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558165850820 0x558165854a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1b4ffd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816585b0e0 0x558165854fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1b4ffd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac1aeb60 con 0x55816584ff00 2024-04-26T18:01:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1b4ffd740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac1affd0 con 0x558165850820 2024-04-26T18:01:08.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1b4ffd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac1af230 con 0x55816585b0e0 2024-04-26T18:01:08.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1b4ffd740 10 client.0 _ll_register_callbacks cb 0x7fe1a0022070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:01:08.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1abfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55816584ff00 0x558165863450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1abfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55816584ff00 0x558165863450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44690/0 (socket says 172.21.15.31:44690) 2024-04-26T18:01:08.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1abfff640 1 -- 172.21.15.31:0/3031592450 learned_addr learned my addr 172.21.15.31:0/3031592450 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:08.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/3031592450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558165850820 0x558165854a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1b0be4640 1 --2- 172.21.15.31:0/3031592450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816585b0e0 0x558165854fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1abfff640 1 -- 172.21.15.31:0/3031592450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816585b0e0 msgr2=0x558165854fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:08.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1abfff640 1 --2- 172.21.15.31:0/3031592450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816585b0e0 0x558165854fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1abfff640 1 -- 172.21.15.31:0/3031592450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558165850820 msgr2=0x558165854a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:08.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1abfff640 1 --2- 172.21.15.31:0/3031592450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558165850820 0x558165854a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1abfff640 1 -- 172.21.15.31:0/3031592450 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581658557e0 con 0x55816584ff00 2024-04-26T18:01:08.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.867+0000 7fe1b0be4640 1 --2- 172.21.15.31:0/3031592450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816585b0e0 0x558165854fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:08.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe1abfff640 1 --2- 172.21.15.31:0/3031592450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55816584ff00 0x558165863450 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe19c004700 tx=0x7fe19c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d8f73fcb405456 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:08.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe18b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:01:08.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe18b7fe640 1 -- 172.21.15.31:0/3031592450 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19c01d680 con 0x55816584ff00 2024-04-26T18:01:08.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe18b7fe640 1 -- 172.21.15.31:0/3031592450 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fe19c01d820 con 0x55816584ff00 2024-04-26T18:01:08.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3031592450 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1a0018ac0 con 0x55816584ff00 2024-04-26T18:01:08.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe18b7fe640 1 -- 172.21.15.31:0/3031592450 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe19c005020 con 0x55816584ff00 2024-04-26T18:01:08.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe1b4ffd740 10 client.15813 fetch_fsmap learned FSMap version 32 2024-04-26T18:01:08.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3031592450 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5581658d14d0 con 0x55816584ff00 2024-04-26T18:01:08.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe18b7fe640 1 -- 172.21.15.31:0/3031592450 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1a0018ac0 con 0x55816584ff00 2024-04-26T18:01:08.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe18b7fe640 1 -- 172.21.15.31:0/3031592450 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fe19c00f420 con 0x55816584ff00 2024-04-26T18:01:08.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe1b4ffd740 10 client.15813 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:01:08.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe1b4ffd740 10 client.15813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:01:08.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe1b4ffd740 20 client.15813 populate_metadata read hostname 'smithi031' 2024-04-26T18:01:08.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/3031592450 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5581658d1c50 0x5581658d4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:08.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe1b4ffd740 4 client.15813 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:01:08.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.868+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3031592450 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5581657395f0 con 0x5581658d1c50 2024-04-26T18:01:08.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.869+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/3031592450 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5581658d1c50 0x5581658d4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:08.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.869+0000 7fe1ab7fe640 1 --2- 172.21.15.31:0/3031592450 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5581658d1c50 0x5581658d4070 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7c2dc0bb4f0ba7 server_cookie=f0fc13e3a70ef396 in_seq=0 out_seq=0 2024-04-26T18:01:08.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.869+0000 7fe18b7fe640 10 client.15813 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:08.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.878+0000 7fe18b7fe640 1 -- 172.21.15.31:0/3031592450 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5581657395f0 con 0x5581658d1c50 2024-04-26T18:01:08.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.878+0000 7fe18b7fe640 10 client.15813 handle_command_reply: tid=0 2024-04-26T18:01:08.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.878+0000 7fe1b4ffd740 1 client.15813 shutdown 2024-04-26T18:01:08.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.878+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3031592450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55816584ff00 msgr2=0x558165863450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:08.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.878+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/3031592450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55816584ff00 0x558165863450 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe19c004700 tx=0x7fe19c011810 comp rx=0 tx=0).stop 2024-04-26T18:01:08.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.878+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3031592450 shutdown_connections 2024-04-26T18:01:08.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.878+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/3031592450 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5581658d1c50 0x5581658d4070 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.878+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/3031592450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55816585b0e0 0x558165854fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.878+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/3031592450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558165850820 0x558165854a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.878+0000 7fe18b7fe640 0 client.15813 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.878+0000 7fe1b4ffd740 1 --2- 172.21.15.31:0/3031592450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55816584ff00 0x558165863450 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.878+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3031592450 >> 172.21.15.31:0/3031592450 conn(0x55816584c410 msgr2=0x55816584b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.878+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3031592450 shutdown_connections 2024-04-26T18:01:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.878+0000 7fe1b4ffd740 1 -- 172.21.15.31:0/3031592450 wait complete. 2024-04-26T18:01:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.879+0000 7fe1b4ffd740 20 client.15813 trim_cache size 0 max 16384 2024-04-26T18:01:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:01:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:01:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:01:08.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:01:08.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.879+0000 7fe1b2647640 1 -- 172.21.15.31:0/2221196680 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe1940a3dc0 msgr2=0x7fe1940a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:08.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.879+0000 7fe1b2647640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe1940a3dc0 0x7fe1940a6280 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0016740 tx=0x7fe1a001a000 comp rx=0 tx=0).stop 2024-04-26T18:01:08.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.879+0000 7fe1b2647640 1 -- 172.21.15.31:0/2221196680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1ac10f1f0 msgr2=0x7fe1ac19fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:08.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.879+0000 7fe1b2647640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1ac10f1f0 0x7fe1ac19fb70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4006010 tx=0x7fe1a402f640 comp rx=0 tx=0).stop 2024-04-26T18:01:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.879+0000 7fe1b2647640 1 -- 172.21.15.31:0/2221196680 shutdown_connections 2024-04-26T18:01:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.879+0000 7fe1b2647640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe1940a3dc0 0x7fe1940a6280 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.879+0000 7fe1b2647640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1ac10f1f0 0x7fe1ac19fb70 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.879+0000 7fe1b2647640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe1ac075600 0x7fe1ac19b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.879+0000 7fe1b2647640 1 --2- 172.21.15.31:0/2221196680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1ac074ce0 0x7fe1ac19b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.879+0000 7fe1b2647640 1 -- 172.21.15.31:0/2221196680 >> 172.21.15.31:0/2221196680 conn(0x7fe1ac0fe400 msgr2=0x7fe1ac0ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.879+0000 7fe1b2647640 1 -- 172.21.15.31:0/2221196680 shutdown_connections 2024-04-26T18:01:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:08.879+0000 7fe1b2647640 1 -- 172.21.15.31:0/2221196680 wait complete. 2024-04-26T18:01:09.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:08 smithi186 ceph-mon[32797]: pgmap v142: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 318 KiB/s wr, 27 op/s 2024-04-26T18:01:09.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:08 smithi031 ceph-mon[29164]: pgmap v142: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 318 KiB/s wr, 27 op/s 2024-04-26T18:01:09.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:08 smithi097 ceph-mon[32419]: pgmap v142: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 318 KiB/s wr, 27 op/s 2024-04-26T18:01:10.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.863+0000 7f28dedee700 1 -- 172.21.15.31:0/998092218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d8108380 msgr2=0x7f28d8108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:10.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.863+0000 7f28dedee700 1 --2- 172.21.15.31:0/998092218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d8108380 0x7f28d8108760 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009990 tx=0x7f28cc009ca0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.863+0000 7f28dedee700 1 -- 172.21.15.31:0/998092218 shutdown_connections 2024-04-26T18:01:10.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.863+0000 7f28dedee700 1 --2- 172.21.15.31:0/998092218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d8102cd0 0x7f28d810f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.863+0000 7f28dedee700 1 --2- 172.21.15.31:0/998092218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d8108380 0x7f28d8108760 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.863+0000 7f28dedee700 1 --2- 172.21.15.31:0/998092218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28d8102310 0x7f28d8102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.863+0000 7f28dedee700 1 -- 172.21.15.31:0/998092218 >> 172.21.15.31:0/998092218 conn(0x7f28d8079290 msgr2=0x7f28d80796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:10.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.863+0000 7f28dedee700 1 -- 172.21.15.31:0/998092218 shutdown_connections 2024-04-26T18:01:10.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.863+0000 7f28dedee700 1 -- 172.21.15.31:0/998092218 wait complete. 2024-04-26T18:01:10.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.864+0000 7f28dedee700 1 Processor -- start 2024-04-26T18:01:10.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.864+0000 7f28dedee700 1 -- start start 2024-04-26T18:01:10.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.865+0000 7f28dedee700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d8102310 0x7f28d819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.865+0000 7f28dedee700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28d8102cd0 0x7f28d819fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.865+0000 7f28dedee700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d8108380 0x7f28d8199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.865+0000 7f28dedee700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28d81124a0 con 0x7f28d8102310 2024-04-26T18:01:10.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.865+0000 7f28dcb8a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d8102310 0x7f28d819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.865+0000 7f28dcb8a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d8102310 0x7f28d819f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44706/0 (socket says 172.21.15.31:44706) 2024-04-26T18:01:10.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.865+0000 7f28dcb8a700 1 -- 172.21.15.31:0/3067923936 learned_addr learned my addr 172.21.15.31:0/3067923936 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.865+0000 7f28dedee700 1 -- 172.21.15.31:0/3067923936 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f28d8112320 con 0x7f28d8102cd0 2024-04-26T18:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.865+0000 7f28dedee700 1 -- 172.21.15.31:0/3067923936 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28d8112620 con 0x7f28d8108380 2024-04-26T18:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.865+0000 7f28d7fff700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28d8102cd0 0x7f28d819fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.865+0000 7f28dd38b700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d8108380 0x7f28d8199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.866+0000 7f28dcb8a700 1 -- 172.21.15.31:0/3067923936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d8108380 msgr2=0x7f28d8199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.866+0000 7f28dcb8a700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d8108380 0x7f28d8199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.866+0000 7f28dcb8a700 1 -- 172.21.15.31:0/3067923936 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28d8102cd0 msgr2=0x7f28d819fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.866+0000 7f28dcb8a700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28d8102cd0 0x7f28d819fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.866+0000 7f28dcb8a700 1 -- 172.21.15.31:0/3067923936 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d819a0f0 con 0x7f28d8102310 2024-04-26T18:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.866+0000 7f28dd38b700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d8108380 0x7f28d8199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.866+0000 7f28dcb8a700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d8102310 0x7f28d819f780 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f28cc000c00 tx=0x7f28cc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.866+0000 7f28d5ffb700 1 -- 172.21.15.31:0/3067923936 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc027070 con 0x7f28d8102310 2024-04-26T18:01:10.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.866+0000 7f28dedee700 1 -- 172.21.15.31:0/3067923936 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d819a380 con 0x7f28d8102310 2024-04-26T18:01:10.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.866+0000 7f28d5ffb700 1 -- 172.21.15.31:0/3067923936 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28cc003f20 con 0x7f28d8102310 2024-04-26T18:01:10.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.866+0000 7f28d5ffb700 1 -- 172.21.15.31:0/3067923936 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc022cf0 con 0x7f28d8102310 2024-04-26T18:01:10.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.866+0000 7f28dedee700 1 -- 172.21.15.31:0/3067923936 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d81a6650 con 0x7f28d8102310 2024-04-26T18:01:10.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.870+0000 7f28d5ffb700 1 -- 172.21.15.31:0/3067923936 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f28cc02a070 con 0x7f28d8102310 2024-04-26T18:01:10.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.870+0000 7f28e9680b80 1 Processor -- start 2024-04-26T18:01:10.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.870+0000 7f28e9680b80 1 -- start start 2024-04-26T18:01:10.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28e9680b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faab1a0 0x55801faab580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28d5ffb700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f28c80a3e30 0x7f28c80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28e9680b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faabac0 0x55801faba840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28e9680b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 0x55801fabd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28e9680b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55801f9fe840 con 0x55801faab1a0 2024-04-26T18:01:10.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28e9680b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55801f99d6e0 con 0x55801fabad80 2024-04-26T18:01:10.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28e9680b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55801f9fa800 con 0x55801faabac0 2024-04-26T18:01:10.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28dcb8a700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f28c80a3e30 0x7f28c80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28d7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faabac0 0x55801faba840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28d5ffb700 1 -- 172.21.15.31:0/3067923936 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f28cc0d4300 con 0x7f28d8102310 2024-04-26T18:01:10.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28dd38b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faab1a0 0x55801faab580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28d7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faabac0 0x55801faba840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:35294/0 (socket says 172.21.15.31:35294) 2024-04-26T18:01:10.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28d7fff700 1 -- 172.21.15.31:0/2056428519 learned_addr learned my addr 172.21.15.31:0/2056428519 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:10.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28d7fff700 1 -- 172.21.15.31:0/2056428519 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 msgr2=0x55801fabd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:10.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28d7fff700 1 --2- 172.21.15.31:0/2056428519 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 0x55801fabd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28d7fff700 1 -- 172.21.15.31:0/2056428519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faab1a0 msgr2=0x55801faab580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:10.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28d7fff700 1 --2- 172.21.15.31:0/2056428519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faab1a0 0x55801faab580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.871+0000 7f28d7fff700 1 -- 172.21.15.31:0/2056428519 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55801f94b9d0 con 0x55801faabac0 2024-04-26T18:01:10.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.872+0000 7f28dd38b700 1 --2- 172.21.15.31:0/2056428519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faab1a0 0x55801faab580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:10.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.872+0000 7f28d7fff700 1 --2- 172.21.15.31:0/2056428519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faabac0 0x55801faba840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f28c400ac80 tx=0x7f28c400af90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bf88d24a613ce93 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:10.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.872+0000 7f28dcb8a700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f28c80a3e30 0x7f28c80a62f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f28d8108ca0 tx=0x7f28cc01ee40 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:10.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.872+0000 7f28c2ffd700 1 -- 172.21.15.31:0/2056428519 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c401d070 con 0x55801faabac0 2024-04-26T18:01:10.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.872+0000 7f28c2ffd700 1 -- 172.21.15.31:0/2056428519 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28c400fe90 con 0x55801faabac0 2024-04-26T18:01:10.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.873+0000 7f28c2ffd700 1 -- 172.21.15.31:0/2056428519 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c40108c0 con 0x55801faabac0 2024-04-26T18:01:10.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.873+0000 7f28e9680b80 1 -- 172.21.15.31:0/2056428519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faabac0 msgr2=0x55801faba840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:10.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.873+0000 7f28e9680b80 1 --2- 172.21.15.31:0/2056428519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faabac0 0x55801faba840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f28c400ac80 tx=0x7f28c400af90 comp rx=0 tx=0).stop 2024-04-26T18:01:10.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.873+0000 7f28e9680b80 1 -- 172.21.15.31:0/2056428519 shutdown_connections 2024-04-26T18:01:10.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.873+0000 7f28e9680b80 1 --2- 172.21.15.31:0/2056428519 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 0x55801fabd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.873+0000 7f28e9680b80 1 --2- 172.21.15.31:0/2056428519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faab1a0 0x55801faab580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.873+0000 7f28e9680b80 1 --2- 172.21.15.31:0/2056428519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faabac0 0x55801faba840 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.873+0000 7f28e9680b80 1 -- 172.21.15.31:0/2056428519 >> 172.21.15.31:0/2056428519 conn(0x55801f9d8e40 msgr2=0x55801faa7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:10.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.873+0000 7f28e9680b80 1 -- 172.21.15.31:0/2056428519 shutdown_connections 2024-04-26T18:01:10.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.873+0000 7f28e9680b80 1 -- 172.21.15.31:0/2056428519 wait complete. 2024-04-26T18:01:10.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.874+0000 7f28e9680b80 1 -- start start 2024-04-26T18:01:10.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.874+0000 7f28e9680b80 1 Processor -- start 2024-04-26T18:01:10.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.874+0000 7f28e9680b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faab1a0 0x55801faba490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28e9680b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faabac0 0x55801fab7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28e9680b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 0x55801fab7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28e9680b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55801fabd9f0 con 0x55801faabac0 2024-04-26T18:01:10.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28e9680b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55801f964a50 con 0x55801fabad80 2024-04-26T18:01:10.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28e9680b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55801fabdb70 con 0x55801faab1a0 2024-04-26T18:01:10.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28e9680b80 10 client.0 _ll_register_callbacks cb 0x55801f992680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:01:10.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28d7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faabac0 0x55801fab7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28d7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faabac0 0x55801fab7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44736/0 (socket says 172.21.15.31:44736) 2024-04-26T18:01:10.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28d7fff700 1 -- 172.21.15.31:0/4161166328 learned_addr learned my addr 172.21.15.31:0/4161166328 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:10.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28dd38b700 1 --2- 172.21.15.31:0/4161166328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faab1a0 0x55801faba490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28dd38b700 1 --2- 172.21.15.31:0/4161166328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 0x55801fab7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28d7fff700 1 -- 172.21.15.31:0/4161166328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faab1a0 msgr2=0x55801faba490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:10.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28d7fff700 1 --2- 172.21.15.31:0/4161166328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faab1a0 0x55801faba490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28d7fff700 1 -- 172.21.15.31:0/4161166328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 msgr2=0x55801fab7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:10.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28d7fff700 1 --2- 172.21.15.31:0/4161166328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 0x55801fab7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28d7fff700 1 -- 172.21.15.31:0/4161166328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55801fab8310 con 0x55801faabac0 2024-04-26T18:01:10.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.875+0000 7f28dd38b700 1 --2- 172.21.15.31:0/4161166328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 0x55801fab7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:10.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.876+0000 7f28d7fff700 1 --2- 172.21.15.31:0/4161166328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faabac0 0x55801fab7790 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f28c4000c00 tx=0x7f28c40056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ea7f6e6901cf68 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:10.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.876+0000 7f28b7fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:01:10.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.876+0000 7f28b7fff700 1 -- 172.21.15.31:0/4161166328 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4026020 con 0x55801faabac0 2024-04-26T18:01:10.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.876+0000 7f28b7fff700 1 -- 172.21.15.31:0/4161166328 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f28c4004750 con 0x55801faabac0 2024-04-26T18:01:10.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.876+0000 7f28e9680b80 1 -- 172.21.15.31:0/4161166328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28c400fe90 con 0x55801faabac0 2024-04-26T18:01:10.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.876+0000 7f28b7fff700 1 -- 172.21.15.31:0/4161166328 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f28c4031050 con 0x55801faabac0 2024-04-26T18:01:10.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.876+0000 7f28e9680b80 10 client.15825 fetch_fsmap learned FSMap version 32 2024-04-26T18:01:10.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.876+0000 7f28e9680b80 1 -- 172.21.15.31:0/4161166328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55801fac8450 con 0x55801faabac0 2024-04-26T18:01:10.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.876+0000 7f28b7fff700 1 -- 172.21.15.31:0/4161166328 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28c4004e00 con 0x55801faabac0 2024-04-26T18:01:10.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.877+0000 7f28b7fff700 1 -- 172.21.15.31:0/4161166328 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f28c4010430 con 0x55801faabac0 2024-04-26T18:01:10.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.877+0000 7f28e9680b80 10 client.15825 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:01:10.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.877+0000 7f28e9680b80 10 client.15825 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T18:01:10.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.877+0000 7f28e9680b80 20 client.15825 populate_metadata read hostname 'smithi031' 2024-04-26T18:01:10.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.877+0000 7f28e9680b80 1 --2- 172.21.15.31:0/4161166328 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55801faca9d0 0x55801faccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.877+0000 7f28e9680b80 4 client.15825 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:01:10.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.877+0000 7f28e9680b80 1 -- 172.21.15.31:0/4161166328 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f28c400fe90 con 0x55801faca9d0 2024-04-26T18:01:10.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.877+0000 7f28dd38b700 1 --2- 172.21.15.31:0/4161166328 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55801faca9d0 0x55801faccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.877+0000 7f28dd38b700 1 --2- 172.21.15.31:0/4161166328 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55801faca9d0 0x55801faccdc0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17cbddf433842a3 server_cookie=74b3493d7a4f643a in_seq=0 out_seq=0 2024-04-26T18:01:10.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.877+0000 7f28b7fff700 10 client.15825 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:10.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.878+0000 7f28b7fff700 1 -- 172.21.15.31:0/4161166328 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f28c400fe90 con 0x55801faca9d0 2024-04-26T18:01:10.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.878+0000 7f28b7fff700 10 client.15825 handle_command_reply: tid=0 2024-04-26T18:01:10.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.878+0000 7f28e9680b80 1 client.15825 shutdown 2024-04-26T18:01:10.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.878+0000 7f28e9680b80 1 -- 172.21.15.31:0/4161166328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faabac0 msgr2=0x55801fab7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:10.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.878+0000 7f28e9680b80 1 --2- 172.21.15.31:0/4161166328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faabac0 0x55801fab7790 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f28c4000c00 tx=0x7f28c40056a0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.878+0000 7f28e9680b80 1 -- 172.21.15.31:0/4161166328 shutdown_connections 2024-04-26T18:01:10.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.878+0000 7f28e9680b80 1 --2- 172.21.15.31:0/4161166328 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55801faca9d0 0x55801faccdc0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.878+0000 7f28e9680b80 1 --2- 172.21.15.31:0/4161166328 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 0x55801fab7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.878+0000 7f28e9680b80 1 --2- 172.21.15.31:0/4161166328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faab1a0 0x55801faba490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.878+0000 7f28b7fff700 0 client.15825 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:10.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.878+0000 7f28e9680b80 1 --2- 172.21.15.31:0/4161166328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faabac0 0x55801fab7790 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.878+0000 7f28e9680b80 1 -- 172.21.15.31:0/4161166328 >> 172.21.15.31:0/4161166328 conn(0x55801f9d8e40 msgr2=0x55801faa72c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:10.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.879+0000 7f28e9680b80 1 -- 172.21.15.31:0/4161166328 shutdown_connections 2024-04-26T18:01:10.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.879+0000 7f28e9680b80 1 -- 172.21.15.31:0/4161166328 wait complete. 2024-04-26T18:01:10.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.879+0000 7f28e9680b80 20 client.15825 trim_cache size 0 max 16384 2024-04-26T18:01:10.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.921+0000 7f28e9680b80 1 Processor -- start 2024-04-26T18:01:10.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.921+0000 7f28e9680b80 1 -- start start 2024-04-26T18:01:10.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.921+0000 7f28e9680b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faab1a0 0x55801faad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.921+0000 7f28e9680b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faabac0 0x55801fac6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.921+0000 7f28e9680b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 0x55801fad23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.921+0000 7f28e9680b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28d81ae240 con 0x55801faabac0 2024-04-26T18:01:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.921+0000 7f28e9680b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f28d81aba70 con 0x55801fabad80 2024-04-26T18:01:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.921+0000 7f28e9680b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28d81a98e0 con 0x55801faab1a0 2024-04-26T18:01:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.921+0000 7f28d7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faabac0 0x55801fac6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.921+0000 7f28d7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faabac0 0x55801fac6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44742/0 (socket says 172.21.15.31:44742) 2024-04-26T18:01:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.921+0000 7f28d7fff700 1 -- 172.21.15.31:0/3972912341 learned_addr learned my addr 172.21.15.31:0/3972912341 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.921+0000 7f28dd38b700 1 --2- 172.21.15.31:0/3972912341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 0x55801fad23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.921+0000 7f28dd38b700 1 --2- 172.21.15.31:0/3972912341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faab1a0 0x55801faad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.922+0000 7f28d7fff700 1 -- 172.21.15.31:0/3972912341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faab1a0 msgr2=0x55801faad9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.922+0000 7f28d7fff700 1 --2- 172.21.15.31:0/3972912341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faab1a0 0x55801faad9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.922+0000 7f28d7fff700 1 -- 172.21.15.31:0/3972912341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 msgr2=0x55801fad23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.922+0000 7f28d7fff700 1 --2- 172.21.15.31:0/3972912341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 0x55801fad23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.922+0000 7f28d7fff700 1 -- 172.21.15.31:0/3972912341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55801fabdb90 con 0x55801faabac0 2024-04-26T18:01:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.922+0000 7f28dd38b700 1 --2- 172.21.15.31:0/3972912341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faab1a0 0x55801faad9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.922+0000 7f28d7fff700 1 --2- 172.21.15.31:0/3972912341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faabac0 0x55801fac6940 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f28c40059d0 tx=0x7f28c400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db7c927430fcc77 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.922+0000 7f28c27fc700 1 -- 172.21.15.31:0/3972912341 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c401d040 con 0x55801faabac0 2024-04-26T18:01:10.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.922+0000 7f28c27fc700 1 -- 172.21.15.31:0/3972912341 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28c4010630 con 0x55801faabac0 2024-04-26T18:01:10.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.922+0000 7f28c27fc700 1 -- 172.21.15.31:0/3972912341 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4004950 con 0x55801faabac0 2024-04-26T18:01:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.923+0000 7f28e9680b80 1 -- 172.21.15.31:0/3972912341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faabac0 msgr2=0x55801fac6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.923+0000 7f28e9680b80 1 --2- 172.21.15.31:0/3972912341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faabac0 0x55801fac6940 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f28c40059d0 tx=0x7f28c400a620 comp rx=0 tx=0).stop 2024-04-26T18:01:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.923+0000 7f28e9680b80 1 -- 172.21.15.31:0/3972912341 shutdown_connections 2024-04-26T18:01:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.923+0000 7f28e9680b80 1 --2- 172.21.15.31:0/3972912341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801fabad80 0x55801fad23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.923+0000 7f28e9680b80 1 --2- 172.21.15.31:0/3972912341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faab1a0 0x55801faad9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.923+0000 7f28e9680b80 1 --2- 172.21.15.31:0/3972912341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801faabac0 0x55801fac6940 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.923+0000 7f28e9680b80 1 -- 172.21.15.31:0/3972912341 >> 172.21.15.31:0/3972912341 conn(0x55801f9d8e40 msgr2=0x55801fab2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.923+0000 7f28e9680b80 1 -- 172.21.15.31:0/3972912341 shutdown_connections 2024-04-26T18:01:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.923+0000 7f28e9680b80 1 -- 172.21.15.31:0/3972912341 wait complete. 2024-04-26T18:01:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.923+0000 7f28e9680b80 1 -- start start 2024-04-26T18:01:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.923+0000 7f28e9680b80 1 Processor -- start 2024-04-26T18:01:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28e9680b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801faab1a0 0x55801fac2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28e9680b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faabac0 0x55801fac3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28e9680b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801fabad80 0x55801fabfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28e9680b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28d81a9f30 con 0x55801fabad80 2024-04-26T18:01:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28e9680b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f28d81ab3a0 con 0x55801faab1a0 2024-04-26T18:01:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28e9680b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28d81aa600 con 0x55801faabac0 2024-04-26T18:01:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28e9680b80 10 client.0 _ll_register_callbacks cb 0x55801f992680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:01:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28dd38b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801fabad80 0x55801fabfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28dd38b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801fabad80 0x55801fabfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44752/0 (socket says 172.21.15.31:44752) 2024-04-26T18:01:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28dd38b700 1 -- 172.21.15.31:0/825156767 learned_addr learned my addr 172.21.15.31:0/825156767 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28dd38b700 1 --2- 172.21.15.31:0/825156767 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801faab1a0 0x55801fac2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28d7fff700 1 --2- 172.21.15.31:0/825156767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faabac0 0x55801fac3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28dd38b700 1 -- 172.21.15.31:0/825156767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faabac0 msgr2=0x55801fac3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28dd38b700 1 --2- 172.21.15.31:0/825156767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faabac0 0x55801fac3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28dd38b700 1 -- 172.21.15.31:0/825156767 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801faab1a0 msgr2=0x55801fac2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28dd38b700 1 --2- 172.21.15.31:0/825156767 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801faab1a0 0x55801fac2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28dd38b700 1 -- 172.21.15.31:0/825156767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55801fac0660 con 0x55801fabad80 2024-04-26T18:01:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28dd38b700 1 --2- 172.21.15.31:0/825156767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801fabad80 0x55801fabfe20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f28d0007730 tx=0x7f28d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f330a76c321c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.924+0000 7f28b7fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:01:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.925+0000 7f28b7fff700 1 -- 172.21.15.31:0/825156767 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d001a650 con 0x55801fabad80 2024-04-26T18:01:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.925+0000 7f28e9680b80 1 -- 172.21.15.31:0/825156767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28c4010630 con 0x55801fabad80 2024-04-26T18:01:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.925+0000 7f28b7fff700 1 -- 172.21.15.31:0/825156767 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f28d001a7f0 con 0x55801fabad80 2024-04-26T18:01:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.925+0000 7f28b7fff700 1 -- 172.21.15.31:0/825156767 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f28d001a9f0 con 0x55801fabad80 2024-04-26T18:01:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.925+0000 7f28e9680b80 10 client.15831 fetch_fsmap learned FSMap version 32 2024-04-26T18:01:10.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.925+0000 7f28e9680b80 1 -- 172.21.15.31:0/825156767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55801fb31750 con 0x55801fabad80 2024-04-26T18:01:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.925+0000 7f28b7fff700 1 -- 172.21.15.31:0/825156767 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28c4010630 con 0x55801fabad80 2024-04-26T18:01:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.925+0000 7f28b7fff700 1 -- 172.21.15.31:0/825156767 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f28d002da40 con 0x55801fabad80 2024-04-26T18:01:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.925+0000 7f28e9680b80 10 client.15831 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:01:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.925+0000 7f28e9680b80 10 client.15831 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T18:01:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.925+0000 7f28e9680b80 20 client.15831 populate_metadata read hostname 'smithi031' 2024-04-26T18:01:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.925+0000 7f28e9680b80 1 --2- 172.21.15.31:0/825156767 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55801fb31e70 0x55801fb34290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.925+0000 7f28e9680b80 4 client.15831 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "7562d4d8", "release": true, "await": true}] 2024-04-26T18:01:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.925+0000 7f28e9680b80 1 -- 172.21.15.31:0/825156767 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "quiesce db", "set_id": "7562d4d8", "release": true, "await": true}) -- 0x55801fa0c220 con 0x55801fb31e70 2024-04-26T18:01:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.926+0000 7f28d7fff700 1 --2- 172.21.15.31:0/825156767 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55801fb31e70 0x55801fb34290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.926+0000 7f28b7fff700 10 client.15831 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.926+0000 7f28d7fff700 1 --2- 172.21.15.31:0/825156767 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55801fb31e70 0x55801fb34290 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa13d43f4f5a52db server_cookie=161e041940c69436 in_seq=0 out_seq=0 2024-04-26T18:01:10.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.945+0000 7f28b7fff700 1 -- 172.21.15.31:0/825156767 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55801fa0c220 con 0x55801fb31e70 2024-04-26T18:01:10.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.945+0000 7f28b7fff700 10 client.15831 handle_command_reply: tid=0 2024-04-26T18:01:10.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.945+0000 7f28e9680b80 1 client.15831 shutdown 2024-04-26T18:01:10.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.945+0000 7f28e9680b80 1 -- 172.21.15.31:0/825156767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801fabad80 msgr2=0x55801fabfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:10.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.945+0000 7f28e9680b80 1 --2- 172.21.15.31:0/825156767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801fabad80 0x55801fabfe20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f28d0007730 tx=0x7f28d0004ad0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.945+0000 7f28e9680b80 1 -- 172.21.15.31:0/825156767 shutdown_connections 2024-04-26T18:01:10.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.945+0000 7f28e9680b80 1 --2- 172.21.15.31:0/825156767 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55801fb31e70 0x55801fb34290 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.945+0000 7f28e9680b80 1 --2- 172.21.15.31:0/825156767 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55801faab1a0 0x55801fac2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.945+0000 7f28e9680b80 1 --2- 172.21.15.31:0/825156767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55801fabad80 0x55801fabfe20 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.945+0000 7f28e9680b80 1 --2- 172.21.15.31:0/825156767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55801faabac0 0x55801fac3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.945+0000 7f28e9680b80 1 -- 172.21.15.31:0/825156767 >> 172.21.15.31:0/825156767 conn(0x55801f9d8e40 msgr2=0x55801fab9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:10.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.945+0000 7f28b7fff700 0 client.15831 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:10.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.946+0000 7f28e9680b80 1 -- 172.21.15.31:0/825156767 shutdown_connections 2024-04-26T18:01:10.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.946+0000 7f28e9680b80 1 -- 172.21.15.31:0/825156767 wait complete. 2024-04-26T18:01:10.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.946+0000 7f28e9680b80 20 client.15831 trim_cache size 0 max 16384 2024-04-26T18:01:10.947 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T18:01:10.947 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 32, 2024-04-26T18:01:10.948 INFO:teuthology.orchestra.run.smithi031.stdout: "leader": 24473, 2024-04-26T18:01:10.948 INFO:teuthology.orchestra.run.smithi031.stdout: "set_version": 12, 2024-04-26T18:01:10.948 INFO:teuthology.orchestra.run.smithi031.stdout: "sets": { 2024-04-26T18:01:10.948 INFO:teuthology.orchestra.run.smithi031.stdout: "7562d4d8": { 2024-04-26T18:01:10.948 INFO:teuthology.orchestra.run.smithi031.stdout: "version": 12, 2024-04-26T18:01:10.948 INFO:teuthology.orchestra.run.smithi031.stdout: "age_ref": 0.0, 2024-04-26T18:01:10.948 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T18:01:10.948 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RELEASED", 2024-04-26T18:01:10.948 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T18:01:10.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:01:10.948 INFO:teuthology.orchestra.run.smithi031.stdout: "timeout": 60.0, 2024-04-26T18:01:10.948 INFO:teuthology.orchestra.run.smithi031.stdout: "expiration": 79.5, 2024-04-26T18:01:10.949 INFO:teuthology.orchestra.run.smithi031.stdout: "members": { 2024-04-26T18:01:10.949 INFO:teuthology.orchestra.run.smithi031.stdout: "file:/": { 2024-04-26T18:01:10.949 INFO:teuthology.orchestra.run.smithi031.stdout: "excluded": false, 2024-04-26T18:01:10.949 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T18:01:10.949 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RELEASED", 2024-04-26T18:01:10.949 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T18:01:10.949 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:10.949 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:10.949 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:10.949 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:10.949 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:10.949 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T18:01:10.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.946+0000 7f28dedee700 1 -- 172.21.15.31:0/3067923936 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f28c80a3e30 msgr2=0x7f28c80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:10.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.946+0000 7f28dedee700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f28c80a3e30 0x7f28c80a62f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f28d8108ca0 tx=0x7f28cc01ee40 comp rx=0 tx=0).stop 2024-04-26T18:01:10.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.946+0000 7f28dedee700 1 -- 172.21.15.31:0/3067923936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d8102310 msgr2=0x7f28d819f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:10.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.946+0000 7f28dedee700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d8102310 0x7f28d819f780 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f28cc000c00 tx=0x7f28cc0047b0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.946+0000 7f28dedee700 1 -- 172.21.15.31:0/3067923936 shutdown_connections 2024-04-26T18:01:10.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.946+0000 7f28dedee700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f28c80a3e30 0x7f28c80a62f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.946+0000 7f28dedee700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d8108380 0x7f28d8199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.946+0000 7f28dedee700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d8102310 0x7f28d819f780 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.946+0000 7f28dedee700 1 --2- 172.21.15.31:0/3067923936 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f28d8102cd0 0x7f28d819fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:10.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.946+0000 7f28dedee700 1 -- 172.21.15.31:0/3067923936 >> 172.21.15.31:0/3067923936 conn(0x7f28d8079290 msgr2=0x7f28d806bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:10.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.947+0000 7f28dedee700 1 -- 172.21.15.31:0/3067923936 shutdown_connections 2024-04-26T18:01:10.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:10.947+0000 7f28dedee700 1 -- 172.21.15.31:0/3067923936 wait complete. 2024-04-26T18:01:11.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:10 smithi186 ceph-mon[32797]: pgmap v143: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 KiB/s wr, 22 op/s 2024-04-26T18:01:11.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:10 smithi031 ceph-mon[29164]: pgmap v143: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 KiB/s wr, 22 op/s 2024-04-26T18:01:11.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:10 smithi097 ceph-mon[32419]: pgmap v143: 97 pgs: 97 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 KiB/s wr, 22 op/s 2024-04-26T18:01:11.971 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '7562d4d8', seconds elapsed: 34.2 2024-04-26T18:01:11.971 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.5 2024-04-26T18:01:11.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T18:01:12.394 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:01:13.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 230.0 s, 555.2 tps, lat 1030.589 ms stddev 4930.981 2024-04-26T18:01:13.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:12 smithi186 ceph-mon[32797]: pgmap v144: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 MiB/s wr, 576 op/s 2024-04-26T18:01:13.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:12 smithi031 ceph-mon[29164]: pgmap v144: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 MiB/s wr, 576 op/s 2024-04-26T18:01:13.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:12 smithi097 ceph-mon[32419]: pgmap v144: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 MiB/s wr, 576 op/s 2024-04-26T18:01:14.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.373+0000 7f57011b6700 1 -- 172.21.15.31:0/4256311065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56fc10fa50 msgr2=0x7f56fc10fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:14.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.373+0000 7f57011b6700 1 --2- 172.21.15.31:0/4256311065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56fc10fa50 0x7f56fc10fe30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009990 tx=0x7f56ec009ca0 comp rx=0 tx=0).stop 2024-04-26T18:01:14.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.374+0000 7f57011b6700 1 -- 172.21.15.31:0/4256311065 shutdown_connections 2024-04-26T18:01:14.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.374+0000 7f57011b6700 1 --2- 172.21.15.31:0/4256311065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc10a410 0x7f56fc116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:14.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.374+0000 7f57011b6700 1 --2- 172.21.15.31:0/4256311065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56fc10fa50 0x7f56fc10fe30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:14.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.374+0000 7f57011b6700 1 --2- 172.21.15.31:0/4256311065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56fc109a50 0x7f56fc109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:14.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.374+0000 7f57011b6700 1 -- 172.21.15.31:0/4256311065 >> 172.21.15.31:0/4256311065 conn(0x7f56fc076bf0 msgr2=0x7f56fc108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:14.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.374+0000 7f57011b6700 1 -- 172.21.15.31:0/4256311065 shutdown_connections 2024-04-26T18:01:14.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.374+0000 7f57011b6700 1 -- 172.21.15.31:0/4256311065 wait complete. 2024-04-26T18:01:14.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.375+0000 7f57011b6700 1 Processor -- start 2024-04-26T18:01:14.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.375+0000 7f57011b6700 1 -- start start 2024-04-26T18:01:14.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.375+0000 7f57011b6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc109a50 0x7f56fc1a0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:14.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f57011b6700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56fc10a410 0x7f56fc1a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:14.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f57011b6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56fc10fa50 0x7f56fc1a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:14.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f57011b6700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc119d40 con 0x7f56fc109a50 2024-04-26T18:01:14.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f57011b6700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc119bc0 con 0x7f56fc10a410 2024-04-26T18:01:14.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f57011b6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc119ec0 con 0x7f56fc10fa50 2024-04-26T18:01:14.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f56fad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc109a50 0x7f56fc1a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:14.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f56fa59c700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56fc10a410 0x7f56fc1a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:14.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f56fa59c700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56fc10a410 0x7f56fc1a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:56594/0 (socket says 172.21.15.31:56594) 2024-04-26T18:01:14.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f56fa59c700 1 -- 172.21.15.31:0/687090557 learned_addr learned my addr 172.21.15.31:0/687090557 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:14.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f56fb59e700 1 --2- 172.21.15.31:0/687090557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56fc10fa50 0x7f56fc1a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:14.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f56fb59e700 1 -- 172.21.15.31:0/687090557 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56fc10a410 msgr2=0x7f56fc1a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:14.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f56fb59e700 1 --2- 172.21.15.31:0/687090557 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56fc10a410 0x7f56fc1a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:14.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f56fb59e700 1 -- 172.21.15.31:0/687090557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc109a50 msgr2=0x7f56fc1a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:14.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f56fb59e700 1 --2- 172.21.15.31:0/687090557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc109a50 0x7f56fc1a0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:14.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.376+0000 7f56fb59e700 1 -- 172.21.15.31:0/687090557 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56fc1a52a0 con 0x7f56fc10fa50 2024-04-26T18:01:14.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.377+0000 7f56fb59e700 1 --2- 172.21.15.31:0/687090557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56fc10fa50 0x7f56fc1a4b60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f56f400be20 tx=0x7f56f4009ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:14.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.377+0000 7f56e3fff700 1 -- 172.21.15.31:0/687090557 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4019070 con 0x7f56fc10fa50 2024-04-26T18:01:14.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.377+0000 7f57011b6700 1 -- 172.21.15.31:0/687090557 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56fc1a5590 con 0x7f56fc10fa50 2024-04-26T18:01:14.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.377+0000 7f56e3fff700 1 -- 172.21.15.31:0/687090557 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56f400cc10 con 0x7f56fc10fa50 2024-04-26T18:01:14.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.377+0000 7f56e3fff700 1 -- 172.21.15.31:0/687090557 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f40125a0 con 0x7f56fc10fa50 2024-04-26T18:01:14.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.377+0000 7f57011b6700 1 -- 172.21.15.31:0/687090557 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56fc1ace10 con 0x7f56fc10fa50 2024-04-26T18:01:14.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.378+0000 7f57011b6700 1 -- 172.21.15.31:0/687090557 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56fc111fe0 con 0x7f56fc10fa50 2024-04-26T18:01:14.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.380+0000 7f56e3fff700 1 -- 172.21.15.31:0/687090557 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f56f401c070 con 0x7f56fc10fa50 2024-04-26T18:01:14.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.381+0000 7f56e3fff700 1 --2- 172.21.15.31:0/687090557 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f56e40a3eb0 0x7f56e40a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:14.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.381+0000 7f56e3fff700 1 -- 172.21.15.31:0/687090557 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f56f40cfc70 con 0x7f56fc10fa50 2024-04-26T18:01:14.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.381+0000 7f56fad9d700 1 --2- 172.21.15.31:0/687090557 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f56e40a3eb0 0x7f56e40a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:14.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.382+0000 7f56fad9d700 1 --2- 172.21.15.31:0/687090557 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f56e40a3eb0 0x7f56e40a6370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009750 tx=0x7f56ec00b540 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:14.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.386+0000 7f56e3fff700 1 -- 172.21.15.31:0/687090557 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56f409c410 con 0x7f56fc10fa50 2024-04-26T18:01:14.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.805+0000 7f57011b6700 1 -- 172.21.15.31:0/687090557 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56fc1119c0 con 0x7f56fc10fa50 2024-04-26T18:01:14.807 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:01:14.807 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T18:00:06.847155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24446,24457,24473]},"id":1}]} 2024-04-26T18:01:14.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.806+0000 7f56e3fff700 1 -- 172.21.15.31:0/687090557 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11877 (secure 0 0 0) 0x7f56f409bd10 con 0x7f56fc10fa50 2024-04-26T18:01:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.810+0000 7f57011b6700 1 -- 172.21.15.31:0/687090557 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f56e40a3eb0 msgr2=0x7f56e40a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.810+0000 7f57011b6700 1 --2- 172.21.15.31:0/687090557 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f56e40a3eb0 0x7f56e40a6370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009750 tx=0x7f56ec00b540 comp rx=0 tx=0).stop 2024-04-26T18:01:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.810+0000 7f57011b6700 1 -- 172.21.15.31:0/687090557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56fc10fa50 msgr2=0x7f56fc1a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.810+0000 7f57011b6700 1 --2- 172.21.15.31:0/687090557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56fc10fa50 0x7f56fc1a4b60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f56f400be20 tx=0x7f56f4009ed0 comp rx=0 tx=0).stop 2024-04-26T18:01:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.810+0000 7f57011b6700 1 -- 172.21.15.31:0/687090557 shutdown_connections 2024-04-26T18:01:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.810+0000 7f57011b6700 1 --2- 172.21.15.31:0/687090557 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f56e40a3eb0 0x7f56e40a6370 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.810+0000 7f57011b6700 1 --2- 172.21.15.31:0/687090557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56fc10fa50 0x7f56fc1a4b60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.810+0000 7f57011b6700 1 --2- 172.21.15.31:0/687090557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56fc109a50 0x7f56fc1a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.810+0000 7f57011b6700 1 --2- 172.21.15.31:0/687090557 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56fc10a410 0x7f56fc1a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.810+0000 7f57011b6700 1 -- 172.21.15.31:0/687090557 >> 172.21.15.31:0/687090557 conn(0x7f56fc076bf0 msgr2=0x7f56fc0772d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.810+0000 7f57011b6700 1 -- 172.21.15.31:0/687090557 shutdown_connections 2024-04-26T18:01:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:14.810+0000 7f57011b6700 1 -- 172.21.15.31:0/687090557 wait complete. 2024-04-26T18:01:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 32 2024-04-26T18:01:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:14 smithi031 ceph-mon[29164]: pgmap v145: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 MiB/s wr, 556 op/s 2024-04-26T18:01:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:14 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/687090557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:01:15.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:14 smithi186 ceph-mon[32797]: pgmap v145: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 MiB/s wr, 556 op/s 2024-04-26T18:01:15.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:14 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/687090557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:01:15.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:14 smithi097 ceph-mon[32419]: pgmap v145: 97 pgs: 97 active+clean; 2.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 MiB/s wr, 556 op/s 2024-04-26T18:01:15.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:14 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/687090557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:01:16.302 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24473 quiesce db / --timeout 60 --expiration 79.5 --await-for 42.0' 2024-04-26T18:01:16.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24473 quiesce db / --timeout 60 --expiration 79.5 --await-for 42.0 2024-04-26T18:01:16.676 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:01:17.070 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:16 smithi031 ceph-mon[29164]: pgmap v146: 97 pgs: 97 active+clean; 2.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 23 MiB/s wr, 884 op/s 2024-04-26T18:01:17.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:16 smithi186 ceph-mon[32797]: pgmap v146: 97 pgs: 97 active+clean; 2.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 23 MiB/s wr, 884 op/s 2024-04-26T18:01:17.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:16 smithi097 ceph-mon[32419]: pgmap v146: 97 pgs: 97 active+clean; 2.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 23 MiB/s wr, 884 op/s 2024-04-26T18:01:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 235.0 s, 3123.0 tps, lat 31.977 ms stddev 32.789 2024-04-26T18:01:18.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.701+0000 7f1937fff700 1 -- 172.21.15.31:0/2032334190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19381099d0 msgr2=0x7f1938110350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:18.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.701+0000 7f1937fff700 1 --2- 172.21.15.31:0/2032334190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19381099d0 0x7f1938110350 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f192c0099c0 tx=0x7f192c009cd0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.701+0000 7f1937fff700 1 -- 172.21.15.31:0/2032334190 shutdown_connections 2024-04-26T18:01:18.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.701+0000 7f1937fff700 1 --2- 172.21.15.31:0/2032334190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19381099d0 0x7f1938110350 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.701+0000 7f1937fff700 1 --2- 172.21.15.31:0/2032334190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1938104440 0x7f1938104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.701+0000 7f1937fff700 1 --2- 172.21.15.31:0/2032334190 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1938104df0 0x7f19381092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.701+0000 7f1937fff700 1 -- 172.21.15.31:0/2032334190 >> 172.21.15.31:0/2032334190 conn(0x7f19380fdc70 msgr2=0x7f1938100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:18.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.701+0000 7f1937fff700 1 -- 172.21.15.31:0/2032334190 shutdown_connections 2024-04-26T18:01:18.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.701+0000 7f1937fff700 1 -- 172.21.15.31:0/2032334190 wait complete. 2024-04-26T18:01:18.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.702+0000 7f1937fff700 1 Processor -- start 2024-04-26T18:01:18.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.702+0000 7f1937fff700 1 -- start start 2024-04-26T18:01:18.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.703+0000 7f1937fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1938104440 0x7f19381a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.703+0000 7f1937fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1938104df0 0x7f19381a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.703+0000 7f1937fff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19381099d0 0x7f193819ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.703+0000 7f1937fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1938113600 con 0x7f1938104df0 2024-04-26T18:01:18.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.703+0000 7f1937fff700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f1938113480 con 0x7f19381099d0 2024-04-26T18:01:18.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.703+0000 7f1937fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1938113780 con 0x7f1938104440 2024-04-26T18:01:18.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.703+0000 7f19377fe700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19381099d0 0x7f193819ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:18.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.703+0000 7f19377fe700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19381099d0 0x7f193819ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:45856/0 (socket says 172.21.15.31:45856) 2024-04-26T18:01:18.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.703+0000 7f19377fe700 1 -- 172.21.15.31:0/281907343 learned_addr learned my addr 172.21.15.31:0/281907343 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:18.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.704+0000 7f19377fe700 1 -- 172.21.15.31:0/281907343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1938104440 msgr2=0x7f19381a1840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T18:01:18.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.704+0000 7f19377fe700 1 --2- 172.21.15.31:0/281907343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1938104440 0x7f19381a1840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.704+0000 7f19377fe700 1 -- 172.21.15.31:0/281907343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1938104df0 msgr2=0x7f19381a1d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T18:01:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.704+0000 7f19377fe700 1 --2- 172.21.15.31:0/281907343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1938104df0 0x7f19381a1d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.704+0000 7f19377fe700 1 -- 172.21.15.31:0/281907343 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193819c1b0 con 0x7f19381099d0 2024-04-26T18:01:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.705+0000 7f19377fe700 1 --2- 172.21.15.31:0/281907343 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19381099d0 0x7f193819ba50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f192c005b40 tx=0x7f192c0056a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.705+0000 7f191ffff700 1 -- 172.21.15.31:0/281907343 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c026070 con 0x7f19381099d0 2024-04-26T18:01:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.705+0000 7f191ffff700 1 -- 172.21.15.31:0/281907343 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f192c00f460 con 0x7f19381099d0 2024-04-26T18:01:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.705+0000 7f191ffff700 1 -- 172.21.15.31:0/281907343 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c009e60 con 0x7f19381099d0 2024-04-26T18:01:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.705+0000 7f1937fff700 1 -- 172.21.15.31:0/281907343 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f193819c440 con 0x7f19381099d0 2024-04-26T18:01:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.705+0000 7f1937fff700 1 -- 172.21.15.31:0/281907343 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19381a8710 con 0x7f19381099d0 2024-04-26T18:01:18.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.708+0000 7f191ffff700 1 -- 172.21.15.31:0/281907343 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f192c029070 con 0x7f19381099d0 2024-04-26T18:01:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.708+0000 7f191ffff700 1 --2- 172.21.15.31:0/281907343 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f19200a3f50 0x7f19200a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.709+0000 7f191ffff700 1 -- 172.21.15.31:0/281907343 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f192c0d10f0 con 0x7f19381099d0 2024-04-26T18:01:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.709+0000 7f1936ffd700 1 --2- 172.21.15.31:0/281907343 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f19200a3f50 0x7f19200a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:18.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.710+0000 7f1936ffd700 1 --2- 172.21.15.31:0/281907343 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f19200a3f50 0x7f19200a6410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1938109750 tx=0x7f1928009260 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:18.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.711+0000 7f1947f08b80 1 Processor -- start 2024-04-26T18:01:18.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.711+0000 7f1947f08b80 1 -- start start 2024-04-26T18:01:18.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f1947f08b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 0x560d8ea89820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f1947f08b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea93740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f1947f08b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8ea96070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f1947f08b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560d8e9db8d0 con 0x560d8ea89440 2024-04-26T18:01:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f1947f08b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x560d8e97a770 con 0x560d8ea89d60 2024-04-26T18:01:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f1947f08b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560d8e9d7890 con 0x560d8ea93c80 2024-04-26T18:01:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f1936ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 0x560d8ea89820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f1936ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 0x560d8ea89820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36052/0 (socket says 172.21.15.31:36052) 2024-04-26T18:01:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f1936ffd700 1 -- 172.21.15.31:0/512757323 learned_addr learned my addr 172.21.15.31:0/512757323 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:18.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f19367fc700 1 --2- 172.21.15.31:0/512757323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea93740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:18.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f19377fe700 1 --2- 172.21.15.31:0/512757323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8ea96070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:18.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f19367fc700 1 -- 172.21.15.31:0/512757323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 msgr2=0x560d8ea96070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:18.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f19367fc700 1 --2- 172.21.15.31:0/512757323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8ea96070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f19367fc700 1 -- 172.21.15.31:0/512757323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 msgr2=0x560d8ea89820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:18.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.712+0000 7f19367fc700 1 --2- 172.21.15.31:0/512757323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 0x560d8ea89820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.713+0000 7f19367fc700 1 -- 172.21.15.31:0/512757323 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d8e928a60 con 0x560d8ea89d60 2024-04-26T18:01:18.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.713+0000 7f19377fe700 1 --2- 172.21.15.31:0/512757323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8ea96070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:01:18.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.713+0000 7f19367fc700 1 --2- 172.21.15.31:0/512757323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea93740 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f193806b3e0 tx=0x7f1930009dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9f2c7c45c6a07d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:18.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.713+0000 7f191d7fa700 1 -- 172.21.15.31:0/512757323 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1930007c00 con 0x560d8ea89d60 2024-04-26T18:01:18.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.713+0000 7f191d7fa700 1 -- 172.21.15.31:0/512757323 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1930007da0 con 0x560d8ea89d60 2024-04-26T18:01:18.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.713+0000 7f191d7fa700 1 -- 172.21.15.31:0/512757323 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19300133f0 con 0x560d8ea89d60 2024-04-26T18:01:18.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.714+0000 7f1947f08b80 1 -- 172.21.15.31:0/512757323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 msgr2=0x560d8ea93740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:18.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.714+0000 7f1947f08b80 1 --2- 172.21.15.31:0/512757323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea93740 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f193806b3e0 tx=0x7f1930009dd0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.714+0000 7f1947f08b80 1 -- 172.21.15.31:0/512757323 shutdown_connections 2024-04-26T18:01:18.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.714+0000 7f1947f08b80 1 --2- 172.21.15.31:0/512757323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8ea96070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.714+0000 7f1947f08b80 1 --2- 172.21.15.31:0/512757323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 0x560d8ea89820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.714+0000 7f1947f08b80 1 --2- 172.21.15.31:0/512757323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea93740 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.714+0000 7f1947f08b80 1 -- 172.21.15.31:0/512757323 >> 172.21.15.31:0/512757323 conn(0x560d8e9b5ed0 msgr2=0x560d8ea848f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.715+0000 7f1947f08b80 1 -- 172.21.15.31:0/512757323 shutdown_connections 2024-04-26T18:01:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.715+0000 7f1947f08b80 1 -- 172.21.15.31:0/512757323 wait complete. 2024-04-26T18:01:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.715+0000 7f1947f08b80 1 -- start start 2024-04-26T18:01:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.715+0000 7f1947f08b80 1 Processor -- start 2024-04-26T18:01:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.715+0000 7f1947f08b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 0x560d8ea93410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.715+0000 7f1947f08b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea90710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.715+0000 7f1947f08b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8ea90c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.715+0000 7f1947f08b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560d8ea96900 con 0x560d8ea89440 2024-04-26T18:01:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.715+0000 7f1947f08b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x560d8e941ae0 con 0x560d8ea89d60 2024-04-26T18:01:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.715+0000 7f1947f08b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560d8ea96a80 con 0x560d8ea93c80 2024-04-26T18:01:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.716+0000 7f1947f08b80 10 client.0 _ll_register_callbacks cb 0x560d8e96f710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:01:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.716+0000 7f19367fc700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea90710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.716+0000 7f1936ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 0x560d8ea93410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.716+0000 7f19367fc700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea90710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:45872/0 (socket says 172.21.15.31:45872) 2024-04-26T18:01:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.716+0000 7f1936ffd700 1 --2- 172.21.15.31:0/3119928298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 0x560d8ea93410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36062/0 (socket says 172.21.15.31:36062) 2024-04-26T18:01:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.716+0000 7f19367fc700 1 -- 172.21.15.31:0/3119928298 learned_addr learned my addr 172.21.15.31:0/3119928298 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.716+0000 7f19377fe700 1 --2- 172.21.15.31:0/3119928298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8ea90c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.716+0000 7f1936ffd700 1 -- 172.21.15.31:0/3119928298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 msgr2=0x560d8ea90c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.716+0000 7f1936ffd700 1 --2- 172.21.15.31:0/3119928298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8ea90c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.716+0000 7f1936ffd700 1 -- 172.21.15.31:0/3119928298 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 msgr2=0x560d8ea90710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.716+0000 7f1936ffd700 1 --2- 172.21.15.31:0/3119928298 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea90710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.716+0000 7f1936ffd700 1 -- 172.21.15.31:0/3119928298 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d8ea91310 con 0x560d8ea89440 2024-04-26T18:01:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.716+0000 7f1936ffd700 1 --2- 172.21.15.31:0/3119928298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 0x560d8ea93410 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f192801d4b0 tx=0x7f19280047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712cc1e99e2c4654 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.716+0000 7f191aff5700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:01:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.717+0000 7f191aff5700 1 -- 172.21.15.31:0/3119928298 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192802a060 con 0x560d8ea89440 2024-04-26T18:01:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.717+0000 7f191aff5700 1 -- 172.21.15.31:0/3119928298 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f1928010900 con 0x560d8ea89440 2024-04-26T18:01:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.717+0000 7f1947f08b80 1 -- 172.21.15.31:0/3119928298 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1930007da0 con 0x560d8ea89440 2024-04-26T18:01:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.717+0000 7f191aff5700 1 -- 172.21.15.31:0/3119928298 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1928010b00 con 0x560d8ea89440 2024-04-26T18:01:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.717+0000 7f1947f08b80 10 client.15843 fetch_fsmap learned FSMap version 32 2024-04-26T18:01:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.717+0000 7f1947f08b80 1 -- 172.21.15.31:0/3119928298 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560d8eaa9bc0 con 0x560d8ea89440 2024-04-26T18:01:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.718+0000 7f191aff5700 1 -- 172.21.15.31:0/3119928298 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1930007da0 con 0x560d8ea89440 2024-04-26T18:01:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.718+0000 7f191aff5700 1 -- 172.21.15.31:0/3119928298 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f1928024030 con 0x560d8ea89440 2024-04-26T18:01:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.718+0000 7f1947f08b80 10 client.15843 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:01:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.718+0000 7f1947f08b80 10 client.15843 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T18:01:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.718+0000 7f1947f08b80 20 client.15843 populate_metadata read hostname 'smithi031' 2024-04-26T18:01:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.718+0000 7f1947f08b80 1 --2- 172.21.15.31:0/3119928298 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560d8eaac6e0 0x560d8eaaeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.718+0000 7f1947f08b80 4 client.15843 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:01:18.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.718+0000 7f1947f08b80 1 -- 172.21.15.31:0/3119928298 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d8e9e92b0 con 0x560d8eaac6e0 2024-04-26T18:01:18.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.718+0000 7f19367fc700 1 --2- 172.21.15.31:0/3119928298 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560d8eaac6e0 0x560d8eaaeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:18.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.719+0000 7f19367fc700 1 --2- 172.21.15.31:0/3119928298 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560d8eaac6e0 0x560d8eaaeb20 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eca1fd414b6f3ca6 server_cookie=e97f31de154cc5d1 in_seq=0 out_seq=0 2024-04-26T18:01:18.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.719+0000 7f191aff5700 10 client.15843 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:18.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.720+0000 7f191aff5700 1 -- 172.21.15.31:0/3119928298 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560d8e9e92b0 con 0x560d8eaac6e0 2024-04-26T18:01:18.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.720+0000 7f191aff5700 10 client.15843 handle_command_reply: tid=0 2024-04-26T18:01:18.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.720+0000 7f1947f08b80 1 client.15843 shutdown 2024-04-26T18:01:18.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.720+0000 7f1947f08b80 1 -- 172.21.15.31:0/3119928298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 msgr2=0x560d8ea93410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:18.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.720+0000 7f1947f08b80 1 --2- 172.21.15.31:0/3119928298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 0x560d8ea93410 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f192801d4b0 tx=0x7f19280047c0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.720+0000 7f1947f08b80 1 -- 172.21.15.31:0/3119928298 shutdown_connections 2024-04-26T18:01:18.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.720+0000 7f1947f08b80 1 --2- 172.21.15.31:0/3119928298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8ea90c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.720+0000 7f1947f08b80 1 --2- 172.21.15.31:0/3119928298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 0x560d8ea93410 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.720+0000 7f1947f08b80 1 --2- 172.21.15.31:0/3119928298 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560d8eaac6e0 0x560d8eaaeb20 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.720+0000 7f1947f08b80 1 --2- 172.21.15.31:0/3119928298 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea90710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.720+0000 7f1947f08b80 1 -- 172.21.15.31:0/3119928298 >> 172.21.15.31:0/3119928298 conn(0x560d8e9b5ed0 msgr2=0x560d8ea844e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:18.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.720+0000 7f191aff5700 0 client.15843 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:18.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.720+0000 7f1947f08b80 1 -- 172.21.15.31:0/3119928298 shutdown_connections 2024-04-26T18:01:18.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.721+0000 7f1947f08b80 1 -- 172.21.15.31:0/3119928298 wait complete. 2024-04-26T18:01:18.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.721+0000 7f1947f08b80 20 client.15843 trim_cache size 0 max 16384 2024-04-26T18:01:18.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.751+0000 7f1947f08b80 1 Processor -- start 2024-04-26T18:01:18.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.751+0000 7f1947f08b80 1 -- start start 2024-04-26T18:01:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f1947f08b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89440 0x560d8eaa8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f1947f08b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89d60 0x560d8ea92f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f1947f08b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8eab43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f1947f08b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f19381adf80 con 0x560d8ea89d60 2024-04-26T18:01:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f1947f08b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f19381ab7b0 con 0x560d8ea89440 2024-04-26T18:01:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f1947f08b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f193810e1a0 con 0x560d8ea93c80 2024-04-26T18:01:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f19367fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89d60 0x560d8ea92f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f19367fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89d60 0x560d8ea92f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36076/0 (socket says 172.21.15.31:36076) 2024-04-26T18:01:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f19377fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8eab43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f19377fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8eab43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:40910/0 (socket says 172.21.15.31:40910) 2024-04-26T18:01:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f19367fc700 1 -- 172.21.15.31:0/552369586 learned_addr learned my addr 172.21.15.31:0/552369586 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:18.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f1936ffd700 1 --2- 172.21.15.31:0/552369586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89440 0x560d8eaa8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:18.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f19377fe700 1 -- 172.21.15.31:0/552369586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89440 msgr2=0x560d8eaa8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:18.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f19377fe700 1 --2- 172.21.15.31:0/552369586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89440 0x560d8eaa8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f19377fe700 1 -- 172.21.15.31:0/552369586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89d60 msgr2=0x560d8ea92f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:18.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f19377fe700 1 --2- 172.21.15.31:0/552369586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89d60 0x560d8ea92f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.752+0000 7f19377fe700 1 -- 172.21.15.31:0/552369586 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d8ea89a70 con 0x560d8ea93c80 2024-04-26T18:01:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.753+0000 7f19377fe700 1 --2- 172.21.15.31:0/552369586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8eab43f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f192c0ce830 tx=0x7f192c09d810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10b3f51d6a7345d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.753+0000 7f191cff9700 1 -- 172.21.15.31:0/552369586 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c0a5cf0 con 0x560d8ea93c80 2024-04-26T18:01:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.753+0000 7f191cff9700 1 -- 172.21.15.31:0/552369586 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f192c09b410 con 0x560d8ea93c80 2024-04-26T18:01:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.753+0000 7f191cff9700 1 -- 172.21.15.31:0/552369586 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c09d370 con 0x560d8ea93c80 2024-04-26T18:01:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.754+0000 7f1947f08b80 1 -- 172.21.15.31:0/552369586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 msgr2=0x560d8eab43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.754+0000 7f1947f08b80 1 --2- 172.21.15.31:0/552369586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8eab43f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f192c0ce830 tx=0x7f192c09d810 comp rx=0 tx=0).stop 2024-04-26T18:01:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.754+0000 7f1947f08b80 1 -- 172.21.15.31:0/552369586 shutdown_connections 2024-04-26T18:01:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.754+0000 7f1947f08b80 1 --2- 172.21.15.31:0/552369586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89440 0x560d8eaa8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.754+0000 7f1947f08b80 1 --2- 172.21.15.31:0/552369586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8eab43f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.754+0000 7f1947f08b80 1 --2- 172.21.15.31:0/552369586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89d60 0x560d8ea92f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.754+0000 7f1947f08b80 1 -- 172.21.15.31:0/552369586 >> 172.21.15.31:0/552369586 conn(0x560d8e9b5ed0 msgr2=0x560d8ea8bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.755+0000 7f1947f08b80 1 -- 172.21.15.31:0/552369586 shutdown_connections 2024-04-26T18:01:18.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.755+0000 7f1947f08b80 1 -- 172.21.15.31:0/552369586 wait complete. 2024-04-26T18:01:18.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.755+0000 7f1947f08b80 1 -- start start 2024-04-26T18:01:18.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.755+0000 7f1947f08b80 1 Processor -- start 2024-04-26T18:01:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.756+0000 7f1947f08b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 0x560d8ea8d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.756+0000 7f1947f08b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea8d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.756+0000 7f1947f08b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8ea8de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.756+0000 7f1947f08b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f19381a9c90 con 0x560d8ea89440 2024-04-26T18:01:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.756+0000 7f1947f08b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f19381ab0e0 con 0x560d8ea89d60 2024-04-26T18:01:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.756+0000 7f1947f08b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f19381aa340 con 0x560d8ea93c80 2024-04-26T18:01:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.756+0000 7f19367fc700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea8d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.756+0000 7f19367fc700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea8d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:45894/0 (socket says 172.21.15.31:45894) 2024-04-26T18:01:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.756+0000 7f19367fc700 1 -- 172.21.15.31:0/1002091427 learned_addr learned my addr 172.21.15.31:0/1002091427 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.756+0000 7f19367fc700 1 -- 172.21.15.31:0/1002091427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 msgr2=0x560d8ea8de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.756+0000 7f19367fc700 1 --2- 172.21.15.31:0/1002091427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8ea8de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.756+0000 7f19367fc700 1 -- 172.21.15.31:0/1002091427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 msgr2=0x560d8ea8d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.756+0000 7f19367fc700 1 --2- 172.21.15.31:0/1002091427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 0x560d8ea8d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.756+0000 7f19367fc700 1 -- 172.21.15.31:0/1002091427 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d8eb11c60 con 0x560d8ea89d60 2024-04-26T18:01:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.757+0000 7f191aff5700 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T18:01:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.757+0000 7f19367fc700 1 --2- 172.21.15.31:0/1002091427 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea8d8e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1930009330 tx=0x7f1930005950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bdf06c00eed87ec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.757+0000 7f1947f08b80 10 client.0 _ll_register_callbacks cb 0x560d8e96f710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:01:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.757+0000 7f1947f08b80 1 -- 172.21.15.31:0/1002091427 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f192c09b410 con 0x560d8ea89d60 2024-04-26T18:01:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.757+0000 7f191aff5700 1 -- 172.21.15.31:0/1002091427 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1930003a00 con 0x560d8ea89d60 2024-04-26T18:01:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.757+0000 7f191aff5700 1 -- 172.21.15.31:0/1002091427 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f1930024070 con 0x560d8ea89d60 2024-04-26T18:01:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.758+0000 7f191aff5700 1 -- 172.21.15.31:0/1002091427 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1930032050 con 0x560d8ea89d60 2024-04-26T18:01:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.758+0000 7f1947f08b80 10 client.25532 fetch_fsmap learned FSMap version 32 2024-04-26T18:01:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.758+0000 7f1947f08b80 1 -- 172.21.15.31:0/1002091427 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({fsmap=32}) -- 0x560d8eb12d90 con 0x560d8ea89d60 2024-04-26T18:01:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.758+0000 7f191aff5700 1 -- 172.21.15.31:0/1002091427 <== mon.2 v2:172.21.15.97:3300/0 4 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f1930003ba0 con 0x560d8ea89d60 2024-04-26T18:01:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.758+0000 7f1947f08b80 10 client.25532 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:01:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.758+0000 7f1947f08b80 10 client.25532 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T18:01:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.759+0000 7f1947f08b80 20 client.25532 populate_metadata read hostname 'smithi031' 2024-04-26T18:01:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.759+0000 7f1947f08b80 1 --2- 172.21.15.31:0/1002091427 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560d8eb13100 0x560d8eb15580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.759+0000 7f1947f08b80 4 client.25532 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.5, "await_for": 42.0}] 2024-04-26T18:01:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.759+0000 7f1947f08b80 1 -- 172.21.15.31:0/1002091427 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.5, "await_for": 42.0}) -- 0x560d8e9e3350 con 0x560d8eb13100 2024-04-26T18:01:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.759+0000 7f1936ffd700 1 --2- 172.21.15.31:0/1002091427 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560d8eb13100 0x560d8eb15580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:18.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.760+0000 7f1936ffd700 1 --2- 172.21.15.31:0/1002091427 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560d8eb13100 0x560d8eb15580 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7672e94b8a9ad411 server_cookie=cc651864d0b81e58 in_seq=0 out_seq=0 2024-04-26T18:01:18.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.760+0000 7f191aff5700 10 client.25532 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.935+0000 7f191aff5700 1 -- 172.21.15.31:0/1002091427 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x560d8e9e3350 con 0x560d8eb13100 2024-04-26T18:01:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.935+0000 7f191aff5700 10 client.25532 handle_command_reply: tid=0 2024-04-26T18:01:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.935+0000 7f1947f08b80 1 client.25532 shutdown 2024-04-26T18:01:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.935+0000 7f1947f08b80 1 -- 172.21.15.31:0/1002091427 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 msgr2=0x560d8ea8d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.935+0000 7f1947f08b80 1 --2- 172.21.15.31:0/1002091427 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea8d8e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1930009330 tx=0x7f1930005950 comp rx=0 tx=0).stop 2024-04-26T18:01:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.935+0000 7f1947f08b80 1 -- 172.21.15.31:0/1002091427 shutdown_connections 2024-04-26T18:01:18.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.935+0000 7f1947f08b80 1 --2- 172.21.15.31:0/1002091427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d8ea93c80 0x560d8ea8de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.935+0000 7f1947f08b80 1 --2- 172.21.15.31:0/1002091427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d8ea89440 0x560d8ea8d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.935+0000 7f1947f08b80 1 --2- 172.21.15.31:0/1002091427 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560d8eb13100 0x560d8eb15580 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.935+0000 7f1947f08b80 1 --2- 172.21.15.31:0/1002091427 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560d8ea89d60 0x560d8ea8d8e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.935+0000 7f191aff5700 0 client.25532 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:18.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.935+0000 7f1947f08b80 1 -- 172.21.15.31:0/1002091427 >> 172.21.15.31:0/1002091427 conn(0x560d8e9b5ed0 msgr2=0x560d8ea88390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:18.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.935+0000 7f1947f08b80 1 -- 172.21.15.31:0/1002091427 shutdown_connections 2024-04-26T18:01:18.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.935+0000 7f1947f08b80 1 -- 172.21.15.31:0/1002091427 wait complete. 2024-04-26T18:01:18.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.936+0000 7f1947f08b80 20 client.25532 trim_cache size 0 max 16384 2024-04-26T18:01:18.938 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T18:01:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 32, 2024-04-26T18:01:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "leader": 24473, 2024-04-26T18:01:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "set_version": 14, 2024-04-26T18:01:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "sets": { 2024-04-26T18:01:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "38609351": { 2024-04-26T18:01:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "version": 14, 2024-04-26T18:01:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "age_ref": 0.0, 2024-04-26T18:01:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T18:01:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "QUIESCED", 2024-04-26T18:01:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T18:01:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:01:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "timeout": 60.0, 2024-04-26T18:01:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "expiration": 79.5, 2024-04-26T18:01:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "members": { 2024-04-26T18:01:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "file:/": { 2024-04-26T18:01:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "excluded": false, 2024-04-26T18:01:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T18:01:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "QUIESCED", 2024-04-26T18:01:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T18:01:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:18.940 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T18:01:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.936+0000 7f1937fff700 1 -- 172.21.15.31:0/281907343 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f19200a3f50 msgr2=0x7f19200a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.936+0000 7f1937fff700 1 --2- 172.21.15.31:0/281907343 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f19200a3f50 0x7f19200a6410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1938109750 tx=0x7f1928009260 comp rx=0 tx=0).stop 2024-04-26T18:01:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.936+0000 7f1937fff700 1 -- 172.21.15.31:0/281907343 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19381099d0 msgr2=0x7f193819ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.936+0000 7f1937fff700 1 --2- 172.21.15.31:0/281907343 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19381099d0 0x7f193819ba50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f192c005b40 tx=0x7f192c0056a0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.936+0000 7f1937fff700 1 -- 172.21.15.31:0/281907343 shutdown_connections 2024-04-26T18:01:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.936+0000 7f1937fff700 1 --2- 172.21.15.31:0/281907343 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f19200a3f50 0x7f19200a6410 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.936+0000 7f1937fff700 1 --2- 172.21.15.31:0/281907343 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19381099d0 0x7f193819ba50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.936+0000 7f1937fff700 1 --2- 172.21.15.31:0/281907343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1938104440 0x7f19381a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.936+0000 7f1937fff700 1 --2- 172.21.15.31:0/281907343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1938104df0 0x7f19381a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.936+0000 7f1937fff700 1 -- 172.21.15.31:0/281907343 >> 172.21.15.31:0/281907343 conn(0x7f19380fdc70 msgr2=0x7f19380ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.936+0000 7f1937fff700 1 -- 172.21.15.31:0/281907343 shutdown_connections 2024-04-26T18:01:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:18.936+0000 7f1937fff700 1 -- 172.21.15.31:0/281907343 wait complete. 2024-04-26T18:01:19.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:18 smithi186 ceph-mon[32797]: pgmap v147: 97 pgs: 97 active+clean; 2.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 29 MiB/s wr, 1.19k op/s 2024-04-26T18:01:19.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:18 smithi031 ceph-mon[29164]: pgmap v147: 97 pgs: 97 active+clean; 2.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 29 MiB/s wr, 1.19k op/s 2024-04-26T18:01:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:18 smithi097 ceph-mon[32419]: pgmap v147: 97 pgs: 97 active+clean; 2.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 29 MiB/s wr, 1.19k op/s 2024-04-26T18:01:20.110 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '38609351', quiesce took 8.1 seconds. Will release after: 19.5 2024-04-26T18:01:21.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:20 smithi186 ceph-mon[32797]: pgmap v148: 97 pgs: 97 active+clean; 2.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 29 MiB/s wr, 1.19k op/s 2024-04-26T18:01:21.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:20 smithi031 ceph-mon[29164]: pgmap v148: 97 pgs: 97 active+clean; 2.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 29 MiB/s wr, 1.19k op/s 2024-04-26T18:01:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:20 smithi097 ceph-mon[32419]: pgmap v148: 97 pgs: 97 active+clean; 2.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 29 MiB/s wr, 1.19k op/s 2024-04-26T18:01:21.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.973+0000 7f53d2c09640 1 -- 172.21.15.31:0/1755541154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc107860 msgr2=0x7f53cc107c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:21.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.973+0000 7f53d2c09640 1 --2- 172.21.15.31:0/1755541154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc107860 0x7f53cc107c40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f53c0009930 tx=0x7f53c002a420 comp rx=0 tx=0).stop 2024-04-26T18:01:21.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.973+0000 7f53d2c09640 1 -- 172.21.15.31:0/1755541154 shutdown_connections 2024-04-26T18:01:21.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.973+0000 7f53d2c09640 1 --2- 172.21.15.31:0/1755541154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53cc102210 0x7f53cc10e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.973+0000 7f53d2c09640 1 --2- 172.21.15.31:0/1755541154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53cc101850 0x7f53cc101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.973+0000 7f53d2c09640 1 --2- 172.21.15.31:0/1755541154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc107860 0x7f53cc107c40 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.973+0000 7f53d2c09640 1 -- 172.21.15.31:0/1755541154 >> 172.21.15.31:0/1755541154 conn(0x7f53cc077bb0 msgr2=0x7f53cc077fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.973+0000 7f53d2c09640 1 -- 172.21.15.31:0/1755541154 shutdown_connections 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.973+0000 7f53d2c09640 1 -- 172.21.15.31:0/1755541154 wait complete. 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.974+0000 7f53d2c09640 1 Processor -- start 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.974+0000 7f53d2c09640 1 -- start start 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.974+0000 7f53d2c09640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53cc101850 0x7f53cc199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.974+0000 7f53d2c09640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53cc102210 0x7f53cc199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.974+0000 7f53d2c09640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc107860 0x7f53cc19d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.974+0000 7f53d2c09640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc111ae0 con 0x7f53cc107860 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.974+0000 7f53d2c09640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc111960 con 0x7f53cc101850 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.974+0000 7f53d2c09640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc111c60 con 0x7f53cc102210 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.974+0000 7f53d11a6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc107860 0x7f53cc19d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:21.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.974+0000 7f53d11a6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc107860 0x7f53cc19d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36098/0 (socket says 172.21.15.31:36098) 2024-04-26T18:01:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.974+0000 7f53d11a6640 1 -- 172.21.15.31:0/1109784837 learned_addr learned my addr 172.21.15.31:0/1109784837 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.974+0000 7f53cbfff640 1 --2- 172.21.15.31:0/1109784837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53cc102210 0x7f53cc199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.975+0000 7f53d09a5640 1 --2- 172.21.15.31:0/1109784837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53cc101850 0x7f53cc199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.975+0000 7f53d11a6640 1 -- 172.21.15.31:0/1109784837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53cc102210 msgr2=0x7f53cc199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.975+0000 7f53d11a6640 1 --2- 172.21.15.31:0/1109784837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53cc102210 0x7f53cc199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.975+0000 7f53d11a6640 1 -- 172.21.15.31:0/1109784837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53cc101850 msgr2=0x7f53cc199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.975+0000 7f53d11a6640 1 --2- 172.21.15.31:0/1109784837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53cc101850 0x7f53cc199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.975+0000 7f53d11a6640 1 -- 172.21.15.31:0/1109784837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53cc19e0b0 con 0x7f53cc107860 2024-04-26T18:01:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.975+0000 7f53d11a6640 1 --2- 172.21.15.31:0/1109784837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc107860 0x7f53cc19d980 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f53c400eb10 tx=0x7f53c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.975+0000 7f53c9ffb640 1 -- 172.21.15.31:0/1109784837 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4004b20 con 0x7f53cc107860 2024-04-26T18:01:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.975+0000 7f53c9ffb640 1 -- 172.21.15.31:0/1109784837 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53c4004cc0 con 0x7f53cc107860 2024-04-26T18:01:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.975+0000 7f53d2c09640 1 -- 172.21.15.31:0/1109784837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53cc19e3a0 con 0x7f53cc107860 2024-04-26T18:01:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.975+0000 7f53c9ffb640 1 -- 172.21.15.31:0/1109784837 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4010670 con 0x7f53cc107860 2024-04-26T18:01:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.975+0000 7f53d2c09640 1 -- 172.21.15.31:0/1109784837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53cc1a5c30 con 0x7f53cc107860 2024-04-26T18:01:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.977+0000 7f53c9ffb640 1 -- 172.21.15.31:0/1109784837 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f53c4005020 con 0x7f53cc107860 2024-04-26T18:01:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53d5a03740 1 Processor -- start 2024-04-26T18:01:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53d5a03740 1 -- start start 2024-04-26T18:01:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53c9ffb640 1 --2- 172.21.15.31:0/1109784837 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f53bc0a3e40 0x7f53bc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53d5a03740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312410e00 0x55e3124111e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53d5a03740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 0x55e31241ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53cbfff640 1 --2- 172.21.15.31:0/1109784837 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f53bc0a3e40 0x7f53bc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53c9ffb640 1 -- 172.21.15.31:0/1109784837 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f53c40d0d10 con 0x7f53cc107860 2024-04-26T18:01:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53d5a03740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3124204f0 0x55e3124228e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53d5a03740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e312377460 con 0x55e312410e00 2024-04-26T18:01:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53d5a03740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55e312331590 con 0x55e3124204f0 2024-04-26T18:01:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53d5a03740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e312373420 con 0x55e312411720 2024-04-26T18:01:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53d09a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312410e00 0x55e3124111e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53d09a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312410e00 0x55e3124111e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36110/0 (socket says 172.21.15.31:36110) 2024-04-26T18:01:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53d09a5640 1 -- 172.21.15.31:0/2788186437 learned_addr learned my addr 172.21.15.31:0/2788186437 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.978+0000 7f53d11a6640 1 --2- 172.21.15.31:0/2788186437 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3124204f0 0x55e3124228e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.979+0000 7f53cbfff640 1 --2- 172.21.15.31:0/2788186437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 0x55e31241ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.979+0000 7f53d09a5640 1 -- 172.21.15.31:0/2788186437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 msgr2=0x55e31241ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.979+0000 7f53d09a5640 1 --2- 172.21.15.31:0/2788186437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 0x55e31241ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.979+0000 7f53d09a5640 1 -- 172.21.15.31:0/2788186437 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3124204f0 msgr2=0x55e3124228e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.979+0000 7f53d09a5640 1 --2- 172.21.15.31:0/2788186437 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3124204f0 0x55e3124228e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.979+0000 7f53d09a5640 1 -- 172.21.15.31:0/2788186437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e31233e0d0 con 0x55e312410e00 2024-04-26T18:01:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.979+0000 7f53cbfff640 1 --2- 172.21.15.31:0/1109784837 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f53bc0a3e40 0x7f53bc0a6300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f53cc078360 tx=0x7f53b80107a0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.979+0000 7f53d09a5640 1 --2- 172.21.15.31:0/2788186437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312410e00 0x55e3124111e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f53c0031440 tx=0x7f53c0032ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c67118ed3ecff7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.979+0000 7f53b6ffd640 1 -- 172.21.15.31:0/2788186437 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c003f070 con 0x55e312410e00 2024-04-26T18:01:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.979+0000 7f53b6ffd640 1 -- 172.21.15.31:0/2788186437 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53c0004030 con 0x55e312410e00 2024-04-26T18:01:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.979+0000 7f53b6ffd640 1 -- 172.21.15.31:0/2788186437 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c0035070 con 0x55e312410e00 2024-04-26T18:01:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.980+0000 7f53d5a03740 1 -- 172.21.15.31:0/2788186437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312410e00 msgr2=0x55e3124111e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.980+0000 7f53d5a03740 1 --2- 172.21.15.31:0/2788186437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312410e00 0x55e3124111e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f53c0031440 tx=0x7f53c0032ba0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.980+0000 7f53d5a03740 1 -- 172.21.15.31:0/2788186437 shutdown_connections 2024-04-26T18:01:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.980+0000 7f53d5a03740 1 --2- 172.21.15.31:0/2788186437 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3124204f0 0x55e3124228e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.980+0000 7f53d5a03740 1 --2- 172.21.15.31:0/2788186437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 0x55e31241ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.980+0000 7f53d5a03740 1 --2- 172.21.15.31:0/2788186437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312410e00 0x55e3124111e0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.980+0000 7f53d5a03740 1 -- 172.21.15.31:0/2788186437 >> 172.21.15.31:0/2788186437 conn(0x55e31240d410 msgr2=0x55e31240d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.980+0000 7f53d5a03740 1 -- 172.21.15.31:0/2788186437 shutdown_connections 2024-04-26T18:01:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.980+0000 7f53d5a03740 1 -- 172.21.15.31:0/2788186437 wait complete. 2024-04-26T18:01:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.980+0000 7f53d5a03740 1 -- start start 2024-04-26T18:01:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.980+0000 7f53d5a03740 1 Processor -- start 2024-04-26T18:01:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.980+0000 7f53d5a03740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312410e00 0x55e31241d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d5a03740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 0x55e31241d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d5a03740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3124204f0 0x55e31241de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d5a03740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e3124230c0 con 0x55e312410e00 2024-04-26T18:01:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d5a03740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55e3122feb60 con 0x55e3124204f0 2024-04-26T18:01:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d5a03740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e312423240 con 0x55e312411720 2024-04-26T18:01:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d5a03740 10 client.0 _ll_register_callbacks cb 0x55e3123232b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:01:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d09a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312410e00 0x55e31241d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d09a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312410e00 0x55e31241d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36120/0 (socket says 172.21.15.31:36120) 2024-04-26T18:01:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d09a5640 1 -- 172.21.15.31:0/1723121021 learned_addr learned my addr 172.21.15.31:0/1723121021 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53cbfff640 1 --2- 172.21.15.31:0/1723121021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 0x55e31241d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d11a6640 1 --2- 172.21.15.31:0/1723121021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3124204f0 0x55e31241de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d09a5640 1 -- 172.21.15.31:0/1723121021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 msgr2=0x55e31241d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d09a5640 1 --2- 172.21.15.31:0/1723121021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 0x55e31241d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d09a5640 1 -- 172.21.15.31:0/1723121021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3124204f0 msgr2=0x55e31241de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d09a5640 1 --2- 172.21.15.31:0/1723121021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3124204f0 0x55e31241de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d09a5640 1 -- 172.21.15.31:0/1723121021 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e312089210 con 0x55e312410e00 2024-04-26T18:01:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d09a5640 1 --2- 172.21.15.31:0/1723121021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312410e00 0x55e31241d380 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f53c0006010 tx=0x7f53c0031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191828401d94ff8a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53abfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:01:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53abfff640 1 -- 172.21.15.31:0/1723121021 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c0035070 con 0x55e312410e00 2024-04-26T18:01:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53abfff640 1 -- 172.21.15.31:0/1723121021 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f53c00055d0 con 0x55e312410e00 2024-04-26T18:01:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d5a03740 1 -- 172.21.15.31:0/1723121021 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53c0004030 con 0x55e312410e00 2024-04-26T18:01:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53abfff640 1 -- 172.21.15.31:0/1723121021 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f53c0055050 con 0x55e312410e00 2024-04-26T18:01:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.981+0000 7f53d5a03740 10 client.15858 fetch_fsmap learned FSMap version 32 2024-04-26T18:01:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.982+0000 7f53d5a03740 1 -- 172.21.15.31:0/1723121021 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e312155620 con 0x55e312410e00 2024-04-26T18:01:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.982+0000 7f53abfff640 1 -- 172.21.15.31:0/1723121021 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53c0004030 con 0x55e312410e00 2024-04-26T18:01:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.982+0000 7f53abfff640 1 -- 172.21.15.31:0/1723121021 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f53c0003680 con 0x55e312410e00 2024-04-26T18:01:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.982+0000 7f53d5a03740 10 client.15858 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:01:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.982+0000 7f53d5a03740 10 client.15858 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:01:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.982+0000 7f53d5a03740 20 client.15858 populate_metadata read hostname 'smithi031' 2024-04-26T18:01:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.982+0000 7f53d5a03740 1 --2- 172.21.15.31:0/1723121021 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e312430420 0x55e312432810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.982+0000 7f53d5a03740 4 client.15858 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:01:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.982+0000 7f53d5a03740 1 -- 172.21.15.31:0/1723121021 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e312385070 con 0x55e312430420 2024-04-26T18:01:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.982+0000 7f53cbfff640 1 --2- 172.21.15.31:0/1723121021 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e312430420 0x55e312432810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.983+0000 7f53cbfff640 1 --2- 172.21.15.31:0/1723121021 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e312430420 0x55e312432810 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1038d6b0464ab2d0 server_cookie=d0f94481f61fba6c in_seq=0 out_seq=0 2024-04-26T18:01:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.983+0000 7f53abfff640 10 client.15858 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.983+0000 7f53abfff640 1 -- 172.21.15.31:0/1723121021 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e312385070 con 0x55e312430420 2024-04-26T18:01:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.983+0000 7f53abfff640 10 client.15858 handle_command_reply: tid=0 2024-04-26T18:01:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.983+0000 7f53d5a03740 1 client.15858 shutdown 2024-04-26T18:01:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.984+0000 7f53d5a03740 1 -- 172.21.15.31:0/1723121021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312410e00 msgr2=0x55e31241d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.984+0000 7f53d5a03740 1 --2- 172.21.15.31:0/1723121021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312410e00 0x55e31241d380 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f53c0006010 tx=0x7f53c0031750 comp rx=0 tx=0).stop 2024-04-26T18:01:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.984+0000 7f53d5a03740 1 -- 172.21.15.31:0/1723121021 shutdown_connections 2024-04-26T18:01:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.984+0000 7f53d5a03740 1 --2- 172.21.15.31:0/1723121021 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e312430420 0x55e312432810 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.984+0000 7f53d5a03740 1 --2- 172.21.15.31:0/1723121021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3124204f0 0x55e31241de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.984+0000 7f53abfff640 0 client.15858 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.984+0000 7f53d5a03740 1 --2- 172.21.15.31:0/1723121021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 0x55e31241d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.984+0000 7f53d5a03740 1 --2- 172.21.15.31:0/1723121021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312410e00 0x55e31241d380 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.984+0000 7f53d5a03740 1 -- 172.21.15.31:0/1723121021 >> 172.21.15.31:0/1723121021 conn(0x55e31240d410 msgr2=0x55e312413280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.984+0000 7f53d5a03740 1 -- 172.21.15.31:0/1723121021 shutdown_connections 2024-04-26T18:01:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.984+0000 7f53d5a03740 1 -- 172.21.15.31:0/1723121021 wait complete. 2024-04-26T18:01:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.984+0000 7f53d5a03740 20 client.15858 trim_cache size 0 max 16384 2024-04-26T18:01:22.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53d5a03740 1 Processor -- start 2024-04-26T18:01:22.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53d5a03740 1 -- start start 2024-04-26T18:01:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53d5a03740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312410e00 0x55e31242bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53d5a03740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312411720 0x55e31241c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53d5a03740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3124204f0 0x55e312437e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53d5a03740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc1aea60 con 0x55e312411720 2024-04-26T18:01:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53d5a03740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc1ac2d0 con 0x55e3124204f0 2024-04-26T18:01:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53d5a03740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc1aa140 con 0x55e312410e00 2024-04-26T18:01:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53cbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312411720 0x55e31241c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53cbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312411720 0x55e31241c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36124/0 (socket says 172.21.15.31:36124) 2024-04-26T18:01:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53cbfff640 1 -- 172.21.15.31:0/1974954350 learned_addr learned my addr 172.21.15.31:0/1974954350 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53d09a5640 1 --2- 172.21.15.31:0/1974954350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312410e00 0x55e31242bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53cbfff640 1 -- 172.21.15.31:0/1974954350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312410e00 msgr2=0x55e31242bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53cbfff640 1 --2- 172.21.15.31:0/1974954350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312410e00 0x55e31242bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53cbfff640 1 -- 172.21.15.31:0/1974954350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3124204f0 msgr2=0x55e312437e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53cbfff640 1 --2- 172.21.15.31:0/1974954350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3124204f0 0x55e312437e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53cbfff640 1 -- 172.21.15.31:0/1974954350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e312432b80 con 0x55e312411720 2024-04-26T18:01:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:21.999+0000 7f53cbfff640 1 --2- 172.21.15.31:0/1974954350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312411720 0x55e31241c4e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f53b800c010 tx=0x7f53b8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aedc1a132d05cee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.000+0000 7f53b67fc640 1 -- 172.21.15.31:0/1974954350 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b8028bc0 con 0x55e312411720 2024-04-26T18:01:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.000+0000 7f53b67fc640 1 -- 172.21.15.31:0/1974954350 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53b8028d60 con 0x55e312411720 2024-04-26T18:01:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.000+0000 7f53b67fc640 1 -- 172.21.15.31:0/1974954350 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b8020b10 con 0x55e312411720 2024-04-26T18:01:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.000+0000 7f53d5a03740 1 -- 172.21.15.31:0/1974954350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312411720 msgr2=0x55e31241c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.000+0000 7f53d5a03740 1 --2- 172.21.15.31:0/1974954350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312411720 0x55e31241c4e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f53b800c010 tx=0x7f53b8015000 comp rx=0 tx=0).stop 2024-04-26T18:01:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.000+0000 7f53d5a03740 1 -- 172.21.15.31:0/1974954350 shutdown_connections 2024-04-26T18:01:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.000+0000 7f53d5a03740 1 --2- 172.21.15.31:0/1974954350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3124204f0 0x55e312437e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.000+0000 7f53d5a03740 1 --2- 172.21.15.31:0/1974954350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e312411720 0x55e31241c4e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.000+0000 7f53d5a03740 1 --2- 172.21.15.31:0/1974954350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312410e00 0x55e31242bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.000+0000 7f53d5a03740 1 -- 172.21.15.31:0/1974954350 >> 172.21.15.31:0/1974954350 conn(0x55e31240d410 msgr2=0x55e31240d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.000+0000 7f53d5a03740 1 -- 172.21.15.31:0/1974954350 shutdown_connections 2024-04-26T18:01:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.000+0000 7f53d5a03740 1 -- 172.21.15.31:0/1974954350 wait complete. 2024-04-26T18:01:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d5a03740 1 -- start start 2024-04-26T18:01:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d5a03740 1 Processor -- start 2024-04-26T18:01:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d5a03740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e312410e00 0x55e312428670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d5a03740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 0x55e312423a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d5a03740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3124204f0 0x55e312423f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d5a03740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc1aa790 con 0x55e3124204f0 2024-04-26T18:01:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d5a03740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc1abc00 con 0x55e312410e00 2024-04-26T18:01:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d5a03740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc1aae60 con 0x55e312411720 2024-04-26T18:01:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d5a03740 10 client.0 _ll_register_callbacks cb 0x7f53c0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:01:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d11a6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3124204f0 0x55e312423f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d11a6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3124204f0 0x55e312423f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36134/0 (socket says 172.21.15.31:36134) 2024-04-26T18:01:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d11a6640 1 -- 172.21.15.31:0/3046421363 learned_addr learned my addr 172.21.15.31:0/3046421363 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53cbfff640 1 --2- 172.21.15.31:0/3046421363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 0x55e312423a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d09a5640 1 --2- 172.21.15.31:0/3046421363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e312410e00 0x55e312428670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d11a6640 1 -- 172.21.15.31:0/3046421363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 msgr2=0x55e312423a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d11a6640 1 --2- 172.21.15.31:0/3046421363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 0x55e312423a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d11a6640 1 -- 172.21.15.31:0/3046421363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e312410e00 msgr2=0x55e312428670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d11a6640 1 --2- 172.21.15.31:0/3046421363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e312410e00 0x55e312428670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53d11a6640 1 -- 172.21.15.31:0/3046421363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3124245b0 con 0x55e3124204f0 2024-04-26T18:01:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.001+0000 7f53cbfff640 1 --2- 172.21.15.31:0/3046421363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 0x55e312423a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53d11a6640 1 --2- 172.21.15.31:0/3046421363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3124204f0 0x55e312423f70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f53c409cc30 tx=0x7f53c409c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a3db783cac6f1b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53abfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:01:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53abfff640 1 -- 172.21.15.31:0/3046421363 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c40bbe60 con 0x55e3124204f0 2024-04-26T18:01:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53abfff640 1 -- 172.21.15.31:0/3046421363 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f53c40bd0c0 con 0x55e3124204f0 2024-04-26T18:01:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53d5a03740 1 -- 172.21.15.31:0/3046421363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53b8028d60 con 0x55e3124204f0 2024-04-26T18:01:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53abfff640 1 -- 172.21.15.31:0/3046421363 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f53c409cca0 con 0x55e3124204f0 2024-04-26T18:01:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53d5a03740 10 client.15864 fetch_fsmap learned FSMap version 32 2024-04-26T18:01:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53d5a03740 1 -- 172.21.15.31:0/3046421363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e312496550 con 0x55e3124204f0 2024-04-26T18:01:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53abfff640 1 -- 172.21.15.31:0/3046421363 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53b8028d60 con 0x55e3124204f0 2024-04-26T18:01:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53abfff640 1 -- 172.21.15.31:0/3046421363 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f53c409cf00 con 0x55e3124204f0 2024-04-26T18:01:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53d5a03740 10 client.15864 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:01:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53d5a03740 10 client.15864 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:01:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53d5a03740 20 client.15864 populate_metadata read hostname 'smithi031' 2024-04-26T18:01:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53d5a03740 1 --2- 172.21.15.31:0/3046421363 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e312496cb0 0x55e3124990d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53d5a03740 4 client.15864 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:01:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53d5a03740 1 -- 172.21.15.31:0/3046421363 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e31237eee0 con 0x55e312496cb0 2024-04-26T18:01:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.002+0000 7f53d09a5640 1 --2- 172.21.15.31:0/3046421363 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e312496cb0 0x55e3124990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.003+0000 7f53abfff640 10 client.15864 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.003+0000 7f53d09a5640 1 --2- 172.21.15.31:0/3046421363 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e312496cb0 0x55e3124990d0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79f2ebb30ed68013 server_cookie=17bd9c65e3d7c17d in_seq=0 out_seq=0 2024-04-26T18:01:22.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53abfff640 1 -- 172.21.15.31:0/3046421363 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e31237eee0 con 0x55e312496cb0 2024-04-26T18:01:22.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53abfff640 10 client.15864 handle_command_reply: tid=0 2024-04-26T18:01:22.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53d5a03740 1 client.15864 shutdown 2024-04-26T18:01:22.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53d5a03740 1 -- 172.21.15.31:0/3046421363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3124204f0 msgr2=0x55e312423f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:22.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53d5a03740 1 --2- 172.21.15.31:0/3046421363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3124204f0 0x55e312423f70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f53c409cc30 tx=0x7f53c409c7e0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53d5a03740 1 -- 172.21.15.31:0/3046421363 shutdown_connections 2024-04-26T18:01:22.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53d5a03740 1 --2- 172.21.15.31:0/3046421363 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e312496cb0 0x55e3124990d0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53d5a03740 1 --2- 172.21.15.31:0/3046421363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3124204f0 0x55e312423f70 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53d5a03740 1 --2- 172.21.15.31:0/3046421363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e312411720 0x55e312423a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53abfff640 0 client.15864 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:22.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53d5a03740 1 --2- 172.21.15.31:0/3046421363 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e312410e00 0x55e312428670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53d5a03740 1 -- 172.21.15.31:0/3046421363 >> 172.21.15.31:0/3046421363 conn(0x55e31240d410 msgr2=0x55e31241c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:22.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53d5a03740 1 -- 172.21.15.31:0/3046421363 shutdown_connections 2024-04-26T18:01:22.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53d5a03740 1 -- 172.21.15.31:0/3046421363 wait complete. 2024-04-26T18:01:22.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.020+0000 7f53d5a03740 20 client.15864 trim_cache size 0 max 16384 2024-04-26T18:01:22.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:01:22.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:01:22.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:01:22.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:01:22.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.021+0000 7f53d2c09640 1 -- 172.21.15.31:0/1109784837 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f53bc0a3e40 msgr2=0x7f53bc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:22.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.021+0000 7f53d2c09640 1 --2- 172.21.15.31:0/1109784837 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f53bc0a3e40 0x7f53bc0a6300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f53cc078360 tx=0x7f53b80107a0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.021+0000 7f53d2c09640 1 -- 172.21.15.31:0/1109784837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc107860 msgr2=0x7f53cc19d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:22.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.021+0000 7f53d2c09640 1 --2- 172.21.15.31:0/1109784837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc107860 0x7f53cc19d980 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f53c400eb10 tx=0x7f53c400c2d0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.021+0000 7f53d2c09640 1 -- 172.21.15.31:0/1109784837 shutdown_connections 2024-04-26T18:01:22.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.021+0000 7f53d2c09640 1 --2- 172.21.15.31:0/1109784837 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f53bc0a3e40 0x7f53bc0a6300 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.021+0000 7f53d2c09640 1 --2- 172.21.15.31:0/1109784837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc107860 0x7f53cc19d980 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.021+0000 7f53d2c09640 1 --2- 172.21.15.31:0/1109784837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53cc102210 0x7f53cc199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.021+0000 7f53d2c09640 1 --2- 172.21.15.31:0/1109784837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f53cc101850 0x7f53cc199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:22.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.021+0000 7f53d2c09640 1 -- 172.21.15.31:0/1109784837 >> 172.21.15.31:0/1109784837 conn(0x7f53cc077bb0 msgr2=0x7f53cc06b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:22.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.021+0000 7f53d2c09640 1 -- 172.21.15.31:0/1109784837 shutdown_connections 2024-04-26T18:01:22.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:22.021+0000 7f53d2c09640 1 -- 172.21.15.31:0/1109784837 wait complete. 2024-04-26T18:01:23.098 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 240.0 s, 349.3 tps, lat 41.327 ms stddev 55.929 2024-04-26T18:01:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:22 smithi186 ceph-mon[32797]: pgmap v149: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 32 MiB/s wr, 1.46k op/s 2024-04-26T18:01:23.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:22 smithi031 ceph-mon[29164]: pgmap v149: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 32 MiB/s wr, 1.46k op/s 2024-04-26T18:01:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:22 smithi097 ceph-mon[32419]: pgmap v149: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 32 MiB/s wr, 1.46k op/s 2024-04-26T18:01:25.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:24 smithi186 ceph-mon[32797]: pgmap v150: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 16 MiB/s wr, 908 op/s 2024-04-26T18:01:25.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:24 smithi031 ceph-mon[29164]: pgmap v150: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 16 MiB/s wr, 908 op/s 2024-04-26T18:01:25.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:24 smithi097 ceph-mon[32419]: pgmap v150: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 16 MiB/s wr, 908 op/s 2024-04-26T18:01:27.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:26 smithi186 ceph-mon[32797]: pgmap v151: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 16 MiB/s wr, 914 op/s 2024-04-26T18:01:27.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:26 smithi031 ceph-mon[29164]: pgmap v151: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 16 MiB/s wr, 914 op/s 2024-04-26T18:01:27.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:26 smithi097 ceph-mon[32419]: pgmap v151: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 16 MiB/s wr, 914 op/s 2024-04-26T18:01:28.099 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 245.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:01:29.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:28 smithi186 ceph-mon[32797]: pgmap v152: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.2 MiB/s wr, 588 op/s 2024-04-26T18:01:29.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:28 smithi031 ceph-mon[29164]: pgmap v152: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.2 MiB/s wr, 588 op/s 2024-04-26T18:01:29.387 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:28 smithi097 ceph-mon[32419]: pgmap v152: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.2 MiB/s wr, 588 op/s 2024-04-26T18:01:31.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:31 smithi097 ceph-mon[32419]: pgmap v153: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.8 MiB/s wr, 280 op/s 2024-04-26T18:01:31.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:31 smithi186 ceph-mon[32797]: pgmap v153: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.8 MiB/s wr, 280 op/s 2024-04-26T18:01:31.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:31 smithi031 ceph-mon[29164]: pgmap v153: 97 pgs: 97 active+clean; 2.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.8 MiB/s wr, 280 op/s 2024-04-26T18:01:33.099 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 250.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:01:33.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:33 smithi097 ceph-mon[32419]: pgmap v154: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 1.24k op/s 2024-04-26T18:01:33.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:33 smithi186 ceph-mon[32797]: pgmap v154: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 1.24k op/s 2024-04-26T18:01:33.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:33 smithi031 ceph-mon[29164]: pgmap v154: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 1.24k op/s 2024-04-26T18:01:35.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.121+0000 7f6a0b571640 1 -- 172.21.15.31:0/2335582117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a04103300 msgr2=0x7f6a0410f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:35.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.121+0000 7f6a0b571640 1 --2- 172.21.15.31:0/2335582117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a04103300 0x7f6a0410f810 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f69fc009930 tx=0x7f69fc02a420 comp rx=0 tx=0).stop 2024-04-26T18:01:35.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.121+0000 7f6a0b571640 1 -- 172.21.15.31:0/2335582117 shutdown_connections 2024-04-26T18:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.121+0000 7f6a0b571640 1 --2- 172.21.15.31:0/2335582117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a04103300 0x7f6a0410f810 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.121+0000 7f6a0b571640 1 --2- 172.21.15.31:0/2335582117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a04102940 0x7f6a04102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.121+0000 7f6a0b571640 1 --2- 172.21.15.31:0/2335582117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a04108950 0x7f6a04108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.121+0000 7f6a0b571640 1 -- 172.21.15.31:0/2335582117 >> 172.21.15.31:0/2335582117 conn(0x7f6a04079dc0 msgr2=0x7f6a0407a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.121+0000 7f6a0b571640 1 -- 172.21.15.31:0/2335582117 shutdown_connections 2024-04-26T18:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.121+0000 7f6a0b571640 1 -- 172.21.15.31:0/2335582117 wait complete. 2024-04-26T18:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.121+0000 7f6a0b571640 1 Processor -- start 2024-04-26T18:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a0b571640 1 -- start start 2024-04-26T18:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a0b571640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a04102940 0x7f6a0419b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a0b571640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a04103300 0x7f6a0419b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a0b571640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a04108950 0x7f6a0419fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a0b571640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6a04112c10 con 0x7f6a04108950 2024-04-26T18:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a0b571640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6a04112a90 con 0x7f6a04103300 2024-04-26T18:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a0b571640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6a04112d90 con 0x7f6a04102940 2024-04-26T18:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a09b0e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a04108950 0x7f6a0419fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a09b0e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a04108950 0x7f6a0419fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40898/0 (socket says 172.21.15.31:40898) 2024-04-26T18:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a09b0e640 1 -- 172.21.15.31:0/3687410350 learned_addr learned my addr 172.21.15.31:0/3687410350 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a0930d640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a04102940 0x7f6a0419b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a04103300 0x7f6a0419b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a09b0e640 1 -- 172.21.15.31:0/3687410350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a04102940 msgr2=0x7f6a0419b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a09b0e640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a04102940 0x7f6a0419b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a09b0e640 1 -- 172.21.15.31:0/3687410350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a04103300 msgr2=0x7f6a0419b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a09b0e640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a04103300 0x7f6a0419b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a09b0e640 1 -- 172.21.15.31:0/3687410350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a041a0210 con 0x7f6a04108950 2024-04-26T18:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a0930d640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a04102940 0x7f6a0419b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.122+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a04103300 0x7f6a0419b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:35.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.123+0000 7f6a09b0e640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a04108950 0x7f6a0419fae0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f69fc005940 tx=0x7f69fc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:35.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.123+0000 7f69fa7fc640 1 -- 172.21.15.31:0/3687410350 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69fc03f030 con 0x7f6a04108950 2024-04-26T18:01:35.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.123+0000 7f69fa7fc640 1 -- 172.21.15.31:0/3687410350 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69fc0044c0 con 0x7f6a04108950 2024-04-26T18:01:35.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.123+0000 7f6a0b571640 1 -- 172.21.15.31:0/3687410350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a041a04a0 con 0x7f6a04108950 2024-04-26T18:01:35.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.123+0000 7f69fa7fc640 1 -- 172.21.15.31:0/3687410350 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69fc035070 con 0x7f6a04108950 2024-04-26T18:01:35.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.123+0000 7f6a0b571640 1 -- 172.21.15.31:0/3687410350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a041a7d90 con 0x7f6a04108950 2024-04-26T18:01:35.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.125+0000 7f69fa7fc640 1 -- 172.21.15.31:0/3687410350 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f69fc042030 con 0x7f6a04108950 2024-04-26T18:01:35.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.126+0000 7f6a0e365740 1 Processor -- start 2024-04-26T18:01:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.126+0000 7f6a0e365740 1 -- start start 2024-04-26T18:01:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.126+0000 7f6a0e365740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149a3f00 0x5594149a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.126+0000 7f6a0e365740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a4820 0x5594149aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.126+0000 7f6a0e365740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149af0e0 0x5594149b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.126+0000 7f6a0e365740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55941490a460 con 0x5594149af0e0 2024-04-26T18:01:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.126+0000 7f6a0e365740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5594148c4590 con 0x5594149a4820 2024-04-26T18:01:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.126+0000 7f6a0e365740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559414906420 con 0x5594149a3f00 2024-04-26T18:01:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.126+0000 7f69fa7fc640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f69f00a3dc0 0x7f69f00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.126+0000 7f6a08b0c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149af0e0 0x5594149b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.126+0000 7f6a09b0e640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f69f00a3dc0 0x7f69f00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.126+0000 7f6a08b0c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149af0e0 0x5594149b14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40902/0 (socket says 172.21.15.31:40902) 2024-04-26T18:01:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.126+0000 7f6a08b0c640 1 -- 172.21.15.31:0/347649126 learned_addr learned my addr 172.21.15.31:0/347649126 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.127+0000 7f69fa7fc640 1 -- 172.21.15.31:0/3687410350 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f69fc0f0c90 con 0x7f6a04108950 2024-04-26T18:01:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.127+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/347649126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149a3f00 0x5594149a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.127+0000 7f6a0930d640 1 --2- 172.21.15.31:0/347649126 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a4820 0x5594149aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.127+0000 7f6a08b0c640 1 -- 172.21.15.31:0/347649126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149a3f00 msgr2=0x5594149a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.127+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/347649126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149a3f00 0x5594149a42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.127+0000 7f6a08b0c640 1 -- 172.21.15.31:0/347649126 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a4820 msgr2=0x5594149aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.127+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/347649126 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a4820 0x5594149aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.127+0000 7f6a08b0c640 1 -- 172.21.15.31:0/347649126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594148d10d0 con 0x5594149af0e0 2024-04-26T18:01:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.127+0000 7f6a09b0e640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f69f00a3dc0 0x7f69f00a6280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f69fc0bc920 tx=0x7f69fc0876c0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.127+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/347649126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149af0e0 0x5594149b14d0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f69f40149f0 tx=0x7f69f400b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c269d8465c5baec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.127+0000 7f69ef7fe640 1 -- 172.21.15.31:0/347649126 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f40142d0 con 0x5594149af0e0 2024-04-26T18:01:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.127+0000 7f69ef7fe640 1 -- 172.21.15.31:0/347649126 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69f4005020 con 0x5594149af0e0 2024-04-26T18:01:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.127+0000 7f69ef7fe640 1 -- 172.21.15.31:0/347649126 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f4004a00 con 0x5594149af0e0 2024-04-26T18:01:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.128+0000 7f6a0e365740 1 -- 172.21.15.31:0/347649126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149af0e0 msgr2=0x5594149b14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.128+0000 7f6a0e365740 1 --2- 172.21.15.31:0/347649126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149af0e0 0x5594149b14d0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f69f40149f0 tx=0x7f69f400b540 comp rx=0 tx=0).stop 2024-04-26T18:01:35.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.128+0000 7f6a0e365740 1 -- 172.21.15.31:0/347649126 shutdown_connections 2024-04-26T18:01:35.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.128+0000 7f6a0e365740 1 --2- 172.21.15.31:0/347649126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149af0e0 0x5594149b14d0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.128+0000 7f6a0e365740 1 --2- 172.21.15.31:0/347649126 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a4820 0x5594149aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.128+0000 7f6a0e365740 1 --2- 172.21.15.31:0/347649126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149a3f00 0x5594149a42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.128+0000 7f6a0e365740 1 -- 172.21.15.31:0/347649126 >> 172.21.15.31:0/347649126 conn(0x5594149a0410 msgr2=0x5594149a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:35.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.128+0000 7f6a0e365740 1 -- 172.21.15.31:0/347649126 shutdown_connections 2024-04-26T18:01:35.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.128+0000 7f6a0e365740 1 -- 172.21.15.31:0/347649126 wait complete. 2024-04-26T18:01:35.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.128+0000 7f6a0e365740 1 -- start start 2024-04-26T18:01:35.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a0e365740 1 Processor -- start 2024-04-26T18:01:35.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a0e365740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149a3f00 0x5594149ac0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a0e365740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a4820 0x5594149ac610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a0e365740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149af0e0 0x5594149acb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a0e365740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5594149b1d10 con 0x5594149af0e0 2024-04-26T18:01:35.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a0e365740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x559414891b60 con 0x5594149a4820 2024-04-26T18:01:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a0e365740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5594149b1e90 con 0x5594149a3f00 2024-04-26T18:01:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a0e365740 10 client.0 _ll_register_callbacks cb 0x5594148b62b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:01:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a08b0c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149af0e0 0x5594149acb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a08b0c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149af0e0 0x5594149acb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40916/0 (socket says 172.21.15.31:40916) 2024-04-26T18:01:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a08b0c640 1 -- 172.21.15.31:0/3377593309 learned_addr learned my addr 172.21.15.31:0/3377593309 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a0930d640 1 --2- 172.21.15.31:0/3377593309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a4820 0x5594149ac610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/3377593309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149a3f00 0x5594149ac0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a08b0c640 1 -- 172.21.15.31:0/3377593309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149a3f00 msgr2=0x5594149ac0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/3377593309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149a3f00 0x5594149ac0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a08b0c640 1 -- 172.21.15.31:0/3377593309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a4820 msgr2=0x5594149ac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.129+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/3377593309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a4820 0x5594149ac610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.130+0000 7f6a08b0c640 1 -- 172.21.15.31:0/3377593309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55941461c210 con 0x5594149af0e0 2024-04-26T18:01:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.130+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/3377593309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149af0e0 0x5594149acb50 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f69f40057c0 tx=0x7f69f4014f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b95c0a4fb17189 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.130+0000 7f69ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:01:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.130+0000 7f69ecff9640 1 -- 172.21.15.31:0/3377593309 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f400f970 con 0x5594149af0e0 2024-04-26T18:01:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.130+0000 7f69ecff9640 1 -- 172.21.15.31:0/3377593309 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f69f400fb10 con 0x5594149af0e0 2024-04-26T18:01:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.130+0000 7f6a0e365740 1 -- 172.21.15.31:0/3377593309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69f4005020 con 0x5594149af0e0 2024-04-26T18:01:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.130+0000 7f69ecff9640 1 -- 172.21.15.31:0/3377593309 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f69f400fd10 con 0x5594149af0e0 2024-04-26T18:01:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.130+0000 7f6a0e365740 10 client.15876 fetch_fsmap learned FSMap version 32 2024-04-26T18:01:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.130+0000 7f6a0e365740 1 -- 172.21.15.31:0/3377593309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5594146e8620 con 0x5594149af0e0 2024-04-26T18:01:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.130+0000 7f69ecff9640 1 -- 172.21.15.31:0/3377593309 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69f4005020 con 0x5594149af0e0 2024-04-26T18:01:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.130+0000 7f69ecff9640 1 -- 172.21.15.31:0/3377593309 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f69f4012570 con 0x5594149af0e0 2024-04-26T18:01:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.130+0000 7f6a0e365740 10 client.15876 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:01:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.130+0000 7f6a0e365740 10 client.15876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:01:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.131+0000 7f6a0e365740 20 client.15876 populate_metadata read hostname 'smithi031' 2024-04-26T18:01:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.131+0000 7f6a0e365740 1 --2- 172.21.15.31:0/3377593309 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5594149bf110 0x5594149c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.131+0000 7f6a0e365740 4 client.15876 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:01:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.131+0000 7f6a0e365740 1 -- 172.21.15.31:0/3377593309 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559414918070 con 0x5594149bf110 2024-04-26T18:01:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.131+0000 7f6a0930d640 1 --2- 172.21.15.31:0/3377593309 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5594149bf110 0x5594149c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.131+0000 7f69ecff9640 10 client.15876 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.131+0000 7f6a0930d640 1 --2- 172.21.15.31:0/3377593309 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5594149bf110 0x5594149c1500 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f822a95876b10dbe server_cookie=8f5b253e1469dc83 in_seq=0 out_seq=0 2024-04-26T18:01:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.132+0000 7f69ecff9640 1 -- 172.21.15.31:0/3377593309 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559414918070 con 0x5594149bf110 2024-04-26T18:01:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.132+0000 7f69ecff9640 10 client.15876 handle_command_reply: tid=0 2024-04-26T18:01:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.132+0000 7f6a0e365740 1 client.15876 shutdown 2024-04-26T18:01:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.132+0000 7f6a0e365740 1 -- 172.21.15.31:0/3377593309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149af0e0 msgr2=0x5594149acb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.132+0000 7f6a0e365740 1 --2- 172.21.15.31:0/3377593309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149af0e0 0x5594149acb50 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f69f40057c0 tx=0x7f69f4014f70 comp rx=0 tx=0).stop 2024-04-26T18:01:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.132+0000 7f6a0e365740 1 -- 172.21.15.31:0/3377593309 shutdown_connections 2024-04-26T18:01:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.132+0000 7f6a0e365740 1 --2- 172.21.15.31:0/3377593309 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5594149bf110 0x5594149c1500 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.132+0000 7f6a0e365740 1 --2- 172.21.15.31:0/3377593309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149af0e0 0x5594149acb50 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.132+0000 7f6a0e365740 1 --2- 172.21.15.31:0/3377593309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a4820 0x5594149ac610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.132+0000 7f6a0e365740 1 --2- 172.21.15.31:0/3377593309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149a3f00 0x5594149ac0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.132+0000 7f69ecff9640 0 client.15876 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.132+0000 7f6a0e365740 1 -- 172.21.15.31:0/3377593309 >> 172.21.15.31:0/3377593309 conn(0x5594149a0410 msgr2=0x55941499f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.132+0000 7f6a0e365740 1 -- 172.21.15.31:0/3377593309 shutdown_connections 2024-04-26T18:01:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.132+0000 7f6a0e365740 1 -- 172.21.15.31:0/3377593309 wait complete. 2024-04-26T18:01:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.133+0000 7f6a0e365740 20 client.15876 trim_cache size 0 max 16384 2024-04-26T18:01:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0e365740 1 Processor -- start 2024-04-26T18:01:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0e365740 1 -- start start 2024-04-26T18:01:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0e365740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a3f00 0x5594149babd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0e365740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149a4820 0x5594149a5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0e365740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149af0e0 0x5594149c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0e365740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6a041b2eb0 con 0x5594149a4820 2024-04-26T18:01:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0e365740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6a041b0720 con 0x5594149a3f00 2024-04-26T18:01:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0e365740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6a041ae570 con 0x5594149af0e0 2024-04-26T18:01:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0930d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149a4820 0x5594149a5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0930d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149a4820 0x5594149a5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40918/0 (socket says 172.21.15.31:40918) 2024-04-26T18:01:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0930d640 1 -- 172.21.15.31:0/480859083 learned_addr learned my addr 172.21.15.31:0/480859083 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0930d640 1 -- 172.21.15.31:0/480859083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149af0e0 msgr2=0x5594149c6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0930d640 1 --2- 172.21.15.31:0/480859083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149af0e0 0x5594149c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/480859083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a3f00 0x5594149babd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0930d640 1 -- 172.21.15.31:0/480859083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a3f00 msgr2=0x5594149babd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0930d640 1 --2- 172.21.15.31:0/480859083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a3f00 0x5594149babd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0930d640 1 -- 172.21.15.31:0/480859083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594149b1e30 con 0x5594149a4820 2024-04-26T18:01:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/480859083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a3f00 0x5594149babd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:01:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.148+0000 7f6a0930d640 1 --2- 172.21.15.31:0/480859083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149a4820 0x5594149a5fc0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6a00009990 tx=0x7f6a00009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d73c34f409f228 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.149+0000 7f69eeffd640 1 -- 172.21.15.31:0/480859083 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a00010b60 con 0x5594149a4820 2024-04-26T18:01:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.149+0000 7f69eeffd640 1 -- 172.21.15.31:0/480859083 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a00010d00 con 0x5594149a4820 2024-04-26T18:01:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.149+0000 7f69eeffd640 1 -- 172.21.15.31:0/480859083 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0001d810 con 0x5594149a4820 2024-04-26T18:01:35.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.149+0000 7f6a0e365740 1 -- 172.21.15.31:0/480859083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149a4820 msgr2=0x5594149a5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:35.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.149+0000 7f6a0e365740 1 --2- 172.21.15.31:0/480859083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149a4820 0x5594149a5fc0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6a00009990 tx=0x7f6a00009500 comp rx=0 tx=0).stop 2024-04-26T18:01:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.149+0000 7f6a0e365740 1 -- 172.21.15.31:0/480859083 shutdown_connections 2024-04-26T18:01:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.149+0000 7f6a0e365740 1 --2- 172.21.15.31:0/480859083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149af0e0 0x5594149c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.149+0000 7f6a0e365740 1 --2- 172.21.15.31:0/480859083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149a4820 0x5594149a5fc0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.149+0000 7f6a0e365740 1 --2- 172.21.15.31:0/480859083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a3f00 0x5594149babd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.149+0000 7f6a0e365740 1 -- 172.21.15.31:0/480859083 >> 172.21.15.31:0/480859083 conn(0x5594149a0410 msgr2=0x5594149a58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.149+0000 7f6a0e365740 1 -- 172.21.15.31:0/480859083 shutdown_connections 2024-04-26T18:01:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.149+0000 7f6a0e365740 1 -- 172.21.15.31:0/480859083 wait complete. 2024-04-26T18:01:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0e365740 1 -- start start 2024-04-26T18:01:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0e365740 1 Processor -- start 2024-04-26T18:01:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0e365740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a3f00 0x5594149a9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0e365740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149a4820 0x5594149a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0e365740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149af0e0 0x5594149a9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0e365740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6a041aebe0 con 0x5594149a4820 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0e365740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6a041b0050 con 0x5594149a3f00 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0e365740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6a041af2b0 con 0x5594149af0e0 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0e365740 10 client.0 _ll_register_callbacks cb 0x7f6a000099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0930d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149a4820 0x5594149a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0930d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149a4820 0x5594149a95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40934/0 (socket says 172.21.15.31:40934) 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0930d640 1 -- 172.21.15.31:0/2209684836 learned_addr learned my addr 172.21.15.31:0/2209684836 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/2209684836 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a3f00 0x5594149a9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/2209684836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149af0e0 0x5594149a9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0930d640 1 -- 172.21.15.31:0/2209684836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149af0e0 msgr2=0x5594149a9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0930d640 1 --2- 172.21.15.31:0/2209684836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149af0e0 0x5594149a9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0930d640 1 -- 172.21.15.31:0/2209684836 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a3f00 msgr2=0x5594149a9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.150+0000 7f6a0930d640 1 --2- 172.21.15.31:0/2209684836 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a3f00 0x5594149a9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.151+0000 7f6a0930d640 1 -- 172.21.15.31:0/2209684836 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559414a24610 con 0x5594149a4820 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.151+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/2209684836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149af0e0 0x5594149a9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:01:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.151+0000 7f6a0930d640 1 --2- 172.21.15.31:0/2209684836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149a4820 0x5594149a95d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6a00000f80 tx=0x7f6a00011850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=499efa1221ecd151 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.151+0000 7f69ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:01:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.151+0000 7f69ecff9640 1 -- 172.21.15.31:0/2209684836 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a00020160 con 0x5594149a4820 2024-04-26T18:01:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.151+0000 7f69ecff9640 1 -- 172.21.15.31:0/2209684836 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f6a00020300 con 0x5594149a4820 2024-04-26T18:01:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.151+0000 7f6a0e365740 1 -- 172.21.15.31:0/2209684836 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a00010d00 con 0x5594149a4820 2024-04-26T18:01:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.151+0000 7f69ecff9640 1 -- 172.21.15.31:0/2209684836 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6a00020500 con 0x5594149a4820 2024-04-26T18:01:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.151+0000 7f6a0e365740 10 client.15882 fetch_fsmap learned FSMap version 32 2024-04-26T18:01:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.151+0000 7f6a0e365740 1 -- 172.21.15.31:0/2209684836 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559414a25890 con 0x5594149a4820 2024-04-26T18:01:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.151+0000 7f69ecff9640 1 -- 172.21.15.31:0/2209684836 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a00010d00 con 0x5594149a4820 2024-04-26T18:01:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.151+0000 7f69ecff9640 1 -- 172.21.15.31:0/2209684836 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f6a00030070 con 0x5594149a4820 2024-04-26T18:01:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.152+0000 7f6a0e365740 10 client.15882 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:01:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.152+0000 7f6a0e365740 10 client.15882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:01:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.152+0000 7f6a0e365740 20 client.15882 populate_metadata read hostname 'smithi031' 2024-04-26T18:01:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.152+0000 7f6a0e365740 1 --2- 172.21.15.31:0/2209684836 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x559414a25fe0 0x559414a28400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.152+0000 7f6a0e365740 4 client.15882 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:01:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.152+0000 7f6a0e365740 1 -- 172.21.15.31:0/2209684836 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559414911ee0 con 0x559414a25fe0 2024-04-26T18:01:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.152+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/2209684836 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x559414a25fe0 0x559414a28400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.152+0000 7f6a08b0c640 1 --2- 172.21.15.31:0/2209684836 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x559414a25fe0 0x559414a28400 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fdb70a5601498da server_cookie=df6f285577c8dee4 in_seq=0 out_seq=0 2024-04-26T18:01:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.152+0000 7f69ecff9640 10 client.15882 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:35.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f69ecff9640 1 -- 172.21.15.31:0/2209684836 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559414911ee0 con 0x559414a25fe0 2024-04-26T18:01:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f69ecff9640 10 client.15882 handle_command_reply: tid=0 2024-04-26T18:01:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f6a0e365740 1 client.15882 shutdown 2024-04-26T18:01:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f6a0e365740 1 -- 172.21.15.31:0/2209684836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149a4820 msgr2=0x5594149a95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f6a0e365740 1 --2- 172.21.15.31:0/2209684836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149a4820 0x5594149a95d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6a00000f80 tx=0x7f6a00011850 comp rx=0 tx=0).stop 2024-04-26T18:01:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f6a0e365740 1 -- 172.21.15.31:0/2209684836 shutdown_connections 2024-04-26T18:01:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f6a0e365740 1 --2- 172.21.15.31:0/2209684836 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x559414a25fe0 0x559414a28400 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f6a0e365740 1 --2- 172.21.15.31:0/2209684836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5594149af0e0 0x5594149a9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f69ecff9640 0 client.15882 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f6a0e365740 1 --2- 172.21.15.31:0/2209684836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594149a4820 0x5594149a95d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f6a0e365740 1 --2- 172.21.15.31:0/2209684836 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5594149a3f00 0x5594149a9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f6a0e365740 1 -- 172.21.15.31:0/2209684836 >> 172.21.15.31:0/2209684836 conn(0x5594149a0410 msgr2=0x5594149ac920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f6a0e365740 1 -- 172.21.15.31:0/2209684836 shutdown_connections 2024-04-26T18:01:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f6a0e365740 1 -- 172.21.15.31:0/2209684836 wait complete. 2024-04-26T18:01:35.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.166+0000 7f6a0e365740 20 client.15882 trim_cache size 0 max 16384 2024-04-26T18:01:35.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:01:35.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:01:35.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:01:35.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:01:35.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.167+0000 7f6a0b571640 1 -- 172.21.15.31:0/3687410350 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f69f00a3dc0 msgr2=0x7f69f00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:35.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.167+0000 7f6a0b571640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f69f00a3dc0 0x7f69f00a6280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f69fc0bc920 tx=0x7f69fc0876c0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.167+0000 7f6a0b571640 1 -- 172.21.15.31:0/3687410350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a04108950 msgr2=0x7f6a0419fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:35.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.167+0000 7f6a0b571640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a04108950 0x7f6a0419fae0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f69fc005940 tx=0x7f69fc0040c0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.167+0000 7f6a0b571640 1 -- 172.21.15.31:0/3687410350 shutdown_connections 2024-04-26T18:01:35.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.167+0000 7f6a0b571640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f69f00a3dc0 0x7f69f00a6280 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.167+0000 7f6a0b571640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a04108950 0x7f6a0419fae0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.167+0000 7f6a0b571640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a04103300 0x7f6a0419b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.167+0000 7f6a0b571640 1 --2- 172.21.15.31:0/3687410350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a04102940 0x7f6a0419b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:35.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.167+0000 7f6a0b571640 1 -- 172.21.15.31:0/3687410350 >> 172.21.15.31:0/3687410350 conn(0x7f6a04079dc0 msgr2=0x7f6a04100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:35.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.167+0000 7f6a0b571640 1 -- 172.21.15.31:0/3687410350 shutdown_connections 2024-04-26T18:01:35.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:35.167+0000 7f6a0b571640 1 -- 172.21.15.31:0/3687410350 wait complete. 2024-04-26T18:01:35.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:35 smithi097 ceph-mon[32419]: pgmap v155: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 MiB/s wr, 964 op/s 2024-04-26T18:01:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:35 smithi186 ceph-mon[32797]: pgmap v155: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 MiB/s wr, 964 op/s 2024-04-26T18:01:35.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:35 smithi031 ceph-mon[29164]: pgmap v155: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 MiB/s wr, 964 op/s 2024-04-26T18:01:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:37 smithi097 ceph-mon[32419]: pgmap v156: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 MiB/s wr, 967 op/s 2024-04-26T18:01:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:37 smithi186 ceph-mon[32797]: pgmap v156: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 MiB/s wr, 967 op/s 2024-04-26T18:01:37.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:37 smithi031 ceph-mon[29164]: pgmap v156: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 MiB/s wr, 967 op/s 2024-04-26T18:01:38.099 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 255.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:01:39.363 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:39 smithi031 ceph-mon[29164]: pgmap v157: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 963 op/s 2024-04-26T18:01:39.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:39 smithi097 ceph-mon[32419]: pgmap v157: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 963 op/s 2024-04-26T18:01:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:39 smithi186 ceph-mon[32797]: pgmap v157: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 963 op/s 2024-04-26T18:01:39.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T18:01:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:01:40.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:40 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:01:40.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:40 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:01:40.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:40 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:01:41.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:41 smithi031 ceph-mon[29164]: pgmap v158: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 MiB/s wr, 961 op/s 2024-04-26T18:01:41.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:41 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:01:41.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:41 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:01:41.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:41 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:01:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:41 smithi097 ceph-mon[32419]: pgmap v158: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 MiB/s wr, 961 op/s 2024-04-26T18:01:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:41 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:01:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:41 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:01:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:41 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:01:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:41 smithi186 ceph-mon[32797]: pgmap v158: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 MiB/s wr, 961 op/s 2024-04-26T18:01:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:41 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:01:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:41 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:01:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:41 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:01:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.354+0000 7fdee291c700 1 -- 172.21.15.31:0/2294733364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc1034a0 msgr2=0x7fdedc103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.354+0000 7fdee291c700 1 --2- 172.21.15.31:0/2294733364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc1034a0 0x7fdedc103920 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fdec4009990 tx=0x7fdec4009ca0 comp rx=0 tx=0).stop 2024-04-26T18:01:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.354+0000 7fdee291c700 1 -- 172.21.15.31:0/2294733364 shutdown_connections 2024-04-26T18:01:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.354+0000 7fdee291c700 1 --2- 172.21.15.31:0/2294733364 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdedc1094e0 0x7fdedc1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.354+0000 7fdee291c700 1 --2- 172.21.15.31:0/2294733364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdedc103e60 0x7fdedc1102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.354+0000 7fdee291c700 1 --2- 172.21.15.31:0/2294733364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc1034a0 0x7fdedc103920 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.354+0000 7fdee291c700 1 -- 172.21.15.31:0/2294733364 >> 172.21.15.31:0/2294733364 conn(0x7fdedc07a590 msgr2=0x7fdedc07a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.354+0000 7fdee291c700 1 -- 172.21.15.31:0/2294733364 shutdown_connections 2024-04-26T18:01:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.354+0000 7fdee291c700 1 -- 172.21.15.31:0/2294733364 wait complete. 2024-04-26T18:01:42.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.355+0000 7fdee291c700 1 Processor -- start 2024-04-26T18:01:42.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.355+0000 7fdee291c700 1 -- start start 2024-04-26T18:01:42.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.355+0000 7fdee291c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc1034a0 0x7fdedc075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.355+0000 7fdee291c700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdedc103e60 0x7fdedc075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.355+0000 7fdee291c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdedc1094e0 0x7fdedc0761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.355+0000 7fdee291c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc1135e0 con 0x7fdedc1034a0 2024-04-26T18:01:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.355+0000 7fdee291c700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc113460 con 0x7fdedc103e60 2024-04-26T18:01:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.355+0000 7fdee291c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc113760 con 0x7fdedc1094e0 2024-04-26T18:01:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.355+0000 7fdedbfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc1034a0 0x7fdedc075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.355+0000 7fdedbfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc1034a0 0x7fdedc075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40948/0 (socket says 172.21.15.31:40948) 2024-04-26T18:01:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.355+0000 7fdedbfff700 1 -- 172.21.15.31:0/1122774522 learned_addr learned my addr 172.21.15.31:0/1122774522 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fdedb7fe700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdedc103e60 0x7fdedc075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fdee0eb9700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdedc1094e0 0x7fdedc0761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fdedbfff700 1 -- 172.21.15.31:0/1122774522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdedc1094e0 msgr2=0x7fdedc0761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fdedbfff700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdedc1094e0 0x7fdedc0761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fdedbfff700 1 -- 172.21.15.31:0/1122774522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdedc103e60 msgr2=0x7fdedc075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fdedbfff700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdedc103e60 0x7fdedc075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fdedbfff700 1 -- 172.21.15.31:0/1122774522 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdedc1a7fa0 con 0x7fdedc1034a0 2024-04-26T18:01:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fdee0eb9700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdedc1094e0 0x7fdedc0761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:01:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fdedb7fe700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdedc103e60 0x7fdedc075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fdedbfff700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc1034a0 0x7fdedc075720 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdecc00c9b0 tx=0x7fdecc00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fded97fa700 1 -- 172.21.15.31:0/1122774522 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc007b00 con 0x7fdedc1034a0 2024-04-26T18:01:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fdee291c700 1 -- 172.21.15.31:0/1122774522 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdedc1a8290 con 0x7fdedc1034a0 2024-04-26T18:01:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fded97fa700 1 -- 172.21.15.31:0/1122774522 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdecc007ca0 con 0x7fdedc1034a0 2024-04-26T18:01:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fdee291c700 1 -- 172.21.15.31:0/1122774522 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdedc1a8820 con 0x7fdedc1034a0 2024-04-26T18:01:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.356+0000 7fded97fa700 1 -- 172.21.15.31:0/1122774522 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc018730 con 0x7fdedc1034a0 2024-04-26T18:01:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.357+0000 7fdee291c700 1 -- 172.21.15.31:0/1122774522 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdedc06a110 con 0x7fdedc1034a0 2024-04-26T18:01:42.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.358+0000 7fded97fa700 1 -- 172.21.15.31:0/1122774522 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdecc00f450 con 0x7fdedc1034a0 2024-04-26T18:01:42.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.359+0000 7fded97fa700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fdec80a3e30 0x7fdec80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:42.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.359+0000 7fdedb7fe700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fdec80a3e30 0x7fdec80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:42.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.359+0000 7fded97fa700 1 -- 172.21.15.31:0/1122774522 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fdecc0d0a70 con 0x7fdedc1034a0 2024-04-26T18:01:42.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.360+0000 7fdedb7fe700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fdec80a3e30 0x7fdec80a62f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdec4006010 tx=0x7fdec401a040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:42.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.363+0000 7fded97fa700 1 -- 172.21.15.31:0/1122774522 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdecc014030 con 0x7fdedc1034a0 2024-04-26T18:01:42.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.624+0000 7fdee291c700 1 -- 172.21.15.31:0/1122774522 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdedc076c50 con 0x7fdedc1034a0 2024-04-26T18:01:42.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.624+0000 7fded97fa700 1 -- 172.21.15.31:0/1122774522 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11877 (secure 0 0 0) 0x7fdecc09d210 con 0x7fdedc1034a0 2024-04-26T18:01:42.625 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:01:42.625 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T18:00:06.847155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24446,24457,24473]},"id":1}]} 2024-04-26T18:01:42.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.627+0000 7fdee291c700 1 -- 172.21.15.31:0/1122774522 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fdec80a3e30 msgr2=0x7fdec80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:42.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.627+0000 7fdee291c700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fdec80a3e30 0x7fdec80a62f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdec4006010 tx=0x7fdec401a040 comp rx=0 tx=0).stop 2024-04-26T18:01:42.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.627+0000 7fdee291c700 1 -- 172.21.15.31:0/1122774522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc1034a0 msgr2=0x7fdedc075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:42.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.627+0000 7fdee291c700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc1034a0 0x7fdedc075720 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdecc00c9b0 tx=0x7fdecc00cd70 comp rx=0 tx=0).stop 2024-04-26T18:01:42.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.627+0000 7fdee291c700 1 -- 172.21.15.31:0/1122774522 shutdown_connections 2024-04-26T18:01:42.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.627+0000 7fdee291c700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fdec80a3e30 0x7fdec80a62f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:42.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.627+0000 7fdee291c700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdedc1094e0 0x7fdedc0761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:42.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.627+0000 7fdee291c700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc1034a0 0x7fdedc075720 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:42.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.627+0000 7fdee291c700 1 --2- 172.21.15.31:0/1122774522 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdedc103e60 0x7fdedc075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:42.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.627+0000 7fdee291c700 1 -- 172.21.15.31:0/1122774522 >> 172.21.15.31:0/1122774522 conn(0x7fdedc07a590 msgr2=0x7fdedc102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:42.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.627+0000 7fdee291c700 1 -- 172.21.15.31:0/1122774522 shutdown_connections 2024-04-26T18:01:42.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:42.627+0000 7fdee291c700 1 -- 172.21.15.31:0/1122774522 wait complete. 2024-04-26T18:01:42.628 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 32 2024-04-26T18:01:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 260.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:01:43.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:43 smithi031 ceph-mon[29164]: pgmap v159: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 MiB/s wr, 964 op/s 2024-04-26T18:01:43.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:43 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1122774522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:01:43.354 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24473 quiesce db --set-id 38609351 --release --await' 2024-04-26T18:01:43.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24473 quiesce db --set-id 38609351 --release --await 2024-04-26T18:01:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:43 smithi097 ceph-mon[32419]: pgmap v159: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 MiB/s wr, 964 op/s 2024-04-26T18:01:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:43 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1122774522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:01:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:43 smithi186 ceph-mon[32797]: pgmap v159: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 MiB/s wr, 964 op/s 2024-04-26T18:01:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:43 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1122774522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:01:43.614 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:01:45.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.235+0000 7f7aa3ce2700 1 -- 172.21.15.31:0/2029298226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a9c103e60 msgr2=0x7f7a9c1102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:45.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.235+0000 7f7aa3ce2700 1 --2- 172.21.15.31:0/2029298226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a9c103e60 0x7f7a9c1102b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7a9400b0a0 tx=0x7f7a9400b3b0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.235+0000 7f7aa3ce2700 1 -- 172.21.15.31:0/2029298226 shutdown_connections 2024-04-26T18:01:45.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.235+0000 7f7aa3ce2700 1 --2- 172.21.15.31:0/2029298226 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a9c1094e0 0x7f7a9c1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.235+0000 7f7aa3ce2700 1 --2- 172.21.15.31:0/2029298226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a9c103e60 0x7f7a9c1102b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.235+0000 7f7aa3ce2700 1 --2- 172.21.15.31:0/2029298226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a9c1034a0 0x7f7a9c103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.235+0000 7f7aa3ce2700 1 -- 172.21.15.31:0/2029298226 >> 172.21.15.31:0/2029298226 conn(0x7f7a9c07a590 msgr2=0x7f7a9c07a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.235+0000 7f7aa3ce2700 1 -- 172.21.15.31:0/2029298226 shutdown_connections 2024-04-26T18:01:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.235+0000 7f7aa3ce2700 1 -- 172.21.15.31:0/2029298226 wait complete. 2024-04-26T18:01:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa3ce2700 1 Processor -- start 2024-04-26T18:01:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa3ce2700 1 -- start start 2024-04-26T18:01:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa3ce2700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a9c1034a0 0x7f7a9c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa3ce2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a9c103e60 0x7f7a9c19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa3ce2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a9c1094e0 0x7f7a9c199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa3ce2700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c113610 con 0x7f7a9c1094e0 2024-04-26T18:01:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa3ce2700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c113490 con 0x7f7a9c1034a0 2024-04-26T18:01:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa3ce2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c113790 con 0x7f7a9c103e60 2024-04-26T18:01:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa227f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a9c1094e0 0x7f7a9c199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa227f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a9c1094e0 0x7f7a9c199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54666/0 (socket says 172.21.15.31:54666) 2024-04-26T18:01:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa227f700 1 -- 172.21.15.31:0/1472961946 learned_addr learned my addr 172.21.15.31:0/1472961946 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa1a7e700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a9c1034a0 0x7f7a9c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa127d700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a9c103e60 0x7f7a9c19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa227f700 1 -- 172.21.15.31:0/1472961946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a9c103e60 msgr2=0x7f7a9c19fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa227f700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a9c103e60 0x7f7a9c19fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa227f700 1 -- 172.21.15.31:0/1472961946 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a9c1034a0 msgr2=0x7f7a9c19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa227f700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a9c1034a0 0x7f7a9c19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa227f700 1 -- 172.21.15.31:0/1472961946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a9c19a090 con 0x7f7a9c1094e0 2024-04-26T18:01:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa127d700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a9c103e60 0x7f7a9c19fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:01:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.236+0000 7f7aa1a7e700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a9c1034a0 0x7f7a9c19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.237+0000 7f7aa227f700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a9c1094e0 0x7f7a9c199930 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7a9400b830 tx=0x7f7a94004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:45.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.237+0000 7f7a92ffd700 1 -- 172.21.15.31:0/1472961946 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a94021070 con 0x7f7a9c1094e0 2024-04-26T18:01:45.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.237+0000 7f7a92ffd700 1 -- 172.21.15.31:0/1472961946 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a94003c00 con 0x7f7a9c1094e0 2024-04-26T18:01:45.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.237+0000 7f7aa3ce2700 1 -- 172.21.15.31:0/1472961946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a9c19a320 con 0x7f7a9c1094e0 2024-04-26T18:01:45.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.237+0000 7f7a92ffd700 1 -- 172.21.15.31:0/1472961946 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a94004810 con 0x7f7a9c1094e0 2024-04-26T18:01:45.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.237+0000 7f7aa3ce2700 1 -- 172.21.15.31:0/1472961946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a9c1a6640 con 0x7f7a9c1094e0 2024-04-26T18:01:45.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.239+0000 7f7a92ffd700 1 -- 172.21.15.31:0/1472961946 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7a94007a00 con 0x7f7a9c1094e0 2024-04-26T18:01:45.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.239+0000 7f7a92ffd700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7a880a3db0 0x7f7a880a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.240+0000 7f7aa1a7e700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7a880a3db0 0x7f7a880a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.240+0000 7f7a92ffd700 1 -- 172.21.15.31:0/1472961946 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7a940d1970 con 0x7f7a9c1094e0 2024-04-26T18:01:45.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:45 smithi031 ceph-mon[29164]: pgmap v160: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-26T18:01:45.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.240+0000 7f7aa1a7e700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7a880a3db0 0x7f7a880a6270 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7a98005e10 tx=0x7f7a98005d60 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.241+0000 7f7aae574b80 1 Processor -- start 2024-04-26T18:01:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.241+0000 7f7aae574b80 1 -- start start 2024-04-26T18:01:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.241+0000 7f7aae574b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5158ea0 0x555fc5159280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.241+0000 7f7aae574b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc51597c0 0x555fc5153250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.241+0000 7f7aae574b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5153870 0x555fc5153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.241+0000 7f7aae574b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555fc50a5930 con 0x555fc5153870 2024-04-26T18:01:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.241+0000 7f7aae574b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x555fc500ba30 con 0x555fc51597c0 2024-04-26T18:01:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.241+0000 7f7aae574b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555fc50a18f0 con 0x555fc5158ea0 2024-04-26T18:01:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.241+0000 7f7aa227f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5153870 0x555fc5153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.241+0000 7f7aa227f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5153870 0x555fc5153cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54672/0 (socket says 172.21.15.31:54672) 2024-04-26T18:01:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.241+0000 7f7aa227f700 1 -- 172.21.15.31:0/2039222710 learned_addr learned my addr 172.21.15.31:0/2039222710 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.242+0000 7f7aa1a7e700 1 --2- 172.21.15.31:0/2039222710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5158ea0 0x555fc5159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.242+0000 7f7aa127d700 1 --2- 172.21.15.31:0/2039222710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc51597c0 0x555fc5153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.242+0000 7f7aa227f700 1 -- 172.21.15.31:0/2039222710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5158ea0 msgr2=0x555fc5159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.242+0000 7f7aa227f700 1 --2- 172.21.15.31:0/2039222710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5158ea0 0x555fc5159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.242+0000 7f7aa227f700 1 -- 172.21.15.31:0/2039222710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc51597c0 msgr2=0x555fc5153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.242+0000 7f7aa227f700 1 --2- 172.21.15.31:0/2039222710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc51597c0 0x555fc5153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.242+0000 7f7aa227f700 1 -- 172.21.15.31:0/2039222710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fc4ff29d0 con 0x555fc5153870 2024-04-26T18:01:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.242+0000 7f7aa227f700 1 --2- 172.21.15.31:0/2039222710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5153870 0x555fc5153cd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7a94099e50 tx=0x7f7a94010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db802d640a5fc3c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.242+0000 7f7a87fff700 1 -- 172.21.15.31:0/2039222710 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9402a070 con 0x555fc5153870 2024-04-26T18:01:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.242+0000 7f7a87fff700 1 -- 172.21.15.31:0/2039222710 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a940a1da0 con 0x555fc5153870 2024-04-26T18:01:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.242+0000 7f7a87fff700 1 -- 172.21.15.31:0/2039222710 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9409a060 con 0x555fc5153870 2024-04-26T18:01:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.243+0000 7f7aae574b80 1 -- 172.21.15.31:0/2039222710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5153870 msgr2=0x555fc5153cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.243+0000 7f7aae574b80 1 --2- 172.21.15.31:0/2039222710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5153870 0x555fc5153cd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7a94099e50 tx=0x7f7a94010000 comp rx=0 tx=0).stop 2024-04-26T18:01:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.243+0000 7f7aae574b80 1 -- 172.21.15.31:0/2039222710 shutdown_connections 2024-04-26T18:01:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.243+0000 7f7aae574b80 1 --2- 172.21.15.31:0/2039222710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5153870 0x555fc5153cd0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.243+0000 7f7aae574b80 1 --2- 172.21.15.31:0/2039222710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5158ea0 0x555fc5159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.243+0000 7f7aae574b80 1 --2- 172.21.15.31:0/2039222710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc51597c0 0x555fc5153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.243+0000 7f7aae574b80 1 -- 172.21.15.31:0/2039222710 >> 172.21.15.31:0/2039222710 conn(0x555fc50816c0 msgr2=0x555fc514e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.243+0000 7f7aae574b80 1 -- 172.21.15.31:0/2039222710 shutdown_connections 2024-04-26T18:01:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.243+0000 7f7aae574b80 1 -- 172.21.15.31:0/2039222710 wait complete. 2024-04-26T18:01:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.243+0000 7f7aae574b80 1 -- start start 2024-04-26T18:01:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.244+0000 7f7aae574b80 1 Processor -- start 2024-04-26T18:01:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.244+0000 7f7aae574b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5153870 0x555fc515f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.244+0000 7f7aae574b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc5158ea0 0x555fc515c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.244+0000 7f7aae574b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc51597c0 0x555fc515cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.244+0000 7f7aae574b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555fc515fff0 con 0x555fc5153870 2024-04-26T18:01:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.244+0000 7f7aae574b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x555fc505fb80 con 0x555fc5158ea0 2024-04-26T18:01:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.244+0000 7f7aae574b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555fc5160170 con 0x555fc51597c0 2024-04-26T18:01:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.244+0000 7f7aae574b80 10 client.0 _ll_register_callbacks cb 0x555fc503da50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:01:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.244+0000 7f7aa1a7e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5153870 0x555fc515f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.244+0000 7f7aa1a7e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5153870 0x555fc515f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54684/0 (socket says 172.21.15.31:54684) 2024-04-26T18:01:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.244+0000 7f7aa1a7e700 1 -- 172.21.15.31:0/608482789 learned_addr learned my addr 172.21.15.31:0/608482789 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.244+0000 7f7aa227f700 1 --2- 172.21.15.31:0/608482789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc51597c0 0x555fc515cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.244+0000 7f7aa127d700 1 --2- 172.21.15.31:0/608482789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc5158ea0 0x555fc515c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7aa1a7e700 1 -- 172.21.15.31:0/608482789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc51597c0 msgr2=0x555fc515cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7aa1a7e700 1 --2- 172.21.15.31:0/608482789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc51597c0 0x555fc515cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7aa1a7e700 1 -- 172.21.15.31:0/608482789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc5158ea0 msgr2=0x555fc515c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7aa1a7e700 1 --2- 172.21.15.31:0/608482789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc5158ea0 0x555fc515c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7aa1a7e700 1 -- 172.21.15.31:0/608482789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fc5063580 con 0x555fc5153870 2024-04-26T18:01:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7aa127d700 1 --2- 172.21.15.31:0/608482789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc5158ea0 0x555fc515c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:01:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7aa1a7e700 1 --2- 172.21.15.31:0/608482789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5153870 0x555fc515f500 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7a9800ff30 tx=0x7f7a98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb84a31593ce33e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7a857fa700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:01:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7a857fa700 1 -- 172.21.15.31:0/608482789 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a98004230 con 0x555fc5153870 2024-04-26T18:01:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7aae574b80 1 -- 172.21.15.31:0/608482789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a940a1da0 con 0x555fc5153870 2024-04-26T18:01:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7a857fa700 1 -- 172.21.15.31:0/608482789 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7a980043d0 con 0x555fc5153870 2024-04-26T18:01:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7a857fa700 1 -- 172.21.15.31:0/608482789 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7a9802c020 con 0x555fc5153870 2024-04-26T18:01:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7aae574b80 10 client.15900 fetch_fsmap learned FSMap version 32 2024-04-26T18:01:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7aae574b80 1 -- 172.21.15.31:0/608482789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555fc516ec50 con 0x555fc5153870 2024-04-26T18:01:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.245+0000 7f7a857fa700 1 -- 172.21.15.31:0/608482789 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a940a1da0 con 0x555fc5153870 2024-04-26T18:01:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.246+0000 7f7a857fa700 1 -- 172.21.15.31:0/608482789 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f7a98031530 con 0x555fc5153870 2024-04-26T18:01:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.246+0000 7f7aae574b80 10 client.15900 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:01:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.246+0000 7f7aae574b80 10 client.15900 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T18:01:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.246+0000 7f7aae574b80 20 client.15900 populate_metadata read hostname 'smithi031' 2024-04-26T18:01:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.246+0000 7f7aae574b80 1 --2- 172.21.15.31:0/608482789 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555fc51711e0 0x555fc5173610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.246+0000 7f7aae574b80 4 client.15900 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:01:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.246+0000 7f7aae574b80 1 -- 172.21.15.31:0/608482789 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555fc50b3310 con 0x555fc51711e0 2024-04-26T18:01:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.246+0000 7f7aa127d700 1 --2- 172.21.15.31:0/608482789 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555fc51711e0 0x555fc5173610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.246+0000 7f7a857fa700 10 client.15900 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.246+0000 7f7aa127d700 1 --2- 172.21.15.31:0/608482789 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555fc51711e0 0x555fc5173610 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=996c096e2210ae5c server_cookie=5db7f9f03c1889f in_seq=0 out_seq=0 2024-04-26T18:01:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.247+0000 7f7a857fa700 1 -- 172.21.15.31:0/608482789 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555fc50b3310 con 0x555fc51711e0 2024-04-26T18:01:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.247+0000 7f7a857fa700 10 client.15900 handle_command_reply: tid=0 2024-04-26T18:01:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.247+0000 7f7aae574b80 1 client.15900 shutdown 2024-04-26T18:01:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.247+0000 7f7aae574b80 1 -- 172.21.15.31:0/608482789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5153870 msgr2=0x555fc515f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.247+0000 7f7aae574b80 1 --2- 172.21.15.31:0/608482789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5153870 0x555fc515f500 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7a9800ff30 tx=0x7f7a98004750 comp rx=0 tx=0).stop 2024-04-26T18:01:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.248+0000 7f7aae574b80 1 -- 172.21.15.31:0/608482789 shutdown_connections 2024-04-26T18:01:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.248+0000 7f7aae574b80 1 --2- 172.21.15.31:0/608482789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc51597c0 0x555fc515cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.248+0000 7f7aae574b80 1 --2- 172.21.15.31:0/608482789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5153870 0x555fc515f500 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.248+0000 7f7aae574b80 1 --2- 172.21.15.31:0/608482789 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555fc51711e0 0x555fc5173610 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.248+0000 7f7aae574b80 1 --2- 172.21.15.31:0/608482789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc5158ea0 0x555fc515c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.248+0000 7f7a857fa700 0 client.15900 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.248+0000 7f7aae574b80 1 -- 172.21.15.31:0/608482789 >> 172.21.15.31:0/608482789 conn(0x555fc50816c0 msgr2=0x555fc5152240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.248+0000 7f7aae574b80 1 -- 172.21.15.31:0/608482789 shutdown_connections 2024-04-26T18:01:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.248+0000 7f7aae574b80 1 -- 172.21.15.31:0/608482789 wait complete. 2024-04-26T18:01:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.248+0000 7f7aae574b80 20 client.15900 trim_cache size 0 max 16384 2024-04-26T18:01:45.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aae574b80 1 Processor -- start 2024-04-26T18:01:45.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aae574b80 1 -- start start 2024-04-26T18:01:45.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aae574b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5153870 0x555fc5153250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aae574b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc5158ea0 0x555fc516d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aae574b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc51597c0 0x555fc5147d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aae574b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1ae280 con 0x555fc51597c0 2024-04-26T18:01:45.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aae574b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1abab0 con 0x555fc5158ea0 2024-04-26T18:01:45.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aae574b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1a9920 con 0x555fc5153870 2024-04-26T18:01:45.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aa227f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc51597c0 0x555fc5147d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aa227f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc51597c0 0x555fc5147d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54688/0 (socket says 172.21.15.31:54688) 2024-04-26T18:01:45.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aa227f700 1 -- 172.21.15.31:0/2168117182 learned_addr learned my addr 172.21.15.31:0/2168117182 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:45.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aa1a7e700 1 --2- 172.21.15.31:0/2168117182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5153870 0x555fc5153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aa127d700 1 --2- 172.21.15.31:0/2168117182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc5158ea0 0x555fc516d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aa227f700 1 -- 172.21.15.31:0/2168117182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5153870 msgr2=0x555fc5153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:45.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aa227f700 1 --2- 172.21.15.31:0/2168117182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5153870 0x555fc5153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aa227f700 1 -- 172.21.15.31:0/2168117182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc5158ea0 msgr2=0x555fc516d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:45.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aa227f700 1 --2- 172.21.15.31:0/2168117182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc5158ea0 0x555fc516d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aa227f700 1 -- 172.21.15.31:0/2168117182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fc5173c90 con 0x555fc51597c0 2024-04-26T18:01:45.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aa127d700 1 --2- 172.21.15.31:0/2168117182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc5158ea0 0x555fc516d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:01:45.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.268+0000 7f7aa227f700 1 --2- 172.21.15.31:0/2168117182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc51597c0 0x555fc5147d80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009fd0 tx=0x7f7a94004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9fe38f7daa62ba9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:45.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.269+0000 7f7a877fe700 1 -- 172.21.15.31:0/2168117182 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9402a070 con 0x555fc51597c0 2024-04-26T18:01:45.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.269+0000 7f7a877fe700 1 -- 172.21.15.31:0/2168117182 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a94003db0 con 0x555fc51597c0 2024-04-26T18:01:45.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.269+0000 7f7a877fe700 1 -- 172.21.15.31:0/2168117182 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9409a060 con 0x555fc51597c0 2024-04-26T18:01:45.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.269+0000 7f7aae574b80 1 -- 172.21.15.31:0/2168117182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc51597c0 msgr2=0x555fc5147d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:45.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.269+0000 7f7aae574b80 1 --2- 172.21.15.31:0/2168117182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc51597c0 0x555fc5147d80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009fd0 tx=0x7f7a94004270 comp rx=0 tx=0).stop 2024-04-26T18:01:45.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.269+0000 7f7aae574b80 1 -- 172.21.15.31:0/2168117182 shutdown_connections 2024-04-26T18:01:45.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.269+0000 7f7aae574b80 1 --2- 172.21.15.31:0/2168117182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc51597c0 0x555fc5147d80 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.269+0000 7f7aae574b80 1 --2- 172.21.15.31:0/2168117182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5153870 0x555fc5153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.269+0000 7f7aae574b80 1 --2- 172.21.15.31:0/2168117182 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc5158ea0 0x555fc516d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.269+0000 7f7aae574b80 1 -- 172.21.15.31:0/2168117182 >> 172.21.15.31:0/2168117182 conn(0x555fc50816c0 msgr2=0x555fc5147670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:45.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.269+0000 7f7aae574b80 1 -- 172.21.15.31:0/2168117182 shutdown_connections 2024-04-26T18:01:45.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.269+0000 7f7aae574b80 1 -- 172.21.15.31:0/2168117182 wait complete. 2024-04-26T18:01:45.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.270+0000 7f7aae574b80 1 -- start start 2024-04-26T18:01:45.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.270+0000 7f7aae574b80 1 Processor -- start 2024-04-26T18:01:45.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.270+0000 7f7aae574b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5153870 0x555fc51667a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.270+0000 7f7aae574b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5158ea0 0x555fc5166ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.270+0000 7f7aae574b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc51597c0 0x555fc51d77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.270+0000 7f7aae574b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1a9f70 con 0x555fc5158ea0 2024-04-26T18:01:45.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.270+0000 7f7aae574b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1ab3e0 con 0x555fc51597c0 2024-04-26T18:01:45.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.270+0000 7f7aae574b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1aa640 con 0x555fc5153870 2024-04-26T18:01:45.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.270+0000 7f7aae574b80 10 client.0 _ll_register_callbacks cb 0x555fc503da50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:01:45.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.270+0000 7f7aa127d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5158ea0 0x555fc5166ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.270+0000 7f7aa127d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5158ea0 0x555fc5166ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54700/0 (socket says 172.21.15.31:54700) 2024-04-26T18:01:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.270+0000 7f7aa127d700 1 -- 172.21.15.31:0/3280533951 learned_addr learned my addr 172.21.15.31:0/3280533951 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.270+0000 7f7aa1a7e700 1 --2- 172.21.15.31:0/3280533951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5153870 0x555fc51667a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.270+0000 7f7aa227f700 1 --2- 172.21.15.31:0/3280533951 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc51597c0 0x555fc51d77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.271+0000 7f7aa127d700 1 -- 172.21.15.31:0/3280533951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5153870 msgr2=0x555fc51667a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.271+0000 7f7aa127d700 1 --2- 172.21.15.31:0/3280533951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5153870 0x555fc51667a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.271+0000 7f7aa127d700 1 -- 172.21.15.31:0/3280533951 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc51597c0 msgr2=0x555fc51d77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.271+0000 7f7aa127d700 1 --2- 172.21.15.31:0/3280533951 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc51597c0 0x555fc51d77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.271+0000 7f7aa127d700 1 -- 172.21.15.31:0/3280533951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fc516d800 con 0x555fc5158ea0 2024-04-26T18:01:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.271+0000 7f7aa127d700 1 --2- 172.21.15.31:0/3280533951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5158ea0 0x555fc5166ce0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x555fc5173f10 tx=0x7f7a8c009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eb64e6bcad4a26f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.271+0000 7f7a857fa700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:01:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.271+0000 7f7a857fa700 1 -- 172.21.15.31:0/3280533951 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c012e20 con 0x555fc5158ea0 2024-04-26T18:01:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.271+0000 7f7aae574b80 1 -- 172.21.15.31:0/3280533951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a94003db0 con 0x555fc5158ea0 2024-04-26T18:01:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.271+0000 7f7a857fa700 1 -- 172.21.15.31:0/3280533951 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7a8c01b2c0 con 0x555fc5158ea0 2024-04-26T18:01:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.271+0000 7f7a857fa700 1 -- 172.21.15.31:0/3280533951 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7a8c028020 con 0x555fc5158ea0 2024-04-26T18:01:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.271+0000 7f7aae574b80 10 client.15906 fetch_fsmap learned FSMap version 32 2024-04-26T18:01:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.271+0000 7f7aae574b80 1 -- 172.21.15.31:0/3280533951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555fc51d8cd0 con 0x555fc5158ea0 2024-04-26T18:01:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.271+0000 7f7a857fa700 1 -- 172.21.15.31:0/3280533951 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a94003db0 con 0x555fc5158ea0 2024-04-26T18:01:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.272+0000 7f7a857fa700 1 -- 172.21.15.31:0/3280533951 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f7a8c00f040 con 0x555fc5158ea0 2024-04-26T18:01:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.272+0000 7f7aae574b80 10 client.15906 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:01:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.272+0000 7f7aae574b80 10 client.15906 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T18:01:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.272+0000 7f7aae574b80 20 client.15906 populate_metadata read hostname 'smithi031' 2024-04-26T18:01:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.272+0000 7f7aae574b80 1 --2- 172.21.15.31:0/3280533951 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555fc51d9470 0x555fc51db890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.272+0000 7f7aae574b80 4 client.15906 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "38609351", "release": true, "await": true}] 2024-04-26T18:01:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.272+0000 7f7aae574b80 1 -- 172.21.15.31:0/3280533951 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "quiesce db", "set_id": "38609351", "release": true, "await": true}) -- 0x555fc514e560 con 0x555fc51d9470 2024-04-26T18:01:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.272+0000 7f7aa1a7e700 1 --2- 172.21.15.31:0/3280533951 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555fc51d9470 0x555fc51db890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.272+0000 7f7aa1a7e700 1 --2- 172.21.15.31:0/3280533951 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555fc51d9470 0x555fc51db890 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ab7d66b8bdec3b2 server_cookie=c03fdf9e41e7d35c in_seq=0 out_seq=0 2024-04-26T18:01:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.272+0000 7f7a857fa700 10 client.15906 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:45.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.293+0000 7f7a857fa700 1 -- 172.21.15.31:0/3280533951 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x555fc514e560 con 0x555fc51d9470 2024-04-26T18:01:45.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.293+0000 7f7a857fa700 10 client.15906 handle_command_reply: tid=0 2024-04-26T18:01:45.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.293+0000 7f7aae574b80 1 client.15906 shutdown 2024-04-26T18:01:45.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.293+0000 7f7aae574b80 1 -- 172.21.15.31:0/3280533951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5158ea0 msgr2=0x555fc5166ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:45.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.293+0000 7f7aae574b80 1 --2- 172.21.15.31:0/3280533951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5158ea0 0x555fc5166ce0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x555fc5173f10 tx=0x7f7a8c009cb0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.293+0000 7f7aae574b80 1 -- 172.21.15.31:0/3280533951 shutdown_connections 2024-04-26T18:01:45.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.293+0000 7f7aae574b80 1 --2- 172.21.15.31:0/3280533951 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555fc51d9470 0x555fc51db890 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.293+0000 7f7aae574b80 1 --2- 172.21.15.31:0/3280533951 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555fc51597c0 0x555fc51d77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.293+0000 7f7a857fa700 0 client.15906 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:45.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.293+0000 7f7aae574b80 1 --2- 172.21.15.31:0/3280533951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fc5153870 0x555fc51667a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.293+0000 7f7aae574b80 1 --2- 172.21.15.31:0/3280533951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc5158ea0 0x555fc5166ce0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.293+0000 7f7aae574b80 1 -- 172.21.15.31:0/3280533951 >> 172.21.15.31:0/3280533951 conn(0x555fc50816c0 msgr2=0x555fc5147f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:45.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.293+0000 7f7aae574b80 1 -- 172.21.15.31:0/3280533951 shutdown_connections 2024-04-26T18:01:45.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.294+0000 7f7aae574b80 1 -- 172.21.15.31:0/3280533951 wait complete. 2024-04-26T18:01:45.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.294+0000 7f7aae574b80 20 client.15906 trim_cache size 0 max 16384 2024-04-26T18:01:45.297 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T18:01:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 32, 2024-04-26T18:01:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "leader": 24473, 2024-04-26T18:01:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "set_version": 16, 2024-04-26T18:01:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "sets": { 2024-04-26T18:01:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "38609351": { 2024-04-26T18:01:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "version": 16, 2024-04-26T18:01:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "age_ref": 0.0, 2024-04-26T18:01:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T18:01:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RELEASED", 2024-04-26T18:01:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T18:01:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:01:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "timeout": 60.0, 2024-04-26T18:01:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "expiration": 79.5, 2024-04-26T18:01:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "members": { 2024-04-26T18:01:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "file:/": { 2024-04-26T18:01:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "excluded": false, 2024-04-26T18:01:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T18:01:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RELEASED", 2024-04-26T18:01:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T18:01:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:01:45.300 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T18:01:45.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.294+0000 7f7aa3ce2700 1 -- 172.21.15.31:0/1472961946 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7a880a3db0 msgr2=0x7f7a880a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:45.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.294+0000 7f7aa3ce2700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7a880a3db0 0x7f7a880a6270 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7a98005e10 tx=0x7f7a98005d60 comp rx=0 tx=0).stop 2024-04-26T18:01:45.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.294+0000 7f7aa3ce2700 1 -- 172.21.15.31:0/1472961946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a9c1094e0 msgr2=0x7f7a9c199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:45.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.294+0000 7f7aa3ce2700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a9c1094e0 0x7f7a9c199930 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7a9400b830 tx=0x7f7a94004d90 comp rx=0 tx=0).stop 2024-04-26T18:01:45.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.294+0000 7f7aa3ce2700 1 -- 172.21.15.31:0/1472961946 shutdown_connections 2024-04-26T18:01:45.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.294+0000 7f7aa3ce2700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7a880a3db0 0x7f7a880a6270 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.294+0000 7f7aa3ce2700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a9c1034a0 0x7f7a9c19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.294+0000 7f7aa3ce2700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a9c1094e0 0x7f7a9c199930 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.294+0000 7f7aa3ce2700 1 --2- 172.21.15.31:0/1472961946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a9c103e60 0x7f7a9c19fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:45.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.294+0000 7f7aa3ce2700 1 -- 172.21.15.31:0/1472961946 >> 172.21.15.31:0/1472961946 conn(0x7f7a9c07a590 msgr2=0x7f7a9c1021a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:45.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.294+0000 7f7aa3ce2700 1 -- 172.21.15.31:0/1472961946 shutdown_connections 2024-04-26T18:01:45.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:01:45.294+0000 7f7aa3ce2700 1 -- 172.21.15.31:0/1472961946 wait complete. 2024-04-26T18:01:45.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:45 smithi097 ceph-mon[32419]: pgmap v160: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-26T18:01:45.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:45 smithi186 ceph-mon[32797]: pgmap v160: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-26T18:01:46.051 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '38609351', seconds elapsed: 34.1 2024-04-26T18:01:47.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:47 smithi097 ceph-mon[32419]: pgmap v161: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 323 KiB/s wr, 41 op/s 2024-04-26T18:01:47.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:47 smithi186 ceph-mon[32797]: pgmap v161: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 323 KiB/s wr, 41 op/s 2024-04-26T18:01:47.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:47 smithi031 ceph-mon[29164]: pgmap v161: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 323 KiB/s wr, 41 op/s 2024-04-26T18:01:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 265.0 s, 1691.9 tps, lat 346.540 ms stddev 2870.180 2024-04-26T18:01:48.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.293+0000 7f5d27417640 1 -- 172.21.15.31:0/3558001368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d2011c110 msgr2=0x7f5d2011e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:48.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.293+0000 7f5d27417640 1 --2- 172.21.15.31:0/3558001368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d2011c110 0x7f5d2011e500 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5d18009cd0 tx=0x7f5d1802a970 comp rx=0 tx=0).stop 2024-04-26T18:01:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.296+0000 7f5d27417640 1 -- 172.21.15.31:0/3558001368 shutdown_connections 2024-04-26T18:01:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.296+0000 7f5d27417640 1 --2- 172.21.15.31:0/3558001368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d2011c110 0x7f5d2011e500 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.296+0000 7f5d27417640 1 --2- 172.21.15.31:0/3558001368 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d20111500 0x7f5d20109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.296+0000 7f5d27417640 1 --2- 172.21.15.31:0/3558001368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d20111120 0x7f5d20109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.296+0000 7f5d27417640 1 -- 172.21.15.31:0/3558001368 >> 172.21.15.31:0/3558001368 conn(0x7f5d2006da00 msgr2=0x7f5d2006de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.299+0000 7f5d27417640 1 -- 172.21.15.31:0/3558001368 shutdown_connections 2024-04-26T18:01:48.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.300+0000 7f5d27417640 1 -- 172.21.15.31:0/3558001368 wait complete. 2024-04-26T18:01:48.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.300+0000 7f5d27417640 1 Processor -- start 2024-04-26T18:01:48.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.301+0000 7f5d27417640 1 -- start start 2024-04-26T18:01:48.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.301+0000 7f5d27417640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d20111120 0x7f5d201a7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.301+0000 7f5d27417640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2011c110 0x7f5d201a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.301+0000 7f5d27417640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d201ac7e0 0x7f5d201acc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.301+0000 7f5d27417640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5d201219e0 con 0x7f5d201ac7e0 2024-04-26T18:01:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.301+0000 7f5d27417640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f5d20121860 con 0x7f5d20111120 2024-04-26T18:01:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.301+0000 7f5d27417640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5d20121b60 con 0x7f5d2011c110 2024-04-26T18:01:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.301+0000 7f5d259b4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d201ac7e0 0x7f5d201acc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.301+0000 7f5d259b4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d201ac7e0 0x7f5d201acc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54710/0 (socket says 172.21.15.31:54710) 2024-04-26T18:01:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.301+0000 7f5d259b4640 1 -- 172.21.15.31:0/2132808626 learned_addr learned my addr 172.21.15.31:0/2132808626 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.301+0000 7f5d251b3640 1 --2- 172.21.15.31:0/2132808626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d20111120 0x7f5d201a7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.302+0000 7f5d259b4640 1 -- 172.21.15.31:0/2132808626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2011c110 msgr2=0x7f5d201a8460 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T18:01:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.302+0000 7f5d259b4640 1 --2- 172.21.15.31:0/2132808626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2011c110 0x7f5d201a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.302+0000 7f5d259b4640 1 -- 172.21.15.31:0/2132808626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d20111120 msgr2=0x7f5d201a7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.302+0000 7f5d259b4640 1 --2- 172.21.15.31:0/2132808626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d20111120 0x7f5d201a7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.302+0000 7f5d259b4640 1 -- 172.21.15.31:0/2132808626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d201ad2d0 con 0x7f5d201ac7e0 2024-04-26T18:01:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.308+0000 7f5d259b4640 1 --2- 172.21.15.31:0/2132808626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d201ac7e0 0x7f5d201acc90 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5d18009ca0 tx=0x7f5d1802ffe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.309+0000 7f5d167fc640 1 -- 172.21.15.31:0/2132808626 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d18047070 con 0x7f5d201ac7e0 2024-04-26T18:01:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.309+0000 7f5d27417640 1 -- 172.21.15.31:0/2132808626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d201ad560 con 0x7f5d201ac7e0 2024-04-26T18:01:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.309+0000 7f5d27417640 1 -- 172.21.15.31:0/2132808626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d201b4c70 con 0x7f5d201ac7e0 2024-04-26T18:01:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.309+0000 7f5d167fc640 1 -- 172.21.15.31:0/2132808626 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d1802f830 con 0x7f5d201ac7e0 2024-04-26T18:01:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.309+0000 7f5d167fc640 1 -- 172.21.15.31:0/2132808626 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d18034070 con 0x7f5d201ac7e0 2024-04-26T18:01:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.317+0000 7f5d2a20d740 1 Processor -- start 2024-04-26T18:01:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.317+0000 7f5d167fc640 1 -- 172.21.15.31:0/2132808626 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5d1804a070 con 0x7f5d201ac7e0 2024-04-26T18:01:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.317+0000 7f5d2a20d740 1 -- start start 2024-04-26T18:01:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.317+0000 7f5d2a20d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd8ec0 0x555906fd92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.317+0000 7f5d2a20d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd97e0 0x555906fe3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.317+0000 7f5d2a20d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 0x555906fe6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.317+0000 7f5d2a20d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555906f3f460 con 0x555906fd8ec0 2024-04-26T18:01:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.317+0000 7f5d2a20d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x555906ef9590 con 0x555906fe40a0 2024-04-26T18:01:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.317+0000 7f5d2a20d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555906f3b420 con 0x555906fd97e0 2024-04-26T18:01:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.318+0000 7f5d249b2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd8ec0 0x555906fd92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.318+0000 7f5d249b2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd8ec0 0x555906fd92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54718/0 (socket says 172.21.15.31:54718) 2024-04-26T18:01:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.318+0000 7f5d249b2640 1 -- 172.21.15.31:0/1024694739 learned_addr learned my addr 172.21.15.31:0/1024694739 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.318+0000 7f5d251b3640 1 --2- 172.21.15.31:0/1024694739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd97e0 0x555906fe3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.318+0000 7f5d249b2640 1 --2- 172.21.15.31:0/1024694739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 0x555906fe6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.318+0000 7f5d251b3640 1 -- 172.21.15.31:0/1024694739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 msgr2=0x555906fe6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.318+0000 7f5d251b3640 1 --2- 172.21.15.31:0/1024694739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 0x555906fe6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.318+0000 7f5d251b3640 1 -- 172.21.15.31:0/1024694739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd8ec0 msgr2=0x555906fd92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.318+0000 7f5d251b3640 1 --2- 172.21.15.31:0/1024694739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd8ec0 0x555906fd92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.318+0000 7f5d251b3640 1 -- 172.21.15.31:0/1024694739 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555906f060d0 con 0x555906fd97e0 2024-04-26T18:01:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.318+0000 7f5d249b2640 1 --2- 172.21.15.31:0/1024694739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd8ec0 0x555906fd92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:01:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.318+0000 7f5d251b3640 1 --2- 172.21.15.31:0/1024694739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd97e0 0x555906fe3b60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009c00 tx=0x7f5d1c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4335a842629db49d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.318+0000 7f5d167fc640 1 --2- 172.21.15.31:0/2132808626 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f5d0c0a3f20 0x7f5d0c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.318+0000 7f5d0b7fe640 1 -- 172.21.15.31:0/1024694739 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c004230 con 0x555906fd97e0 2024-04-26T18:01:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.319+0000 7f5d249b2640 1 --2- 172.21.15.31:0/2132808626 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f5d0c0a3f20 0x7f5d0c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.319+0000 7f5d0b7fe640 1 -- 172.21.15.31:0/1024694739 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d1c0043d0 con 0x555906fd97e0 2024-04-26T18:01:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.319+0000 7f5d0b7fe640 1 -- 172.21.15.31:0/1024694739 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c004c70 con 0x555906fd97e0 2024-04-26T18:01:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.319+0000 7f5d167fc640 1 -- 172.21.15.31:0/2132808626 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f5d180f0b10 con 0x7f5d201ac7e0 2024-04-26T18:01:48.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.320+0000 7f5d2a20d740 1 -- 172.21.15.31:0/1024694739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd97e0 msgr2=0x555906fe3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:48.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.320+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/1024694739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd97e0 0x555906fe3b60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009c00 tx=0x7f5d1c011810 comp rx=0 tx=0).stop 2024-04-26T18:01:48.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.320+0000 7f5d249b2640 1 --2- 172.21.15.31:0/2132808626 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f5d0c0a3f20 0x7f5d0c0a63e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x555906fd67f0 tx=0x7f5d10014040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:48.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.321+0000 7f5d2a20d740 1 -- 172.21.15.31:0/1024694739 shutdown_connections 2024-04-26T18:01:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.321+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/1024694739 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 0x555906fe6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.321+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/1024694739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd97e0 0x555906fe3b60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.321+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/1024694739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd8ec0 0x555906fd92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.321+0000 7f5d2a20d740 1 -- 172.21.15.31:0/1024694739 >> 172.21.15.31:0/1024694739 conn(0x555906fd53d0 msgr2=0x555906fd57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.321+0000 7f5d2a20d740 1 -- 172.21.15.31:0/1024694739 shutdown_connections 2024-04-26T18:01:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.322+0000 7f5d2a20d740 1 -- 172.21.15.31:0/1024694739 wait complete. 2024-04-26T18:01:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.322+0000 7f5d2a20d740 1 -- start start 2024-04-26T18:01:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.322+0000 7f5d2a20d740 1 Processor -- start 2024-04-26T18:01:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.323+0000 7f5d2a20d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd8ec0 0x555906fe10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.323+0000 7f5d2a20d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd97e0 0x555906fe15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.323+0000 7f5d2a20d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 0x555906fe1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.323+0000 7f5d2a20d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555906fe6cd0 con 0x555906fd8ec0 2024-04-26T18:01:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.323+0000 7f5d2a20d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x555906ec6b60 con 0x555906fe40a0 2024-04-26T18:01:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.324+0000 7f5d251b3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd8ec0 0x555906fe10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.324+0000 7f5d251b3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd8ec0 0x555906fe10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54726/0 (socket says 172.21.15.31:54726) 2024-04-26T18:01:48.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.324+0000 7f5d251b3640 1 -- 172.21.15.31:0/2318360316 learned_addr learned my addr 172.21.15.31:0/2318360316 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:48.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.324+0000 7f5d249b2640 1 --2- 172.21.15.31:0/2318360316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd97e0 0x555906fe15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.324+0000 7f5d2a20d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555906fe6e50 con 0x555906fd97e0 2024-04-26T18:01:48.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.324+0000 7f5d2a20d740 10 client.0 _ll_register_callbacks cb 0x555906eeb2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:01:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.324+0000 7f5d259b4640 1 --2- 172.21.15.31:0/2318360316 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 0x555906fe1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.325+0000 7f5d259b4640 1 -- 172.21.15.31:0/2318360316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd97e0 msgr2=0x555906fe15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.325+0000 7f5d259b4640 1 --2- 172.21.15.31:0/2318360316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd97e0 0x555906fe15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.325+0000 7f5d259b4640 1 -- 172.21.15.31:0/2318360316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd8ec0 msgr2=0x555906fe10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.325+0000 7f5d259b4640 1 --2- 172.21.15.31:0/2318360316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd8ec0 0x555906fe10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.325+0000 7f5d259b4640 1 -- 172.21.15.31:0/2318360316 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555906c51210 con 0x555906fe40a0 2024-04-26T18:01:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.325+0000 7f5d259b4640 1 --2- 172.21.15.31:0/2318360316 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 0x555906fe1b20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5d180ee2b0 tx=0x7f5d180bc7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=268006dc24853a62 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.325+0000 7f5d08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T18:01:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.325+0000 7f5d08ff9640 1 -- 172.21.15.31:0/2318360316 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d180c0fd0 con 0x555906fe40a0 2024-04-26T18:01:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.325+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2318360316 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d1c0043d0 con 0x555906fe40a0 2024-04-26T18:01:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.325+0000 7f5d08ff9640 1 -- 172.21.15.31:0/2318360316 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f5d180ee530 con 0x555906fe40a0 2024-04-26T18:01:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.325+0000 7f5d08ff9640 1 -- 172.21.15.31:0/2318360316 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5d180c5890 con 0x555906fe40a0 2024-04-26T18:01:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.325+0000 7f5d2a20d740 10 client.25589 fetch_fsmap learned FSMap version 32 2024-04-26T18:01:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.325+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2318360316 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555906d1d620 con 0x555906fe40a0 2024-04-26T18:01:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.326+0000 7f5d08ff9640 1 -- 172.21.15.31:0/2318360316 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d1c0043d0 con 0x555906fe40a0 2024-04-26T18:01:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.326+0000 7f5d08ff9640 1 -- 172.21.15.31:0/2318360316 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f5d18041030 con 0x555906fe40a0 2024-04-26T18:01:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.326+0000 7f5d2a20d740 10 client.25589 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:01:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.326+0000 7f5d2a20d740 10 client.25589 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:01:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.326+0000 7f5d2a20d740 20 client.25589 populate_metadata read hostname 'smithi031' 2024-04-26T18:01:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.326+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/2318360316 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555906ff4290 0x555906ff66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.326+0000 7f5d2a20d740 4 client.25589 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:01:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.326+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2318360316 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555906f4d070 con 0x555906ff4290 2024-04-26T18:01:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.326+0000 7f5d251b3640 1 --2- 172.21.15.31:0/2318360316 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555906ff4290 0x555906ff66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.327+0000 7f5d251b3640 1 --2- 172.21.15.31:0/2318360316 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555906ff4290 0x555906ff66d0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a384b34e2c1f7377 server_cookie=ef2edbcc9e08f617 in_seq=0 out_seq=0 2024-04-26T18:01:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.327+0000 7f5d08ff9640 10 client.25589 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.328+0000 7f5d08ff9640 1 -- 172.21.15.31:0/2318360316 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555906f4d070 con 0x555906ff4290 2024-04-26T18:01:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.328+0000 7f5d08ff9640 10 client.25589 handle_command_reply: tid=0 2024-04-26T18:01:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.328+0000 7f5d2a20d740 1 client.25589 shutdown 2024-04-26T18:01:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.328+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2318360316 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 msgr2=0x555906fe1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.328+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/2318360316 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 0x555906fe1b20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5d180ee2b0 tx=0x7f5d180bc7e0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.328+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2318360316 shutdown_connections 2024-04-26T18:01:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.328+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/2318360316 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555906ff4290 0x555906ff66d0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.328+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/2318360316 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 0x555906fe1b20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.328+0000 7f5d08ff9640 0 client.25589 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.328+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/2318360316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd97e0 0x555906fe15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.328+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/2318360316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd8ec0 0x555906fe10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.329+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2318360316 >> 172.21.15.31:0/2318360316 conn(0x555906fd53d0 msgr2=0x555906fdb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.329+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2318360316 shutdown_connections 2024-04-26T18:01:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.329+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2318360316 wait complete. 2024-04-26T18:01:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.329+0000 7f5d2a20d740 20 client.25589 trim_cache size 0 max 16384 2024-04-26T18:01:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.352+0000 7f5d2a20d740 1 Processor -- start 2024-04-26T18:01:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.352+0000 7f5d2a20d740 1 -- start start 2024-04-26T18:01:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d2a20d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd8ec0 0x555906fefb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d2a20d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fd97e0 0x555906fe1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d2a20d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fe40a0 0x555906ffbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d2a20d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5d201bf910 con 0x555906fe40a0 2024-04-26T18:01:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d251b3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd8ec0 0x555906fefb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d259b4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fe40a0 0x555906ffbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d259b4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fe40a0 0x555906ffbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54742/0 (socket says 172.21.15.31:54742) 2024-04-26T18:01:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d251b3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd8ec0 0x555906fefb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:37214/0 (socket says 172.21.15.31:37214) 2024-04-26T18:01:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d259b4640 1 -- 172.21.15.31:0/2151192601 learned_addr learned my addr 172.21.15.31:0/2151192601 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d249b2640 1 --2- 172.21.15.31:0/2151192601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fd97e0 0x555906fe1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2151192601 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f5d201bd180 con 0x555906fd97e0 2024-04-26T18:01:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2151192601 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5d201bafd0 con 0x555906fd8ec0 2024-04-26T18:01:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d259b4640 1 -- 172.21.15.31:0/2151192601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd8ec0 msgr2=0x555906fefb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d259b4640 1 --2- 172.21.15.31:0/2151192601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd8ec0 0x555906fefb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d259b4640 1 -- 172.21.15.31:0/2151192601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fd97e0 msgr2=0x555906fe1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.353+0000 7f5d259b4640 1 --2- 172.21.15.31:0/2151192601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fd97e0 0x555906fe1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.354+0000 7f5d259b4640 1 -- 172.21.15.31:0/2151192601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555906fcf9c0 con 0x555906fe40a0 2024-04-26T18:01:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.355+0000 7f5d259b4640 1 --2- 172.21.15.31:0/2151192601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fe40a0 0x555906ffbfe0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5d18004d60 tx=0x7f5d18003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7799fef460399b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.355+0000 7f5d0affd640 1 -- 172.21.15.31:0/2151192601 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d180ee530 con 0x555906fe40a0 2024-04-26T18:01:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.355+0000 7f5d0affd640 1 -- 172.21.15.31:0/2151192601 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d180b9290 con 0x555906fe40a0 2024-04-26T18:01:48.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.355+0000 7f5d0affd640 1 -- 172.21.15.31:0/2151192601 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d180b8970 con 0x555906fe40a0 2024-04-26T18:01:48.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.356+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2151192601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fe40a0 msgr2=0x555906ffbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.356+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/2151192601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fe40a0 0x555906ffbfe0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5d18004d60 tx=0x7f5d18003f50 comp rx=0 tx=0).stop 2024-04-26T18:01:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.356+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2151192601 shutdown_connections 2024-04-26T18:01:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.356+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/2151192601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fe40a0 0x555906ffbfe0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.356+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/2151192601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fd97e0 0x555906fe1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.356+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/2151192601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd8ec0 0x555906fefb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.356+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2151192601 >> 172.21.15.31:0/2151192601 conn(0x555906fd53d0 msgr2=0x555906fdf970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.356+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2151192601 shutdown_connections 2024-04-26T18:01:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.356+0000 7f5d2a20d740 1 -- 172.21.15.31:0/2151192601 wait complete. 2024-04-26T18:01:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.357+0000 7f5d2a20d740 1 -- start start 2024-04-26T18:01:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.357+0000 7f5d2a20d740 1 Processor -- start 2024-04-26T18:01:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.357+0000 7f5d2a20d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd8ec0 0x555906fe76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.357+0000 7f5d2a20d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd97e0 0x555906fe7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.357+0000 7f5d2a20d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 0x555906febaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.357+0000 7f5d2a20d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5d201bb640 con 0x555906fd97e0 2024-04-26T18:01:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.357+0000 7f5d2a20d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f5d201bcab0 con 0x555906fe40a0 2024-04-26T18:01:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.357+0000 7f5d2a20d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5d201bbd10 con 0x555906fd8ec0 2024-04-26T18:01:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.357+0000 7f5d2a20d740 10 client.0 _ll_register_callbacks cb 0x7f5d180bc9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:01:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.357+0000 7f5d251b3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd8ec0 0x555906fe76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.357+0000 7f5d249b2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd97e0 0x555906fe7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.357+0000 7f5d251b3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd8ec0 0x555906fe76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:37230/0 (socket says 172.21.15.31:37230) 2024-04-26T18:01:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.357+0000 7f5d251b3640 1 -- 172.21.15.31:0/959706486 learned_addr learned my addr 172.21.15.31:0/959706486 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:01:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.357+0000 7f5d259b4640 1 --2- 172.21.15.31:0/959706486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 0x555906febaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.358+0000 7f5d249b2640 1 -- 172.21.15.31:0/959706486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd8ec0 msgr2=0x555906fe76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.358+0000 7f5d249b2640 1 --2- 172.21.15.31:0/959706486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd8ec0 0x555906fe76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.358+0000 7f5d249b2640 1 -- 172.21.15.31:0/959706486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 msgr2=0x555906febaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:01:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.358+0000 7f5d249b2640 1 --2- 172.21.15.31:0/959706486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 0x555906febaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.358+0000 7f5d249b2640 1 -- 172.21.15.31:0/959706486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555906fec110 con 0x555906fd97e0 2024-04-26T18:01:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.358+0000 7f5d259b4640 1 --2- 172.21.15.31:0/959706486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 0x555906febaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:01:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.359+0000 7f5d249b2640 1 --2- 172.21.15.31:0/959706486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd97e0 0x555906fe7be0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x555906fe2670 tx=0x7f5d10009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b01c950f889643d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:01:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.359+0000 7f5d08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:01:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.359+0000 7f5d08ff9640 1 -- 172.21.15.31:0/959706486 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1000f870 con 0x555906fd97e0 2024-04-26T18:01:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.359+0000 7f5d08ff9640 1 -- 172.21.15.31:0/959706486 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f5d1000fa10 con 0x555906fd97e0 2024-04-26T18:01:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.359+0000 7f5d2a20d740 1 -- 172.21.15.31:0/959706486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d180b9290 con 0x555906fd97e0 2024-04-26T18:01:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.360+0000 7f5d08ff9640 1 -- 172.21.15.31:0/959706486 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5d1000fc10 con 0x555906fd97e0 2024-04-26T18:01:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.360+0000 7f5d2a20d740 10 client.15924 fetch_fsmap learned FSMap version 32 2024-04-26T18:01:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.360+0000 7f5d2a20d740 1 -- 172.21.15.31:0/959706486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55590705a3a0 con 0x555906fd97e0 2024-04-26T18:01:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.360+0000 7f5d08ff9640 1 -- 172.21.15.31:0/959706486 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d1000fdb0 con 0x555906fd97e0 2024-04-26T18:01:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.361+0000 7f5d08ff9640 1 -- 172.21.15.31:0/959706486 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f5d10004b80 con 0x555906fd97e0 2024-04-26T18:01:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.361+0000 7f5d2a20d740 10 client.15924 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:01:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.361+0000 7f5d2a20d740 10 client.15924 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:01:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.361+0000 7f5d2a20d740 20 client.15924 populate_metadata read hostname 'smithi031' 2024-04-26T18:01:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.361+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/959706486 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555906ff4290 0x55590705cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:01:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.361+0000 7f5d2a20d740 4 client.15924 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:01:48.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.361+0000 7f5d2a20d740 1 -- 172.21.15.31:0/959706486 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f5d180b9290 con 0x555906ff4290 2024-04-26T18:01:48.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.361+0000 7f5d251b3640 1 --2- 172.21.15.31:0/959706486 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555906ff4290 0x55590705cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:01:48.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.362+0000 7f5d08ff9640 10 client.15924 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:48.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.362+0000 7f5d251b3640 1 --2- 172.21.15.31:0/959706486 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555906ff4290 0x55590705cb70 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a90cf3dc7e8fed3 server_cookie=1241c9daf14fbeaf in_seq=0 out_seq=0 2024-04-26T18:01:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.374+0000 7f5d08ff9640 1 -- 172.21.15.31:0/959706486 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f5d180b9290 con 0x555906ff4290 2024-04-26T18:01:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.374+0000 7f5d08ff9640 10 client.15924 handle_command_reply: tid=0 2024-04-26T18:01:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.374+0000 7f5d2a20d740 1 client.15924 shutdown 2024-04-26T18:01:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.374+0000 7f5d2a20d740 1 -- 172.21.15.31:0/959706486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd97e0 msgr2=0x555906fe7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.375+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/959706486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd97e0 0x555906fe7be0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x555906fe2670 tx=0x7f5d10009750 comp rx=0 tx=0).stop 2024-04-26T18:01:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.375+0000 7f5d2a20d740 1 -- 172.21.15.31:0/959706486 shutdown_connections 2024-04-26T18:01:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.375+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/959706486 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555906ff4290 0x55590705cb70 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.376+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/959706486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555906fe40a0 0x555906febaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.376+0000 7f5d08ff9640 0 client.15924 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:01:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.376+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/959706486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555906fd97e0 0x555906fe7be0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.376+0000 7f5d2a20d740 1 --2- 172.21.15.31:0/959706486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555906fd8ec0 0x555906fe76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.376+0000 7f5d2a20d740 1 -- 172.21.15.31:0/959706486 >> 172.21.15.31:0/959706486 conn(0x555906fd53d0 msgr2=0x555906fd3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.376+0000 7f5d2a20d740 1 -- 172.21.15.31:0/959706486 shutdown_connections 2024-04-26T18:01:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.376+0000 7f5d2a20d740 1 -- 172.21.15.31:0/959706486 wait complete. 2024-04-26T18:01:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.376+0000 7f5d2a20d740 20 client.15924 trim_cache size 0 max 16384 2024-04-26T18:01:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:01:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:01:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:01:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:01:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.377+0000 7f5d27417640 1 -- 172.21.15.31:0/2132808626 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f5d0c0a3f20 msgr2=0x7f5d0c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.377+0000 7f5d27417640 1 --2- 172.21.15.31:0/2132808626 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f5d0c0a3f20 0x7f5d0c0a63e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x555906fd67f0 tx=0x7f5d10014040 comp rx=0 tx=0).stop 2024-04-26T18:01:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.377+0000 7f5d27417640 1 -- 172.21.15.31:0/2132808626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d201ac7e0 msgr2=0x7f5d201acc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:01:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.377+0000 7f5d27417640 1 --2- 172.21.15.31:0/2132808626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d201ac7e0 0x7f5d201acc90 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5d18009ca0 tx=0x7f5d1802ffe0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.377+0000 7f5d27417640 1 -- 172.21.15.31:0/2132808626 shutdown_connections 2024-04-26T18:01:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.377+0000 7f5d27417640 1 --2- 172.21.15.31:0/2132808626 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f5d0c0a3f20 0x7f5d0c0a63e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.377+0000 7f5d27417640 1 --2- 172.21.15.31:0/2132808626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d201ac7e0 0x7f5d201acc90 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.377+0000 7f5d27417640 1 --2- 172.21.15.31:0/2132808626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2011c110 0x7f5d201a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.377+0000 7f5d27417640 1 --2- 172.21.15.31:0/2132808626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5d20111120 0x7f5d201a7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:01:48.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.377+0000 7f5d27417640 1 -- 172.21.15.31:0/2132808626 >> 172.21.15.31:0/2132808626 conn(0x7f5d2006da00 msgr2=0x7f5d2010ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:01:48.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.378+0000 7f5d27417640 1 -- 172.21.15.31:0/2132808626 shutdown_connections 2024-04-26T18:01:48.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:01:48.379+0000 7f5d27417640 1 -- 172.21.15.31:0/2132808626 wait complete. 2024-04-26T18:01:49.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:49 smithi097 ceph-mon[32419]: pgmap v162: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 356 KiB/s wr, 43 op/s 2024-04-26T18:01:49.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:49 smithi186 ceph-mon[32797]: pgmap v162: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 356 KiB/s wr, 43 op/s 2024-04-26T18:01:49.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:49 smithi031 ceph-mon[29164]: pgmap v162: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 356 KiB/s wr, 43 op/s 2024-04-26T18:01:51.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:51 smithi097 ceph-mon[32419]: pgmap v163: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 351 KiB/s wr, 42 op/s 2024-04-26T18:01:51.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:51 smithi186 ceph-mon[32797]: pgmap v163: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 351 KiB/s wr, 42 op/s 2024-04-26T18:01:51.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:51 smithi031 ceph-mon[29164]: pgmap v163: 97 pgs: 97 active+clean; 2.8 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 351 KiB/s wr, 42 op/s 2024-04-26T18:01:53.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 270.0 s, 3376.6 tps, lat 29.638 ms stddev 30.603 2024-04-26T18:01:53.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:53 smithi097 ceph-mon[32419]: pgmap v164: 97 pgs: 97 active+clean; 2.8 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.7 MiB/s wr, 453 op/s 2024-04-26T18:01:53.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:53 smithi186 ceph-mon[32797]: pgmap v164: 97 pgs: 97 active+clean; 2.8 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.7 MiB/s wr, 453 op/s 2024-04-26T18:01:53.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:53 smithi031 ceph-mon[29164]: pgmap v164: 97 pgs: 97 active+clean; 2.8 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.7 MiB/s wr, 453 op/s 2024-04-26T18:01:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:55 smithi097 ceph-mon[32419]: pgmap v165: 97 pgs: 97 active+clean; 2.8 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.7 MiB/s wr, 450 op/s 2024-04-26T18:01:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:55 smithi186 ceph-mon[32797]: pgmap v165: 97 pgs: 97 active+clean; 2.8 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.7 MiB/s wr, 450 op/s 2024-04-26T18:01:55.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:55 smithi031 ceph-mon[29164]: pgmap v165: 97 pgs: 97 active+clean; 2.8 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.7 MiB/s wr, 450 op/s 2024-04-26T18:01:57.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:57 smithi186 ceph-mon[32797]: pgmap v166: 97 pgs: 97 active+clean; 2.8 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.3 MiB/s wr, 706 op/s 2024-04-26T18:01:57.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:57 smithi031 ceph-mon[29164]: pgmap v166: 97 pgs: 97 active+clean; 2.8 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.3 MiB/s wr, 706 op/s 2024-04-26T18:01:57.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:57 smithi097 ceph-mon[32419]: pgmap v166: 97 pgs: 97 active+clean; 2.8 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.3 MiB/s wr, 706 op/s 2024-04-26T18:01:58.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 275.0 s, 3007.4 tps, lat 33.280 ms stddev 36.520 2024-04-26T18:01:59.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:01:59 smithi186 ceph-mon[32797]: pgmap v167: 97 pgs: 97 active+clean; 2.8 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.3 MiB/s wr, 717 op/s 2024-04-26T18:01:59.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:01:59 smithi031 ceph-mon[29164]: pgmap v167: 97 pgs: 97 active+clean; 2.8 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.3 MiB/s wr, 717 op/s 2024-04-26T18:01:59.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:01:59 smithi097 ceph-mon[32419]: pgmap v167: 97 pgs: 97 active+clean; 2.8 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.3 MiB/s wr, 717 op/s 2024-04-26T18:02:01.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:01 smithi186 ceph-mon[32797]: pgmap v168: 97 pgs: 97 active+clean; 2.8 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.3 MiB/s wr, 711 op/s 2024-04-26T18:02:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.530+0000 7fd1509af640 1 -- 172.21.15.31:0/280785895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd144004c80 msgr2=0x7fd144005100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.530+0000 7fd1509af640 1 --2- 172.21.15.31:0/280785895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd144004c80 0x7fd144005100 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd140009900 tx=0x7fd14002a440 comp rx=0 tx=0).stop 2024-04-26T18:02:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.534+0000 7fd1509af640 1 -- 172.21.15.31:0/280785895 shutdown_connections 2024-04-26T18:02:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.534+0000 7fd1509af640 1 --2- 172.21.15.31:0/280785895 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1440986e0 0x7fd144098b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.534+0000 7fd1509af640 1 --2- 172.21.15.31:0/280785895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd144004c80 0x7fd144005100 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.534+0000 7fd1509af640 1 --2- 172.21.15.31:0/280785895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd14409acf0 0x7fd144004740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.534+0000 7fd1509af640 1 -- 172.21.15.31:0/280785895 >> 172.21.15.31:0/280785895 conn(0x7fd14400a740 msgr2=0x7fd14400ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.542+0000 7fd1509af640 1 -- 172.21.15.31:0/280785895 shutdown_connections 2024-04-26T18:02:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.544+0000 7fd1509af640 1 -- 172.21.15.31:0/280785895 wait complete. 2024-04-26T18:02:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.546+0000 7fd1509af640 1 Processor -- start 2024-04-26T18:02:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.546+0000 7fd1509af640 1 -- start start 2024-04-26T18:02:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.546+0000 7fd1509af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd144004c80 0x7fd1441496e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.546+0000 7fd1509af640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1440986e0 0x7fd144149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.546+0000 7fd1509af640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14409acf0 0x7fd144143890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.546+0000 7fd1509af640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd1440ac5c0 con 0x7fd1440986e0 2024-04-26T18:02:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.546+0000 7fd1509af640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd1440ac440 con 0x7fd14409acf0 2024-04-26T18:02:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.546+0000 7fd1509af640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd1440ac740 con 0x7fd144004c80 2024-04-26T18:02:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.547+0000 7fd14ad9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14409acf0 0x7fd144143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.549+0000 7fd14ad9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14409acf0 0x7fd144143890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:47158/0 (socket says 172.21.15.31:47158) 2024-04-26T18:02:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.549+0000 7fd14ad9d640 1 -- 172.21.15.31:0/3991845413 learned_addr learned my addr 172.21.15.31:0/3991845413 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.550+0000 7fd149d9b640 1 --2- 172.21.15.31:0/3991845413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1440986e0 0x7fd144149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.552+0000 7fd14ad9d640 1 -- 172.21.15.31:0/3991845413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd144004c80 msgr2=0x7fd1441496e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T18:02:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.552+0000 7fd14ad9d640 1 --2- 172.21.15.31:0/3991845413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd144004c80 0x7fd1441496e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.552+0000 7fd14ad9d640 1 -- 172.21.15.31:0/3991845413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1440986e0 msgr2=0x7fd144149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.552+0000 7fd14ad9d640 1 --2- 172.21.15.31:0/3991845413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1440986e0 0x7fd144149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.552+0000 7fd14ad9d640 1 -- 172.21.15.31:0/3991845413 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd144144010 con 0x7fd14409acf0 2024-04-26T18:02:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.556+0000 7fd14ad9d640 1 --2- 172.21.15.31:0/3991845413 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14409acf0 0x7fd144143890 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0680a0 tx=0x7fd14c050de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.560+0000 7fd13b7fe640 1 -- 172.21.15.31:0/3991845413 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c07d450 con 0x7fd14409acf0 2024-04-26T18:02:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.560+0000 7fd13b7fe640 1 -- 172.21.15.31:0/3991845413 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd14c07daa0 con 0x7fd14409acf0 2024-04-26T18:02:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.560+0000 7fd13b7fe640 1 -- 172.21.15.31:0/3991845413 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c07ca90 con 0x7fd14409acf0 2024-04-26T18:02:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.560+0000 7fd1509af640 1 -- 172.21.15.31:0/3991845413 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd144144300 con 0x7fd14409acf0 2024-04-26T18:02:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.560+0000 7fd1509af640 1 -- 172.21.15.31:0/3991845413 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1441505a0 con 0x7fd14409acf0 2024-04-26T18:02:01.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:01 smithi031 ceph-mon[29164]: pgmap v168: 97 pgs: 97 active+clean; 2.8 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.3 MiB/s wr, 711 op/s 2024-04-26T18:02:01.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.569+0000 7fd1537a6740 1 Processor -- start 2024-04-26T18:02:01.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.571+0000 7fd1537a6740 1 -- start start 2024-04-26T18:02:01.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.571+0000 7fd1537a6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 0x558406e7c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.571+0000 7fd1537a6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e7c9f0 0x558406e86d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.571+0000 7fd1537a6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e896a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.571+0000 7fd1537a6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x558406de2460 con 0x558406e872b0 2024-04-26T18:02:01.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.571+0000 7fd1537a6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x558406d9c590 con 0x558406e7c9f0 2024-04-26T18:02:01.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.571+0000 7fd1537a6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558406dde420 con 0x558406e7c0d0 2024-04-26T18:02:01.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.571+0000 7fd13b7fe640 1 -- 172.21.15.31:0/3991845413 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd14c07dc50 con 0x7fd14409acf0 2024-04-26T18:02:01.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.572+0000 7fd14a59c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e7c9f0 0x558406e86d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.572+0000 7fd14a59c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e7c9f0 0x558406e86d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:47172/0 (socket says 172.21.15.31:47172) 2024-04-26T18:02:01.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.572+0000 7fd14a59c640 1 -- 172.21.15.31:0/3944017314 learned_addr learned my addr 172.21.15.31:0/3944017314 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:01.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.572+0000 7fd149d9b640 1 --2- 172.21.15.31:0/3944017314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e896a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.572+0000 7fd149d9b640 1 --2- 172.21.15.31:0/3944017314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 0x558406e7c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.576+0000 7fd14a59c640 1 -- 172.21.15.31:0/3944017314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 msgr2=0x558406e7c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.576+0000 7fd13b7fe640 1 --2- 172.21.15.31:0/3991845413 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fd1340a3f20 0x7fd1340a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.577+0000 7fd13b7fe640 1 -- 172.21.15.31:0/3991845413 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fd14c1341a0 con 0x7fd14409acf0 2024-04-26T18:02:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.578+0000 7fd14a59c640 1 --2- 172.21.15.31:0/3944017314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 0x558406e7c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.578+0000 7fd14a59c640 1 -- 172.21.15.31:0/3944017314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 msgr2=0x558406e896a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.578+0000 7fd14a59c640 1 --2- 172.21.15.31:0/3944017314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e896a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.578+0000 7fd14a59c640 1 -- 172.21.15.31:0/3944017314 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558406da90d0 con 0x558406e7c9f0 2024-04-26T18:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.578+0000 7fd149d9b640 1 --2- 172.21.15.31:0/3944017314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e896a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.579+0000 7fd14ad9d640 1 --2- 172.21.15.31:0/3991845413 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fd1340a3f20 0x7fd1340a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.580+0000 7fd14a59c640 1 --2- 172.21.15.31:0/3944017314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e7c9f0 0x558406e86d70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd13c02b7e0 tx=0x7fd13c02bca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21683a2ffde80771 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.582+0000 7fd138ff9640 1 -- 172.21.15.31:0/3944017314 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c037020 con 0x558406e7c9f0 2024-04-26T18:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.582+0000 7fd138ff9640 1 -- 172.21.15.31:0/3944017314 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd13c02dd90 con 0x558406e7c9f0 2024-04-26T18:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.582+0000 7fd138ff9640 1 -- 172.21.15.31:0/3944017314 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c032390 con 0x558406e7c9f0 2024-04-26T18:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.584+0000 7fd1537a6740 1 -- 172.21.15.31:0/3944017314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e7c9f0 msgr2=0x558406e86d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.584+0000 7fd1537a6740 1 --2- 172.21.15.31:0/3944017314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e7c9f0 0x558406e86d70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd13c02b7e0 tx=0x7fd13c02bca0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.584+0000 7fd1537a6740 1 -- 172.21.15.31:0/3944017314 shutdown_connections 2024-04-26T18:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.584+0000 7fd1537a6740 1 --2- 172.21.15.31:0/3944017314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e896a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.584+0000 7fd1537a6740 1 --2- 172.21.15.31:0/3944017314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e7c9f0 0x558406e86d70 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd13c02b7e0 tx=0x7fd13c02bca0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.584+0000 7fd1537a6740 1 --2- 172.21.15.31:0/3944017314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 0x558406e7c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.584+0000 7fd1537a6740 1 -- 172.21.15.31:0/3944017314 >> 172.21.15.31:0/3944017314 conn(0x558406e785e0 msgr2=0x558406e789c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.587+0000 7fd1537a6740 1 -- 172.21.15.31:0/3944017314 shutdown_connections 2024-04-26T18:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.587+0000 7fd14ad9d640 1 --2- 172.21.15.31:0/3991845413 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fd1340a3f20 0x7fd1340a63e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0fcc10 tx=0x7fd14c076040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.589+0000 7fd1537a6740 1 -- 172.21.15.31:0/3944017314 wait complete. 2024-04-26T18:02:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.590+0000 7fd1537a6740 1 -- start start 2024-04-26T18:02:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.590+0000 7fd1537a6740 1 Processor -- start 2024-04-26T18:02:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.594+0000 7fd1537a6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 0x558406e84460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.594+0000 7fd1537a6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e849a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.594+0000 7fd1537a6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e84ee0 0x558406e932d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.594+0000 7fd1537a6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x558406e89ee0 con 0x558406e872b0 2024-04-26T18:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.594+0000 7fd1537a6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x558406d69b60 con 0x558406e84ee0 2024-04-26T18:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.594+0000 7fd1537a6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558406e8a060 con 0x558406e7c0d0 2024-04-26T18:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.594+0000 7fd1537a6740 10 client.0 _ll_register_callbacks cb 0x558406d8e2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.596+0000 7fd149d9b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e84ee0 0x558406e932d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.596+0000 7fd149d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 0x558406e84460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.596+0000 7fd149d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 0x558406e84460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:36376/0 (socket says 172.21.15.31:36376) 2024-04-26T18:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.596+0000 7fd149d9b640 1 -- 172.21.15.31:0/2048011574 learned_addr learned my addr 172.21.15.31:0/2048011574 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.596+0000 7fd14a59c640 1 --2- 172.21.15.31:0/2048011574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e849a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.596+0000 7fd149d9b640 1 -- 172.21.15.31:0/2048011574 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e84ee0 msgr2=0x558406e932d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.596+0000 7fd149d9b640 1 --2- 172.21.15.31:0/2048011574 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e84ee0 0x558406e932d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.596+0000 7fd149d9b640 1 -- 172.21.15.31:0/2048011574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 msgr2=0x558406e849a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.596+0000 7fd149d9b640 1 --2- 172.21.15.31:0/2048011574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e849a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.596+0000 7fd149d9b640 1 -- 172.21.15.31:0/2048011574 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558406af4210 con 0x558406e7c0d0 2024-04-26T18:02:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.597+0000 7fd149d9b640 1 --2- 172.21.15.31:0/2048011574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 0x558406e84460 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd140009230 tx=0x7fd1400036d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2a6d8e7053e3cc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.597+0000 7fd131ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T18:02:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.598+0000 7fd131ffb640 1 -- 172.21.15.31:0/2048011574 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140046050 con 0x558406e7c0d0 2024-04-26T18:02:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.598+0000 7fd131ffb640 1 -- 172.21.15.31:0/2048011574 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fd14003c440 con 0x558406e7c0d0 2024-04-26T18:02:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.599+0000 7fd1537a6740 1 -- 172.21.15.31:0/2048011574 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd13c02dd90 con 0x558406e7c0d0 2024-04-26T18:02:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.600+0000 7fd131ffb640 1 -- 172.21.15.31:0/2048011574 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd140049040 con 0x558406e7c0d0 2024-04-26T18:02:01.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.603+0000 7fd1537a6740 10 client.25513 fetch_fsmap learned FSMap version 32 2024-04-26T18:02:01.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.603+0000 7fd1537a6740 1 -- 172.21.15.31:0/2048011574 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558406bc0620 con 0x558406e7c0d0 2024-04-26T18:02:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.608+0000 7fd131ffb640 1 -- 172.21.15.31:0/2048011574 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd13c02dd90 con 0x558406e7c0d0 2024-04-26T18:02:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.608+0000 7fd131ffb640 1 -- 172.21.15.31:0/2048011574 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fd14003c8e0 con 0x558406e7c0d0 2024-04-26T18:02:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.611+0000 7fd1537a6740 10 client.25513 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:02:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.611+0000 7fd1537a6740 10 client.25513 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:02:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.611+0000 7fd1537a6740 20 client.25513 populate_metadata read hostname 'smithi031' 2024-04-26T18:02:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.611+0000 7fd1537a6740 1 --2- 172.21.15.31:0/2048011574 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x558406e979e0 0x558406e99e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.611+0000 7fd1537a6740 4 client.25513 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:02:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.611+0000 7fd1537a6740 1 -- 172.21.15.31:0/2048011574 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558406df0070 con 0x558406e979e0 2024-04-26T18:02:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.613+0000 7fd14a59c640 1 --2- 172.21.15.31:0/2048011574 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x558406e979e0 0x558406e99e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.622+0000 7fd14a59c640 1 --2- 172.21.15.31:0/2048011574 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x558406e979e0 0x558406e99e10 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ab75b4cacb938ba server_cookie=57179f27434697da in_seq=0 out_seq=0 2024-04-26T18:02:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.623+0000 7fd131ffb640 10 client.25513 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.624+0000 7fd131ffb640 1 -- 172.21.15.31:0/2048011574 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558406df0070 con 0x558406e979e0 2024-04-26T18:02:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.624+0000 7fd131ffb640 10 client.25513 handle_command_reply: tid=0 2024-04-26T18:02:01.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.624+0000 7fd1537a6740 1 client.25513 shutdown 2024-04-26T18:02:01.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.630+0000 7fd1537a6740 1 -- 172.21.15.31:0/2048011574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 msgr2=0x558406e84460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:01.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.630+0000 7fd1537a6740 1 --2- 172.21.15.31:0/2048011574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 0x558406e84460 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd140009230 tx=0x7fd1400036d0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.632+0000 7fd1537a6740 1 -- 172.21.15.31:0/2048011574 shutdown_connections 2024-04-26T18:02:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.632+0000 7fd1537a6740 1 --2- 172.21.15.31:0/2048011574 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x558406e979e0 0x558406e99e10 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.632+0000 7fd1537a6740 1 --2- 172.21.15.31:0/2048011574 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e84ee0 0x558406e932d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.632+0000 7fd1537a6740 1 --2- 172.21.15.31:0/2048011574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e849a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.632+0000 7fd1537a6740 1 --2- 172.21.15.31:0/2048011574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 0x558406e84460 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.632+0000 7fd1537a6740 1 -- 172.21.15.31:0/2048011574 >> 172.21.15.31:0/2048011574 conn(0x558406e785e0 msgr2=0x558406e7e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.632+0000 7fd131ffb640 0 client.25513 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.632+0000 7fd1537a6740 1 -- 172.21.15.31:0/2048011574 shutdown_connections 2024-04-26T18:02:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.633+0000 7fd1537a6740 1 -- 172.21.15.31:0/2048011574 wait complete. 2024-04-26T18:02:01.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:01 smithi097 ceph-mon[32419]: pgmap v168: 97 pgs: 97 active+clean; 2.8 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.3 MiB/s wr, 711 op/s 2024-04-26T18:02:01.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.634+0000 7fd1537a6740 20 client.25513 trim_cache size 0 max 16384 2024-04-26T18:02:01.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.670+0000 7fd1537a6740 1 Processor -- start 2024-04-26T18:02:01.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.673+0000 7fd1537a6740 1 -- start start 2024-04-26T18:02:01.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.673+0000 7fd1537a6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 0x558406e999f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.673+0000 7fd1537a6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e84ee0 0x558406e9f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.674+0000 7fd1537a6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e9fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.674+0000 7fd1537a6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c011090 con 0x558406e872b0 2024-04-26T18:02:01.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.674+0000 7fd1537a6740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c00e900 con 0x558406e84ee0 2024-04-26T18:02:01.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.674+0000 7fd1537a6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c00c750 con 0x558406e7c0d0 2024-04-26T18:02:01.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.674+0000 7fd149d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e9fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.674+0000 7fd149d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e9fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53316/0 (socket says 172.21.15.31:53316) 2024-04-26T18:02:01.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.674+0000 7fd149d9b640 1 -- 172.21.15.31:0/1402205097 learned_addr learned my addr 172.21.15.31:0/1402205097 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:01.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.674+0000 7fd149d9b640 1 --2- 172.21.15.31:0/1402205097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 0x558406e999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.677+0000 7fd149d9b640 1 -- 172.21.15.31:0/1402205097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 msgr2=0x558406e999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:01.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.677+0000 7fd149d9b640 1 --2- 172.21.15.31:0/1402205097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 0x558406e999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.677+0000 7fd149d9b640 1 -- 172.21.15.31:0/1402205097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e84ee0 msgr2=0x558406e9f720 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T18:02:01.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.677+0000 7fd149d9b640 1 --2- 172.21.15.31:0/1402205097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e84ee0 0x558406e9f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.677+0000 7fd149d9b640 1 -- 172.21.15.31:0/1402205097 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558406e84cb0 con 0x558406e872b0 2024-04-26T18:02:01.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.679+0000 7fd149d9b640 1 --2- 172.21.15.31:0/1402205097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e9fc60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd14003ce50 tx=0x7fd140009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e472373dfb4c6e63 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:01.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.680+0000 7fd133fff640 1 -- 172.21.15.31:0/1402205097 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140046070 con 0x558406e872b0 2024-04-26T18:02:01.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.680+0000 7fd133fff640 1 -- 172.21.15.31:0/1402205097 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd14005f020 con 0x558406e872b0 2024-04-26T18:02:01.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.680+0000 7fd133fff640 1 -- 172.21.15.31:0/1402205097 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14004f450 con 0x558406e872b0 2024-04-26T18:02:01.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.681+0000 7fd1537a6740 1 -- 172.21.15.31:0/1402205097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 msgr2=0x558406e9fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:01.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.681+0000 7fd1537a6740 1 --2- 172.21.15.31:0/1402205097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e9fc60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd14003ce50 tx=0x7fd140009260 comp rx=0 tx=0).stop 2024-04-26T18:02:01.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.684+0000 7fd1537a6740 1 -- 172.21.15.31:0/1402205097 shutdown_connections 2024-04-26T18:02:01.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.684+0000 7fd1537a6740 1 --2- 172.21.15.31:0/1402205097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e872b0 0x558406e9fc60 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.684+0000 7fd1537a6740 1 --2- 172.21.15.31:0/1402205097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e84ee0 0x558406e9f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.684+0000 7fd1537a6740 1 --2- 172.21.15.31:0/1402205097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e7c0d0 0x558406e999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.684+0000 7fd1537a6740 1 -- 172.21.15.31:0/1402205097 >> 172.21.15.31:0/1402205097 conn(0x558406e785e0 msgr2=0x558406e82d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.687+0000 7fd1537a6740 1 -- 172.21.15.31:0/1402205097 shutdown_connections 2024-04-26T18:02:01.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.687+0000 7fd1537a6740 1 -- 172.21.15.31:0/1402205097 wait complete. 2024-04-26T18:02:01.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.688+0000 7fd1537a6740 1 -- start start 2024-04-26T18:02:01.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.688+0000 7fd1537a6740 1 Processor -- start 2024-04-26T18:02:01.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.689+0000 7fd1537a6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e7c0d0 0x558406e8ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.689+0000 7fd1537a6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e84ee0 0x558406e8b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.690+0000 7fd1537a6740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e872b0 0x558406e8b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.690+0000 7fd1537a6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c00cdc0 con 0x558406e7c0d0 2024-04-26T18:02:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.690+0000 7fd14a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e84ee0 0x558406e8b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.690+0000 7fd14a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e84ee0 0x558406e8b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:36392/0 (socket says 172.21.15.31:36392) 2024-04-26T18:02:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.690+0000 7fd14a59c640 1 -- 172.21.15.31:0/3797015206 learned_addr learned my addr 172.21.15.31:0/3797015206 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.690+0000 7fd149d9b640 1 --2- 172.21.15.31:0/3797015206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e872b0 0x558406e8b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.690+0000 7fd1537a6740 1 -- 172.21.15.31:0/3797015206 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c00e230 con 0x558406e872b0 2024-04-26T18:02:01.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.690+0000 7fd1537a6740 1 -- 172.21.15.31:0/3797015206 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c00d490 con 0x558406e84ee0 2024-04-26T18:02:01.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.690+0000 7fd1537a6740 10 client.0 _ll_register_callbacks cb 0x7fd140004960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:02:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.694+0000 7fd149d9b640 1 -- 172.21.15.31:0/3797015206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e84ee0 msgr2=0x558406e8b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.694+0000 7fd149d9b640 1 --2- 172.21.15.31:0/3797015206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e84ee0 0x558406e8b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.694+0000 7fd149d9b640 1 -- 172.21.15.31:0/3797015206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e7c0d0 msgr2=0x558406e8ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:01.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.694+0000 7fd149d9b640 1 --2- 172.21.15.31:0/3797015206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e7c0d0 0x558406e8ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:01.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.694+0000 7fd149d9b640 1 -- 172.21.15.31:0/3797015206 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558406efcf90 con 0x558406e872b0 2024-04-26T18:02:01.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.695+0000 7fd149d9b640 1 --2- 172.21.15.31:0/3797015206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e872b0 0x558406e8b790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd14003ce50 tx=0x7fd14003c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64bb37c4eba95bde server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:01.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.695+0000 7fd131ffb640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T18:02:01.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.697+0000 7fd131ffb640 1 -- 172.21.15.31:0/3797015206 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140046040 con 0x558406e872b0 2024-04-26T18:02:01.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.697+0000 7fd131ffb640 1 -- 172.21.15.31:0/3797015206 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fd140064460 con 0x558406e872b0 2024-04-26T18:02:01.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.698+0000 7fd1537a6740 1 -- 172.21.15.31:0/3797015206 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd14005f020 con 0x558406e872b0 2024-04-26T18:02:01.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.700+0000 7fd131ffb640 1 -- 172.21.15.31:0/3797015206 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd140005d10 con 0x558406e872b0 2024-04-26T18:02:01.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.700+0000 7fd1537a6740 10 client.25607 fetch_fsmap learned FSMap version 32 2024-04-26T18:02:01.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.700+0000 7fd1537a6740 1 -- 172.21.15.31:0/3797015206 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558406efe240 con 0x558406e872b0 2024-04-26T18:02:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.702+0000 7fd131ffb640 1 -- 172.21.15.31:0/3797015206 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd140064b10 con 0x558406e872b0 2024-04-26T18:02:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.703+0000 7fd131ffb640 1 -- 172.21.15.31:0/3797015206 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fd14004f8e0 con 0x558406e872b0 2024-04-26T18:02:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.705+0000 7fd1537a6740 10 client.25607 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:02:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.705+0000 7fd1537a6740 10 client.25607 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:02:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.705+0000 7fd1537a6740 20 client.25607 populate_metadata read hostname 'smithi031' 2024-04-26T18:02:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.706+0000 7fd1537a6740 1 --2- 172.21.15.31:0/3797015206 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x558406efe870 0x558406f00c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.706+0000 7fd1537a6740 4 client.25607 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:02:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.706+0000 7fd1537a6740 1 -- 172.21.15.31:0/3797015206 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fd14005f020 con 0x558406efe870 2024-04-26T18:02:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.706+0000 7fd14a59c640 1 --2- 172.21.15.31:0/3797015206 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x558406efe870 0x558406f00c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.709+0000 7fd131ffb640 10 client.25607 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:01.709+0000 7fd14a59c640 1 --2- 172.21.15.31:0/3797015206 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x558406efe870 0x558406f00c90 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea2ab9190d150e8f server_cookie=a6efcedcf82f181 in_seq=0 out_seq=0 2024-04-26T18:02:02.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.446+0000 7fd131ffb640 1 -- 172.21.15.31:0/3797015206 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fd14005f020 con 0x558406efe870 2024-04-26T18:02:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.447+0000 7fd131ffb640 10 client.25607 handle_command_reply: tid=0 2024-04-26T18:02:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.447+0000 7fd1537a6740 1 client.25607 shutdown 2024-04-26T18:02:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.447+0000 7fd1537a6740 1 -- 172.21.15.31:0/3797015206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e872b0 msgr2=0x558406e8b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.447+0000 7fd1537a6740 1 --2- 172.21.15.31:0/3797015206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e872b0 0x558406e8b790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd14003ce50 tx=0x7fd14003c7b0 comp rx=0 tx=0).stop 2024-04-26T18:02:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.447+0000 7fd1537a6740 1 -- 172.21.15.31:0/3797015206 shutdown_connections 2024-04-26T18:02:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.447+0000 7fd1537a6740 1 --2- 172.21.15.31:0/3797015206 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x558406efe870 0x558406f00c90 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.448+0000 7fd131ffb640 0 client.25607 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.448+0000 7fd1537a6740 1 --2- 172.21.15.31:0/3797015206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558406e872b0 0x558406e8b790 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.448+0000 7fd1537a6740 1 --2- 172.21.15.31:0/3797015206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558406e84ee0 0x558406e8b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.448+0000 7fd1537a6740 1 --2- 172.21.15.31:0/3797015206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558406e7c0d0 0x558406e8ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.448+0000 7fd1537a6740 1 -- 172.21.15.31:0/3797015206 >> 172.21.15.31:0/3797015206 conn(0x558406e785e0 msgr2=0x558406e7dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.448+0000 7fd1537a6740 1 -- 172.21.15.31:0/3797015206 shutdown_connections 2024-04-26T18:02:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.448+0000 7fd1537a6740 1 -- 172.21.15.31:0/3797015206 wait complete. 2024-04-26T18:02:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.448+0000 7fd1537a6740 20 client.25607 trim_cache size 0 max 16384 2024-04-26T18:02:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:02:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:02:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:02:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:02:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.449+0000 7fd1509af640 1 -- 172.21.15.31:0/3991845413 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fd1340a3f20 msgr2=0x7fd1340a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.449+0000 7fd1509af640 1 --2- 172.21.15.31:0/3991845413 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fd1340a3f20 0x7fd1340a63e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0fcc10 tx=0x7fd14c076040 comp rx=0 tx=0).stop 2024-04-26T18:02:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.449+0000 7fd1509af640 1 -- 172.21.15.31:0/3991845413 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14409acf0 msgr2=0x7fd144143890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.449+0000 7fd1509af640 1 --2- 172.21.15.31:0/3991845413 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14409acf0 0x7fd144143890 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0680a0 tx=0x7fd14c050de0 comp rx=0 tx=0).stop 2024-04-26T18:02:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.449+0000 7fd1509af640 1 -- 172.21.15.31:0/3991845413 shutdown_connections 2024-04-26T18:02:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.449+0000 7fd1509af640 1 --2- 172.21.15.31:0/3991845413 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd14409acf0 0x7fd144143890 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.449+0000 7fd1509af640 1 --2- 172.21.15.31:0/3991845413 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fd1340a3f20 0x7fd1340a63e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.450+0000 7fd1509af640 1 --2- 172.21.15.31:0/3991845413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1440986e0 0x7fd144149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.450+0000 7fd1509af640 1 --2- 172.21.15.31:0/3991845413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd144004c80 0x7fd1441496e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.450+0000 7fd1509af640 1 -- 172.21.15.31:0/3991845413 >> 172.21.15.31:0/3991845413 conn(0x7fd14400a740 msgr2=0x7fd1440922e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.450+0000 7fd1509af640 1 -- 172.21.15.31:0/3991845413 shutdown_connections 2024-04-26T18:02:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:02.450+0000 7fd1509af640 1 -- 172.21.15.31:0/3991845413 wait complete. 2024-04-26T18:02:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 280.0 s, 2220.8 tps, lat 45.139 ms stddev 92.919 2024-04-26T18:02:03.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:03 smithi186 ceph-mon[32797]: pgmap v169: 97 pgs: 97 active+clean; 2.9 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 21 MiB/s wr, 1.12k op/s 2024-04-26T18:02:03.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:03 smithi031 ceph-mon[29164]: pgmap v169: 97 pgs: 97 active+clean; 2.9 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 21 MiB/s wr, 1.12k op/s 2024-04-26T18:02:03.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:03 smithi097 ceph-mon[32419]: pgmap v169: 97 pgs: 97 active+clean; 2.9 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 21 MiB/s wr, 1.12k op/s 2024-04-26T18:02:05.435 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:05 smithi097 ceph-mon[32419]: pgmap v170: 97 pgs: 97 active+clean; 2.9 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 16 MiB/s wr, 705 op/s 2024-04-26T18:02:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:05 smithi186 ceph-mon[32797]: pgmap v170: 97 pgs: 97 active+clean; 2.9 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 16 MiB/s wr, 705 op/s 2024-04-26T18:02:05.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:05 smithi031 ceph-mon[29164]: pgmap v170: 97 pgs: 97 active+clean; 2.9 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 16 MiB/s wr, 705 op/s 2024-04-26T18:02:07.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:07 smithi186 ceph-mon[32797]: pgmap v171: 97 pgs: 97 active+clean; 3.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 43 MiB/s wr, 1.89k op/s 2024-04-26T18:02:07.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:07 smithi031 ceph-mon[29164]: pgmap v171: 97 pgs: 97 active+clean; 3.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 43 MiB/s wr, 1.89k op/s 2024-04-26T18:02:07.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:07 smithi097 ceph-mon[32419]: pgmap v171: 97 pgs: 97 active+clean; 3.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 43 MiB/s wr, 1.89k op/s 2024-04-26T18:02:08.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 285.0 s, 3640.0 tps, lat 27.298 ms stddev 32.736 2024-04-26T18:02:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:09 smithi186 ceph-mon[32797]: pgmap v172: 97 pgs: 97 active+clean; 3.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 51 MiB/s wr, 1.92k op/s 2024-04-26T18:02:09.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:09 smithi031 ceph-mon[29164]: pgmap v172: 97 pgs: 97 active+clean; 3.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 51 MiB/s wr, 1.92k op/s 2024-04-26T18:02:09.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:09 smithi097 ceph-mon[32419]: pgmap v172: 97 pgs: 97 active+clean; 3.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 51 MiB/s wr, 1.92k op/s 2024-04-26T18:02:11.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:11 smithi186 ceph-mon[32797]: pgmap v173: 97 pgs: 97 active+clean; 3.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 50 MiB/s wr, 1.88k op/s 2024-04-26T18:02:11.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:11 smithi031 ceph-mon[29164]: pgmap v173: 97 pgs: 97 active+clean; 3.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 50 MiB/s wr, 1.88k op/s 2024-04-26T18:02:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:11 smithi097 ceph-mon[32419]: pgmap v173: 97 pgs: 97 active+clean; 3.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 50 MiB/s wr, 1.88k op/s 2024-04-26T18:02:13.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 290.0 s, 3844.8 tps, lat 26.161 ms stddev 31.620 2024-04-26T18:02:13.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:13 smithi186 ceph-mon[32797]: pgmap v174: 97 pgs: 97 active+clean; 3.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 69 MiB/s wr, 2.22k op/s 2024-04-26T18:02:13.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:13 smithi031 ceph-mon[29164]: pgmap v174: 97 pgs: 97 active+clean; 3.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 69 MiB/s wr, 2.22k op/s 2024-04-26T18:02:13.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:13 smithi097 ceph-mon[32419]: pgmap v174: 97 pgs: 97 active+clean; 3.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 69 MiB/s wr, 2.22k op/s 2024-04-26T18:02:15.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:15 smithi186 ceph-mon[32797]: pgmap v175: 97 pgs: 97 active+clean; 3.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 56 MiB/s wr, 1.81k op/s 2024-04-26T18:02:15.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:15 smithi031 ceph-mon[29164]: pgmap v175: 97 pgs: 97 active+clean; 3.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 56 MiB/s wr, 1.81k op/s 2024-04-26T18:02:15.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.627+0000 7f324cce4640 1 -- 172.21.15.31:0/547283963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248103100 msgr2=0x7f324810f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:15.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.627+0000 7f324cce4640 1 --2- 172.21.15.31:0/547283963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248103100 0x7f324810f610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3240009ca0 tx=0x7f324002a8b0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.629+0000 7f324cce4640 1 -- 172.21.15.31:0/547283963 shutdown_connections 2024-04-26T18:02:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.629+0000 7f324cce4640 1 --2- 172.21.15.31:0/547283963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248103100 0x7f324810f610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.629+0000 7f324cce4640 1 --2- 172.21.15.31:0/547283963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3248102740 0x7f3248102bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.629+0000 7f324cce4640 1 --2- 172.21.15.31:0/547283963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3248108750 0x7f3248108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.629+0000 7f324cce4640 1 -- 172.21.15.31:0/547283963 >> 172.21.15.31:0/547283963 conn(0x7f32480fe270 msgr2=0x7f3248100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.632+0000 7f324cce4640 1 -- 172.21.15.31:0/547283963 shutdown_connections 2024-04-26T18:02:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.632+0000 7f324cce4640 1 -- 172.21.15.31:0/547283963 wait complete. 2024-04-26T18:02:15.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:15 smithi097 ceph-mon[32419]: pgmap v175: 97 pgs: 97 active+clean; 3.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 56 MiB/s wr, 1.81k op/s 2024-04-26T18:02:15.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.636+0000 7f324cce4640 1 Processor -- start 2024-04-26T18:02:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.637+0000 7f324cce4640 1 -- start start 2024-04-26T18:02:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.638+0000 7f324cce4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3248102740 0x7f324810f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.638+0000 7f324cce4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3248103100 0x7f324810c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.639+0000 7f324cce4640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248108750 0x7f324810cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.639+0000 7f324cce4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f32480766d0 con 0x7f3248103100 2024-04-26T18:02:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.639+0000 7f324cce4640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3248076550 con 0x7f3248108750 2024-04-26T18:02:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.639+0000 7f32477fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3248102740 0x7f324810f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.639+0000 7f32477fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3248102740 0x7f324810f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:49698/0 (socket says 172.21.15.31:49698) 2024-04-26T18:02:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.639+0000 7f32477fe640 1 -- 172.21.15.31:0/798693097 learned_addr learned my addr 172.21.15.31:0/798693097 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.639+0000 7f3247fff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248108750 0x7f324810cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.639+0000 7f324cce4640 1 -- 172.21.15.31:0/798693097 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3248076850 con 0x7f3248102740 2024-04-26T18:02:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.641+0000 7f3247fff640 1 -- 172.21.15.31:0/798693097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3248102740 msgr2=0x7f324810f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.641+0000 7f3247fff640 1 --2- 172.21.15.31:0/798693097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3248102740 0x7f324810f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.641+0000 7f3247fff640 1 -- 172.21.15.31:0/798693097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3248103100 msgr2=0x7f324810c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.641+0000 7f3247fff640 1 --2- 172.21.15.31:0/798693097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3248103100 0x7f324810c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.641+0000 7f3247fff640 1 -- 172.21.15.31:0/798693097 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32481ab880 con 0x7f3248108750 2024-04-26T18:02:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.646+0000 7f3247fff640 1 --2- 172.21.15.31:0/798693097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248108750 0x7f324810cda0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3240005e30 tx=0x7f3240003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.647+0000 7f3244ff9640 1 -- 172.21.15.31:0/798693097 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3240004240 con 0x7f3248108750 2024-04-26T18:02:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.647+0000 7f324cce4640 1 -- 172.21.15.31:0/798693097 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32481aba50 con 0x7f3248108750 2024-04-26T18:02:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.647+0000 7f324cce4640 1 -- 172.21.15.31:0/798693097 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32481abf70 con 0x7f3248108750 2024-04-26T18:02:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.648+0000 7f3244ff9640 1 -- 172.21.15.31:0/798693097 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3240040070 con 0x7f3248108750 2024-04-26T18:02:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.648+0000 7f3244ff9640 1 -- 172.21.15.31:0/798693097 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3240045c30 con 0x7f3248108750 2024-04-26T18:02:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.654+0000 7f3250c5d740 1 Processor -- start 2024-04-26T18:02:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.656+0000 7f3244ff9640 1 -- 172.21.15.31:0/798693097 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3240045470 con 0x7f3248108750 2024-04-26T18:02:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.656+0000 7f3250c5d740 1 -- start start 2024-04-26T18:02:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.657+0000 7f3250c5d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f20940c0 0x55a4f20944a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.657+0000 7f3250c5d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f20949e0 0x55a4f209ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.657+0000 7f3250c5d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209f2a0 0x55a4f20a1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.657+0000 7f3250c5d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a4f1ffa460 con 0x55a4f20940c0 2024-04-26T18:02:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.657+0000 7f3250c5d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55a4f1fb4590 con 0x55a4f209f2a0 2024-04-26T18:02:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.657+0000 7f3250c5d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a4f1ff6420 con 0x55a4f20949e0 2024-04-26T18:02:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.657+0000 7f32477fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f20949e0 0x55a4f209ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.657+0000 7f32477fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f20949e0 0x55a4f209ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:49708/0 (socket says 172.21.15.31:49708) 2024-04-26T18:02:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.657+0000 7f32477fe640 1 -- 172.21.15.31:0/2559773120 learned_addr learned my addr 172.21.15.31:0/2559773120 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.658+0000 7f3244ff9640 1 --2- 172.21.15.31:0/798693097 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f32300a3ee0 0x7f32300a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.658+0000 7f3244ff9640 1 -- 172.21.15.31:0/798693097 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f32400f1aa0 con 0x7f3248108750 2024-04-26T18:02:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.658+0000 7f32477fe640 1 -- 172.21.15.31:0/2559773120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209f2a0 msgr2=0x55a4f20a1690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T18:02:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.658+0000 7f32477fe640 1 --2- 172.21.15.31:0/2559773120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209f2a0 0x55a4f20a1690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.658+0000 7f32477fe640 1 -- 172.21.15.31:0/2559773120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f20940c0 msgr2=0x55a4f20944a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T18:02:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.658+0000 7f32477fe640 1 --2- 172.21.15.31:0/2559773120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f20940c0 0x55a4f20944a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.658+0000 7f32477fe640 1 -- 172.21.15.31:0/2559773120 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4f1fc10d0 con 0x55a4f20949e0 2024-04-26T18:02:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.659+0000 7f32477fe640 1 --2- 172.21.15.31:0/2559773120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f20949e0 0x55a4f209ed60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f323800bd40 tx=0x7f32380107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d288d9a0e301242 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.659+0000 7f3247fff640 1 --2- 172.21.15.31:0/798693097 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f32300a3ee0 0x7f32300a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.660+0000 7f322dffb640 1 -- 172.21.15.31:0/2559773120 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323801c680 con 0x55a4f20949e0 2024-04-26T18:02:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.660+0000 7f322dffb640 1 -- 172.21.15.31:0/2559773120 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32380118f0 con 0x55a4f20949e0 2024-04-26T18:02:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.661+0000 7f3250c5d740 1 -- 172.21.15.31:0/2559773120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f20949e0 msgr2=0x55a4f209ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.661+0000 7f3250c5d740 1 --2- 172.21.15.31:0/2559773120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f20949e0 0x55a4f209ed60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f323800bd40 tx=0x7f32380107a0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.661+0000 7f3250c5d740 1 -- 172.21.15.31:0/2559773120 shutdown_connections 2024-04-26T18:02:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.661+0000 7f3250c5d740 1 --2- 172.21.15.31:0/2559773120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209f2a0 0x55a4f20a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.661+0000 7f3250c5d740 1 --2- 172.21.15.31:0/2559773120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f20949e0 0x55a4f209ed60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.661+0000 7f3250c5d740 1 --2- 172.21.15.31:0/2559773120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f20940c0 0x55a4f20944a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.661+0000 7f3250c5d740 1 -- 172.21.15.31:0/2559773120 >> 172.21.15.31:0/2559773120 conn(0x55a4f20905d0 msgr2=0x55a4f20909b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.661+0000 7f3247fff640 1 --2- 172.21.15.31:0/798693097 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f32300a3ee0 0x7f32300a63a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f32400ba510 tx=0x7f324003e040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.664+0000 7f3250c5d740 1 -- 172.21.15.31:0/2559773120 shutdown_connections 2024-04-26T18:02:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.664+0000 7f3250c5d740 1 -- 172.21.15.31:0/2559773120 wait complete. 2024-04-26T18:02:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.665+0000 7f3250c5d740 1 -- start start 2024-04-26T18:02:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.665+0000 7f3250c5d740 1 Processor -- start 2024-04-26T18:02:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.665+0000 7f3250c5d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f20940c0 0x55a4f209ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.665+0000 7f3250c5d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209f2a0 0x55a4f209c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.665+0000 7f3250c5d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f209c670 0x55a4f209caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.665+0000 7f3250c5d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a4f20a1ed0 con 0x55a4f209c670 2024-04-26T18:02:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.665+0000 7f3250c5d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55a4f1f81b60 con 0x55a4f209f2a0 2024-04-26T18:02:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.665+0000 7f3250c5d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a4f20a2050 con 0x55a4f20940c0 2024-04-26T18:02:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.665+0000 7f3250c5d740 10 client.0 _ll_register_callbacks cb 0x55a4f1fa62b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:02:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.666+0000 7f3246ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f20940c0 0x55a4f209ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.666+0000 7f3246ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f20940c0 0x55a4f209ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:49720/0 (socket says 172.21.15.31:49720) 2024-04-26T18:02:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.666+0000 7f3246ffd640 1 -- 172.21.15.31:0/3853762911 learned_addr learned my addr 172.21.15.31:0/3853762911 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.666+0000 7f32477fe640 1 --2- 172.21.15.31:0/3853762911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209f2a0 0x55a4f209c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.667+0000 7f3246ffd640 1 -- 172.21.15.31:0/3853762911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209f2a0 msgr2=0x55a4f209c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.667+0000 7f3246ffd640 1 --2- 172.21.15.31:0/3853762911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209f2a0 0x55a4f209c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.667+0000 7f3246ffd640 1 -- 172.21.15.31:0/3853762911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f209c670 msgr2=0x55a4f209caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.667+0000 7f3246ffd640 1 --2- 172.21.15.31:0/3853762911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f209c670 0x55a4f209caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.667+0000 7f3246ffd640 1 -- 172.21.15.31:0/3853762911 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4f1d0c210 con 0x55a4f20940c0 2024-04-26T18:02:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.667+0000 7f3246ffd640 1 --2- 172.21.15.31:0/3853762911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f20940c0 0x55a4f209ed30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f323c00cea0 tx=0x7f323c00cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27ec40d2ef77623e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.667+0000 7f3226ffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T18:02:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.668+0000 7f3226ffd640 1 -- 172.21.15.31:0/3853762911 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c011680 con 0x55a4f20940c0 2024-04-26T18:02:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.668+0000 7f3250c5d740 1 -- 172.21.15.31:0/3853762911 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a4f2008070 con 0x55a4f20940c0 2024-04-26T18:02:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.668+0000 7f3226ffd640 1 -- 172.21.15.31:0/3853762911 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f323c011820 con 0x55a4f20940c0 2024-04-26T18:02:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.669+0000 7f3226ffd640 1 -- 172.21.15.31:0/3853762911 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f323c011a20 con 0x55a4f20940c0 2024-04-26T18:02:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.669+0000 7f3250c5d740 10 client.25528 fetch_fsmap learned FSMap version 32 2024-04-26T18:02:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.669+0000 7f3250c5d740 1 -- 172.21.15.31:0/3853762911 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a4f1dd8620 con 0x55a4f20940c0 2024-04-26T18:02:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.670+0000 7f3226ffd640 1 -- 172.21.15.31:0/3853762911 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a4f2008070 con 0x55a4f20940c0 2024-04-26T18:02:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.670+0000 7f3226ffd640 1 -- 172.21.15.31:0/3853762911 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f323c00f040 con 0x55a4f20940c0 2024-04-26T18:02:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.670+0000 7f3250c5d740 10 client.25528 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:02:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.670+0000 7f3250c5d740 10 client.25528 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:02:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.670+0000 7f3250c5d740 20 client.25528 populate_metadata read hostname 'smithi031' 2024-04-26T18:02:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.670+0000 7f3250c5d740 1 --2- 172.21.15.31:0/3853762911 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a4f20afc70 0x55a4f20b2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.670+0000 7f3250c5d740 4 client.25528 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:02:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.670+0000 7f3250c5d740 1 -- 172.21.15.31:0/3853762911 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4f2001ee0 con 0x55a4f20afc70 2024-04-26T18:02:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.671+0000 7f32477fe640 1 --2- 172.21.15.31:0/3853762911 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a4f20afc70 0x55a4f20b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.671+0000 7f32477fe640 1 --2- 172.21.15.31:0/3853762911 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a4f20afc70 0x55a4f20b2110 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfc9f090867d32f5 server_cookie=88434de1d565c in_seq=0 out_seq=0 2024-04-26T18:02:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.671+0000 7f3226ffd640 10 client.25528 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.672+0000 7f3226ffd640 1 -- 172.21.15.31:0/3853762911 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a4f2001ee0 con 0x55a4f20afc70 2024-04-26T18:02:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.672+0000 7f3226ffd640 10 client.25528 handle_command_reply: tid=0 2024-04-26T18:02:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.673+0000 7f3250c5d740 1 client.25528 shutdown 2024-04-26T18:02:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.673+0000 7f3250c5d740 1 -- 172.21.15.31:0/3853762911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f20940c0 msgr2=0x55a4f209ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.673+0000 7f3250c5d740 1 --2- 172.21.15.31:0/3853762911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f20940c0 0x55a4f209ed30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f323c00cea0 tx=0x7f323c00cf90 comp rx=0 tx=0).stop 2024-04-26T18:02:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.673+0000 7f3250c5d740 1 -- 172.21.15.31:0/3853762911 shutdown_connections 2024-04-26T18:02:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.673+0000 7f3250c5d740 1 --2- 172.21.15.31:0/3853762911 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a4f20afc70 0x55a4f20b2110 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.673+0000 7f3250c5d740 1 --2- 172.21.15.31:0/3853762911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f209c670 0x55a4f209caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.674+0000 7f3250c5d740 1 --2- 172.21.15.31:0/3853762911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209f2a0 0x55a4f209c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.674+0000 7f3250c5d740 1 --2- 172.21.15.31:0/3853762911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f20940c0 0x55a4f209ed30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.674+0000 7f3250c5d740 1 -- 172.21.15.31:0/3853762911 >> 172.21.15.31:0/3853762911 conn(0x55a4f20905d0 msgr2=0x55a4f208eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.674+0000 7f3226ffd640 0 client.25528 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:15.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.674+0000 7f3250c5d740 1 -- 172.21.15.31:0/3853762911 shutdown_connections 2024-04-26T18:02:15.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.675+0000 7f3250c5d740 1 -- 172.21.15.31:0/3853762911 wait complete. 2024-04-26T18:02:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.676+0000 7f3250c5d740 20 client.25528 trim_cache size 0 max 16384 2024-04-26T18:02:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.707+0000 7f3250c5d740 1 Processor -- start 2024-04-26T18:02:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.708+0000 7f3250c5d740 1 -- start start 2024-04-26T18:02:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.708+0000 7f3250c5d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f20940c0 0x55a4f209e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.709+0000 7f3250c5d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209c670 0x55a4f20b79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.709+0000 7f3250c5d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f209f2a0 0x55a4f20b7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.709+0000 7f32477fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209c670 0x55a4f20b79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.709+0000 7f32477fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209c670 0x55a4f20b79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:52628/0 (socket says 172.21.15.31:52628) 2024-04-26T18:02:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.709+0000 7f32477fe640 1 -- 172.21.15.31:0/15038563 learned_addr learned my addr 172.21.15.31:0/15038563 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.709+0000 7f3246ffd640 1 --2- 172.21.15.31:0/15038563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f209f2a0 0x55a4f20b7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.709+0000 7f3250c5d740 1 -- 172.21.15.31:0/15038563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3234011090 con 0x55a4f20940c0 2024-04-26T18:02:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.709+0000 7f3250c5d740 1 -- 172.21.15.31:0/15038563 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f323400e900 con 0x55a4f209c670 2024-04-26T18:02:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.709+0000 7f3250c5d740 1 -- 172.21.15.31:0/15038563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f323400c750 con 0x55a4f209f2a0 2024-04-26T18:02:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.709+0000 7f3246ffd640 1 --2- 172.21.15.31:0/15038563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f20940c0 0x55a4f209e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.710+0000 7f3246ffd640 1 -- 172.21.15.31:0/15038563 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209c670 msgr2=0x55a4f20b79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.710+0000 7f3246ffd640 1 --2- 172.21.15.31:0/15038563 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209c670 0x55a4f20b79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.710+0000 7f3246ffd640 1 -- 172.21.15.31:0/15038563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f20940c0 msgr2=0x55a4f209e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.710+0000 7f3246ffd640 1 --2- 172.21.15.31:0/15038563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f20940c0 0x55a4f209e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.710+0000 7f3246ffd640 1 -- 172.21.15.31:0/15038563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4f209a590 con 0x55a4f209f2a0 2024-04-26T18:02:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.710+0000 7f3246ffd640 1 --2- 172.21.15.31:0/15038563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f209f2a0 0x55a4f20b7f20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f323c00cbc0 tx=0x7f323c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27f3090e69e3b5f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.711+0000 7f322d7fa640 1 -- 172.21.15.31:0/15038563 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c0224a0 con 0x55a4f209f2a0 2024-04-26T18:02:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.711+0000 7f322d7fa640 1 -- 172.21.15.31:0/15038563 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f323c011680 con 0x55a4f209f2a0 2024-04-26T18:02:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.711+0000 7f322d7fa640 1 -- 172.21.15.31:0/15038563 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c023680 con 0x55a4f209f2a0 2024-04-26T18:02:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.712+0000 7f3250c5d740 1 -- 172.21.15.31:0/15038563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f209f2a0 msgr2=0x55a4f20b7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.712+0000 7f3250c5d740 1 --2- 172.21.15.31:0/15038563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f209f2a0 0x55a4f20b7f20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f323c00cbc0 tx=0x7f323c000bc0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.712+0000 7f3250c5d740 1 -- 172.21.15.31:0/15038563 shutdown_connections 2024-04-26T18:02:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.712+0000 7f3250c5d740 1 --2- 172.21.15.31:0/15038563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f209f2a0 0x55a4f20b7f20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.712+0000 7f3250c5d740 1 --2- 172.21.15.31:0/15038563 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f209c670 0x55a4f20b79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.712+0000 7f3250c5d740 1 --2- 172.21.15.31:0/15038563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f20940c0 0x55a4f209e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.712+0000 7f3250c5d740 1 -- 172.21.15.31:0/15038563 >> 172.21.15.31:0/15038563 conn(0x55a4f20905d0 msgr2=0x55a4f2097240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.713+0000 7f3250c5d740 1 -- 172.21.15.31:0/15038563 shutdown_connections 2024-04-26T18:02:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.713+0000 7f3250c5d740 1 -- 172.21.15.31:0/15038563 wait complete. 2024-04-26T18:02:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.713+0000 7f3250c5d740 1 -- start start 2024-04-26T18:02:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.714+0000 7f3250c5d740 1 Processor -- start 2024-04-26T18:02:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.714+0000 7f3250c5d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f20940c0 0x55a4f20a7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f3250c5d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f209c670 0x55a4f20a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f3246ffd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f20940c0 0x55a4f20a7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f3246ffd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f20940c0 0x55a4f20a7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:52632/0 (socket says 172.21.15.31:52632) 2024-04-26T18:02:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f3246ffd640 1 -- 172.21.15.31:0/3045338227 learned_addr learned my addr 172.21.15.31:0/3045338227 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f3250c5d740 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f209f2a0 0x55a4f20a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f32477fe640 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f209c670 0x55a4f20a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f3250c5d740 1 -- 172.21.15.31:0/3045338227 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f323400cdc0 con 0x55a4f209f2a0 2024-04-26T18:02:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f3250c5d740 1 -- 172.21.15.31:0/3045338227 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f323400e230 con 0x55a4f20940c0 2024-04-26T18:02:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f3250c5d740 1 -- 172.21.15.31:0/3045338227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f323400d490 con 0x55a4f209c670 2024-04-26T18:02:15.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f3250c5d740 10 client.0 _ll_register_callbacks cb 0x7f323c00c270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:02:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f3246ffd640 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f209f2a0 0x55a4f20a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f32477fe640 1 -- 172.21.15.31:0/3045338227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f20940c0 msgr2=0x55a4f20a7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f32477fe640 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f20940c0 0x55a4f20a7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f32477fe640 1 -- 172.21.15.31:0/3045338227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f209f2a0 msgr2=0x55a4f20a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f32477fe640 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f209f2a0 0x55a4f20a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.715+0000 7f32477fe640 1 -- 172.21.15.31:0/3045338227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4f20995b0 con 0x55a4f209c670 2024-04-26T18:02:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.716+0000 7f3246ffd640 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f209f2a0 0x55a4f20a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:02:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.716+0000 7f32477fe640 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f209c670 0x55a4f20a09e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x55a4f20ac000 tx=0x7f3238005f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f6c5ae447d6fe80 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.716+0000 7f3226ffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T18:02:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.716+0000 7f3226ffd640 1 -- 172.21.15.31:0/3045338227 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323801acf0 con 0x55a4f209c670 2024-04-26T18:02:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.716+0000 7f3226ffd640 1 -- 172.21.15.31:0/3045338227 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f32380273b0 con 0x55a4f209c670 2024-04-26T18:02:15.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.716+0000 7f3250c5d740 1 -- 172.21.15.31:0/3045338227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f323c011680 con 0x55a4f209c670 2024-04-26T18:02:15.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.719+0000 7f3226ffd640 1 -- 172.21.15.31:0/3045338227 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3238035050 con 0x55a4f209c670 2024-04-26T18:02:15.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.719+0000 7f3250c5d740 10 client.25534 fetch_fsmap learned FSMap version 32 2024-04-26T18:02:15.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.719+0000 7f3250c5d740 1 -- 172.21.15.31:0/3045338227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a4f209fc30 con 0x55a4f209c670 2024-04-26T18:02:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.722+0000 7f3226ffd640 1 -- 172.21.15.31:0/3045338227 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32380050a0 con 0x55a4f209c670 2024-04-26T18:02:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.722+0000 7f3226ffd640 1 -- 172.21.15.31:0/3045338227 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f32380035f0 con 0x55a4f209c670 2024-04-26T18:02:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.724+0000 7f3250c5d740 10 client.25534 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:02:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.724+0000 7f3250c5d740 10 client.25534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:02:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.724+0000 7f3250c5d740 20 client.25534 populate_metadata read hostname 'smithi031' 2024-04-26T18:02:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.724+0000 7f3250c5d740 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a4f20a0100 0x55a4f2118260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.725+0000 7f3250c5d740 4 client.25534 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:02:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.725+0000 7f3250c5d740 1 -- 172.21.15.31:0/3045338227 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f323c011680 con 0x55a4f20a0100 2024-04-26T18:02:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.725+0000 7f3246ffd640 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a4f20a0100 0x55a4f2118260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.727+0000 7f3246ffd640 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a4f20a0100 0x55a4f2118260 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5970cee1947a9406 server_cookie=21f51c4c3e55b819 in_seq=0 out_seq=0 2024-04-26T18:02:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.728+0000 7f3226ffd640 10 client.25534 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.763+0000 7f3226ffd640 1 -- 172.21.15.31:0/3045338227 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f323c011680 con 0x55a4f20a0100 2024-04-26T18:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.763+0000 7f3226ffd640 10 client.25534 handle_command_reply: tid=0 2024-04-26T18:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.764+0000 7f3250c5d740 1 client.25534 shutdown 2024-04-26T18:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.764+0000 7f3250c5d740 1 -- 172.21.15.31:0/3045338227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f209c670 msgr2=0x55a4f20a09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.764+0000 7f3250c5d740 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f209c670 0x55a4f20a09e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x55a4f20ac000 tx=0x7f3238005f50 comp rx=0 tx=0).stop 2024-04-26T18:02:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.765+0000 7f3250c5d740 1 -- 172.21.15.31:0/3045338227 shutdown_connections 2024-04-26T18:02:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.765+0000 7f3250c5d740 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55a4f20a0100 0x55a4f2118260 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.765+0000 7f3250c5d740 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f209f2a0 0x55a4f20a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.765+0000 7f3250c5d740 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a4f209c670 0x55a4f20a09e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.765+0000 7f3250c5d740 1 --2- 172.21.15.31:0/3045338227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55a4f20940c0 0x55a4f20a7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.765+0000 7f3250c5d740 1 -- 172.21.15.31:0/3045338227 >> 172.21.15.31:0/3045338227 conn(0x55a4f20905d0 msgr2=0x55a4f208eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.765+0000 7f3226ffd640 0 client.25534 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:15.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.766+0000 7f3250c5d740 1 -- 172.21.15.31:0/3045338227 shutdown_connections 2024-04-26T18:02:15.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.767+0000 7f3250c5d740 1 -- 172.21.15.31:0/3045338227 wait complete. 2024-04-26T18:02:15.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.767+0000 7f3250c5d740 20 client.25534 trim_cache size 0 max 16384 2024-04-26T18:02:15.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:02:15.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:02:15.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:02:15.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:02:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.768+0000 7f324cce4640 1 -- 172.21.15.31:0/798693097 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f32300a3ee0 msgr2=0x7f32300a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.768+0000 7f324cce4640 1 --2- 172.21.15.31:0/798693097 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f32300a3ee0 0x7f32300a63a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f32400ba510 tx=0x7f324003e040 comp rx=0 tx=0).stop 2024-04-26T18:02:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.768+0000 7f324cce4640 1 -- 172.21.15.31:0/798693097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248108750 msgr2=0x7f324810cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.768+0000 7f324cce4640 1 --2- 172.21.15.31:0/798693097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248108750 0x7f324810cda0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3240005e30 tx=0x7f3240003ba0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.768+0000 7f324cce4640 1 -- 172.21.15.31:0/798693097 shutdown_connections 2024-04-26T18:02:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.768+0000 7f324cce4640 1 --2- 172.21.15.31:0/798693097 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248108750 0x7f324810cda0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.768+0000 7f324cce4640 1 --2- 172.21.15.31:0/798693097 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f32300a3ee0 0x7f32300a63a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.768+0000 7f324cce4640 1 --2- 172.21.15.31:0/798693097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3248103100 0x7f324810c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.768+0000 7f324cce4640 1 --2- 172.21.15.31:0/798693097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3248102740 0x7f324810f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.768+0000 7f324cce4640 1 -- 172.21.15.31:0/798693097 >> 172.21.15.31:0/798693097 conn(0x7f32480fe270 msgr2=0x7f32480fe7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.768+0000 7f324cce4640 1 -- 172.21.15.31:0/798693097 shutdown_connections 2024-04-26T18:02:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:15.768+0000 7f324cce4640 1 -- 172.21.15.31:0/798693097 wait complete. 2024-04-26T18:02:17.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:17 smithi186 ceph-mon[32797]: pgmap v176: 97 pgs: 97 active+clean; 3.3 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 69 MiB/s wr, 2.19k op/s 2024-04-26T18:02:17.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:17 smithi031 ceph-mon[29164]: pgmap v176: 97 pgs: 97 active+clean; 3.3 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 69 MiB/s wr, 2.19k op/s 2024-04-26T18:02:17.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:17 smithi097 ceph-mon[32419]: pgmap v176: 97 pgs: 97 active+clean; 3.3 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 69 MiB/s wr, 2.19k op/s 2024-04-26T18:02:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 295.0 s, 3888.4 tps, lat 25.541 ms stddev 31.407 2024-04-26T18:02:19.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:19 smithi186 ceph-mon[32797]: pgmap v177: 97 pgs: 97 active+clean; 3.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 44 MiB/s wr, 1.08k op/s 2024-04-26T18:02:19.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:19 smithi031 ceph-mon[29164]: pgmap v177: 97 pgs: 97 active+clean; 3.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 44 MiB/s wr, 1.08k op/s 2024-04-26T18:02:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:19 smithi097 ceph-mon[32419]: pgmap v177: 97 pgs: 97 active+clean; 3.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 44 MiB/s wr, 1.08k op/s 2024-04-26T18:02:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:21 smithi186 ceph-mon[32797]: pgmap v178: 97 pgs: 97 active+clean; 3.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 34 MiB/s wr, 791 op/s 2024-04-26T18:02:21.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:21 smithi031 ceph-mon[29164]: pgmap v178: 97 pgs: 97 active+clean; 3.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 34 MiB/s wr, 791 op/s 2024-04-26T18:02:21.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:21 smithi097 ceph-mon[32419]: pgmap v178: 97 pgs: 97 active+clean; 3.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 34 MiB/s wr, 791 op/s 2024-04-26T18:02:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 300.0 s, 3026.4 tps, lat 33.256 ms stddev 64.463 2024-04-26T18:02:23.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:23 smithi186 ceph-mon[32797]: pgmap v179: 97 pgs: 97 active+clean; 3.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 66 MiB/s wr, 2.33k op/s 2024-04-26T18:02:23.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:23 smithi031 ceph-mon[29164]: pgmap v179: 97 pgs: 97 active+clean; 3.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 66 MiB/s wr, 2.33k op/s 2024-04-26T18:02:23.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:23 smithi097 ceph-mon[32419]: pgmap v179: 97 pgs: 97 active+clean; 3.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 66 MiB/s wr, 2.33k op/s 2024-04-26T18:02:25.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:25 smithi186 ceph-mon[32797]: pgmap v180: 97 pgs: 97 active+clean; 3.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 47 MiB/s wr, 2.00k op/s 2024-04-26T18:02:25.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:25 smithi031 ceph-mon[29164]: pgmap v180: 97 pgs: 97 active+clean; 3.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 47 MiB/s wr, 2.00k op/s 2024-04-26T18:02:25.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:25 smithi097 ceph-mon[32419]: pgmap v180: 97 pgs: 97 active+clean; 3.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 47 MiB/s wr, 2.00k op/s 2024-04-26T18:02:27.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:27 smithi186 ceph-mon[32797]: pgmap v181: 97 pgs: 97 active+clean; 3.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 52 MiB/s wr, 2.24k op/s 2024-04-26T18:02:27.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:27 smithi031 ceph-mon[29164]: pgmap v181: 97 pgs: 97 active+clean; 3.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 52 MiB/s wr, 2.24k op/s 2024-04-26T18:02:27.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:27 smithi097 ceph-mon[32419]: pgmap v181: 97 pgs: 97 active+clean; 3.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 52 MiB/s wr, 2.24k op/s 2024-04-26T18:02:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 305.0 s, 3689.6 tps, lat 27.052 ms stddev 28.472 2024-04-26T18:02:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.901+0000 7fcec1b2c640 1 -- 172.21.15.31:0/3286034501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcebc1032f0 msgr2=0x7fcebc10f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.901+0000 7fcec1b2c640 1 --2- 172.21.15.31:0/3286034501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcebc1032f0 0x7fcebc10f800 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009cd0 tx=0x7fceb402a8f0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.901+0000 7fcec1b2c640 1 -- 172.21.15.31:0/3286034501 shutdown_connections 2024-04-26T18:02:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.901+0000 7fcec1b2c640 1 --2- 172.21.15.31:0/3286034501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcebc1032f0 0x7fcebc10f800 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.901+0000 7fcec1b2c640 1 --2- 172.21.15.31:0/3286034501 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcebc102930 0x7fcebc102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.901+0000 7fcec1b2c640 1 --2- 172.21.15.31:0/3286034501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcebc108940 0x7fcebc108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.901+0000 7fcec1b2c640 1 -- 172.21.15.31:0/3286034501 >> 172.21.15.31:0/3286034501 conn(0x7fcebc0fe3e0 msgr2=0x7fcebc100800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.901+0000 7fcec1b2c640 1 -- 172.21.15.31:0/3286034501 shutdown_connections 2024-04-26T18:02:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.901+0000 7fcec1b2c640 1 -- 172.21.15.31:0/3286034501 wait complete. 2024-04-26T18:02:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.902+0000 7fcec1b2c640 1 Processor -- start 2024-04-26T18:02:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.902+0000 7fcec1b2c640 1 -- start start 2024-04-26T18:02:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.902+0000 7fcec1b2c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcebc102930 0x7fcebc1a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.902+0000 7fcec1b2c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcebc1032f0 0x7fcebc1a1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.902+0000 7fcec1b2c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcebc108940 0x7fcebc19b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.902+0000 7fcec1b2c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc075420 con 0x7fcebc1032f0 2024-04-26T18:02:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.902+0000 7fcec1b2c640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc0752a0 con 0x7fcebc102930 2024-04-26T18:02:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.902+0000 7fcec1b2c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc0755a0 con 0x7fcebc108940 2024-04-26T18:02:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.903+0000 7fcebb7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcebc102930 0x7fcebc1a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.903+0000 7fcebb7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcebc102930 0x7fcebc1a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:33520/0 (socket says 172.21.15.31:33520) 2024-04-26T18:02:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.903+0000 7fcebb7fe640 1 -- 172.21.15.31:0/1460513719 learned_addr learned my addr 172.21.15.31:0/1460513719 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.903+0000 7fcebb7fe640 1 -- 172.21.15.31:0/1460513719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcebc108940 msgr2=0x7fcebc19b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.903+0000 7fcebb7fe640 1 --2- 172.21.15.31:0/1460513719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcebc108940 0x7fcebc19b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.903+0000 7fcebb7fe640 1 -- 172.21.15.31:0/1460513719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcebc1032f0 msgr2=0x7fcebc1a1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.903+0000 7fcebaffd640 1 --2- 172.21.15.31:0/1460513719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcebc1032f0 0x7fcebc1a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.903+0000 7fcebb7fe640 1 --2- 172.21.15.31:0/1460513719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcebc1032f0 0x7fcebc1a1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.903+0000 7fcebb7fe640 1 -- 172.21.15.31:0/1460513719 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcebc19bbb0 con 0x7fcebc102930 2024-04-26T18:02:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.904+0000 7fcebaffd640 1 --2- 172.21.15.31:0/1460513719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcebc1032f0 0x7fcebc1a1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:02:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.904+0000 7fcebb7fe640 1 --2- 172.21.15.31:0/1460513719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcebc102930 0x7fcebc1a0ec0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fceac00e9f0 tx=0x7fceac00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.904+0000 7fceb8ff9640 1 -- 172.21.15.31:0/1460513719 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac00cd80 con 0x7fcebc102930 2024-04-26T18:02:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.904+0000 7fceb8ff9640 1 -- 172.21.15.31:0/1460513719 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceac0108e0 con 0x7fcebc102930 2024-04-26T18:02:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.904+0000 7fceb8ff9640 1 -- 172.21.15.31:0/1460513719 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac018840 con 0x7fcebc102930 2024-04-26T18:02:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.905+0000 7fcec1b2c640 1 -- 172.21.15.31:0/1460513719 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcebc19bea0 con 0x7fcebc102930 2024-04-26T18:02:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.905+0000 7fcec1b2c640 1 -- 172.21.15.31:0/1460513719 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcebc1a7d10 con 0x7fcebc102930 2024-04-26T18:02:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.908+0000 7fceb8ff9640 1 -- 172.21.15.31:0/1460513719 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fceac01f020 con 0x7fcebc102930 2024-04-26T18:02:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.909+0000 7fceb8ff9640 1 --2- 172.21.15.31:0/1460513719 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fcea40a3e90 0x7fcea40a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.909+0000 7fceb8ff9640 1 -- 172.21.15.31:0/1460513719 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fceac0d0c10 con 0x7fcebc102930 2024-04-26T18:02:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.909+0000 7fcebaffd640 1 --2- 172.21.15.31:0/1460513719 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fcea40a3e90 0x7fcea40a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.909+0000 7fcec4845740 1 Processor -- start 2024-04-26T18:02:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.909+0000 7fcec4845740 1 -- start start 2024-04-26T18:02:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.909+0000 7fcec4845740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9577f00 0x560ee95782e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.910+0000 7fcec4845740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9578820 0x560ee9582ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.910+0000 7fcec4845740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee95830e0 0x560ee95854d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.910+0000 7fcec4845740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560ee94de460 con 0x560ee9578820 2024-04-26T18:02:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.910+0000 7fcec4845740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x560ee9498590 con 0x560ee9577f00 2024-04-26T18:02:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.910+0000 7fcec4845740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560ee94da420 con 0x560ee95830e0 2024-04-26T18:02:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.910+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee95830e0 0x560ee95854d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.910+0000 7fcebb7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9577f00 0x560ee95782e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.910+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee95830e0 0x560ee95854d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:57722/0 (socket says 172.21.15.31:57722) 2024-04-26T18:02:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.910+0000 7fcebbfff640 1 -- 172.21.15.31:0/104563406 learned_addr learned my addr 172.21.15.31:0/104563406 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.911+0000 7fcebaffd640 1 --2- 172.21.15.31:0/104563406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9578820 0x560ee9582ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.911+0000 7fcebbfff640 1 -- 172.21.15.31:0/104563406 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9577f00 msgr2=0x560ee95782e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.911+0000 7fcebbfff640 1 --2- 172.21.15.31:0/104563406 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9577f00 0x560ee95782e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.911+0000 7fcebbfff640 1 -- 172.21.15.31:0/104563406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9578820 msgr2=0x560ee9582ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.911+0000 7fcebbfff640 1 --2- 172.21.15.31:0/104563406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9578820 0x560ee9582ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.911+0000 7fcebbfff640 1 -- 172.21.15.31:0/104563406 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ee94a50d0 con 0x560ee95830e0 2024-04-26T18:02:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.911+0000 7fcebbfff640 1 --2- 172.21.15.31:0/104563406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee95830e0 0x560ee95854d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fceb402ae60 tx=0x7fceb4009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6ee2c9521ace115 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.911+0000 7fcebaffd640 1 --2- 172.21.15.31:0/104563406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9578820 0x560ee9582ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:02:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.911+0000 7fcebaffd640 1 --2- 172.21.15.31:0/1460513719 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fcea40a3e90 0x7fcea40a6350 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcebc19c7c0 tx=0x7fceb0005e50 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.911+0000 7fcea1ffb640 1 -- 172.21.15.31:0/104563406 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb40305a0 con 0x560ee95830e0 2024-04-26T18:02:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.911+0000 7fcea1ffb640 1 -- 172.21.15.31:0/104563406 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceb403f070 con 0x560ee95830e0 2024-04-26T18:02:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.911+0000 7fcea1ffb640 1 -- 172.21.15.31:0/104563406 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb40042a0 con 0x560ee95830e0 2024-04-26T18:02:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.912+0000 7fcec4845740 1 -- 172.21.15.31:0/104563406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee95830e0 msgr2=0x560ee95854d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.912+0000 7fcec4845740 1 --2- 172.21.15.31:0/104563406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee95830e0 0x560ee95854d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fceb402ae60 tx=0x7fceb4009a20 comp rx=0 tx=0).stop 2024-04-26T18:02:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.912+0000 7fcec4845740 1 -- 172.21.15.31:0/104563406 shutdown_connections 2024-04-26T18:02:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.912+0000 7fcec4845740 1 --2- 172.21.15.31:0/104563406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee95830e0 0x560ee95854d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.912+0000 7fcec4845740 1 --2- 172.21.15.31:0/104563406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9578820 0x560ee9582ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.912+0000 7fcec4845740 1 --2- 172.21.15.31:0/104563406 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9577f00 0x560ee95782e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.912+0000 7fcec4845740 1 -- 172.21.15.31:0/104563406 >> 172.21.15.31:0/104563406 conn(0x560ee9574410 msgr2=0x560ee95747f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:28.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.912+0000 7fcec4845740 1 -- 172.21.15.31:0/104563406 shutdown_connections 2024-04-26T18:02:28.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.912+0000 7fcec4845740 1 -- 172.21.15.31:0/104563406 wait complete. 2024-04-26T18:02:28.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.913+0000 7fcec4845740 1 -- start start 2024-04-26T18:02:28.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.913+0000 7fcec4845740 1 Processor -- start 2024-04-26T18:02:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcec4845740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9577f00 0x560ee9580090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcec4845740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9578820 0x560ee95805d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcec4845740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee95830e0 0x560ee9580b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcebaffd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9578820 0x560ee95805d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcebb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9577f00 0x560ee9580090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcebb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9577f00 0x560ee9580090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58776/0 (socket says 172.21.15.31:58776) 2024-04-26T18:02:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcebaffd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9578820 0x560ee95805d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:33532/0 (socket says 172.21.15.31:33532) 2024-04-26T18:02:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcec4845740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560ee9585cb0 con 0x560ee9577f00 2024-04-26T18:02:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcec4845740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x560ee9465b60 con 0x560ee9578820 2024-04-26T18:02:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcec4845740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560ee9585e30 con 0x560ee95830e0 2024-04-26T18:02:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcebb7fe640 1 -- 172.21.15.31:0/3225446488 learned_addr learned my addr 172.21.15.31:0/3225446488 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcec4845740 10 client.0 _ll_register_callbacks cb 0x560ee948a2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:02:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcebbfff640 1 --2- 172.21.15.31:0/3225446488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee95830e0 0x560ee9580b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcebaffd640 1 -- 172.21.15.31:0/3225446488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee95830e0 msgr2=0x560ee9580b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcebaffd640 1 --2- 172.21.15.31:0/3225446488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee95830e0 0x560ee9580b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcebaffd640 1 -- 172.21.15.31:0/3225446488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9577f00 msgr2=0x560ee9580090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcebaffd640 1 --2- 172.21.15.31:0/3225446488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9577f00 0x560ee9580090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.914+0000 7fcebaffd640 1 -- 172.21.15.31:0/3225446488 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ee91f0210 con 0x560ee9578820 2024-04-26T18:02:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.915+0000 7fcebaffd640 1 --2- 172.21.15.31:0/3225446488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9578820 0x560ee95805d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fceb0012000 tx=0x7fceb00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6199285d95ef2ff server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.915+0000 7fce9affd640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T18:02:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.915+0000 7fce9affd640 1 -- 172.21.15.31:0/3225446488 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0022070 con 0x560ee9578820 2024-04-26T18:02:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.915+0000 7fce9affd640 1 -- 172.21.15.31:0/3225446488 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fceb0010a60 con 0x560ee9578820 2024-04-26T18:02:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.915+0000 7fcec4845740 1 -- 172.21.15.31:0/3225446488 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fceb403f070 con 0x560ee9578820 2024-04-26T18:02:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.915+0000 7fce9affd640 1 -- 172.21.15.31:0/3225446488 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fceb002d050 con 0x560ee9578820 2024-04-26T18:02:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.915+0000 7fcec4845740 10 client.25634 fetch_fsmap learned FSMap version 32 2024-04-26T18:02:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.915+0000 7fcec4845740 1 -- 172.21.15.31:0/3225446488 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560ee92bc620 con 0x560ee9578820 2024-04-26T18:02:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.915+0000 7fce9affd640 1 -- 172.21.15.31:0/3225446488 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceb0017040 con 0x560ee9578820 2024-04-26T18:02:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.915+0000 7fce9affd640 1 -- 172.21.15.31:0/3225446488 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fceb000f040 con 0x560ee9578820 2024-04-26T18:02:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.916+0000 7fcec4845740 10 client.25634 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:02:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.916+0000 7fcec4845740 10 client.25634 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:02:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.916+0000 7fcec4845740 20 client.25634 populate_metadata read hostname 'smithi031' 2024-04-26T18:02:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.916+0000 7fcec4845740 1 --2- 172.21.15.31:0/3225446488 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560ee9593230 0x560ee95956a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.916+0000 7fcec4845740 4 client.25634 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:02:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.916+0000 7fcec4845740 1 -- 172.21.15.31:0/3225446488 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fceb403f070 con 0x560ee9593230 2024-04-26T18:02:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.916+0000 7fcebb7fe640 1 --2- 172.21.15.31:0/3225446488 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560ee9593230 0x560ee95956a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.917+0000 7fcebb7fe640 1 --2- 172.21.15.31:0/3225446488 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560ee9593230 0x560ee95956a0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44edc28617461a09 server_cookie=6687d1a3a38af52d in_seq=0 out_seq=0 2024-04-26T18:02:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.917+0000 7fce9affd640 10 client.25634 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.917+0000 7fce9affd640 1 -- 172.21.15.31:0/3225446488 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fceb403f070 con 0x560ee9593230 2024-04-26T18:02:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.917+0000 7fce9affd640 10 client.25634 handle_command_reply: tid=0 2024-04-26T18:02:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.918+0000 7fcec4845740 1 client.25634 shutdown 2024-04-26T18:02:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.918+0000 7fcec4845740 1 -- 172.21.15.31:0/3225446488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9578820 msgr2=0x560ee95805d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.918+0000 7fcec4845740 1 --2- 172.21.15.31:0/3225446488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9578820 0x560ee95805d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fceb0012000 tx=0x7fceb00046c0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.918+0000 7fcec4845740 1 -- 172.21.15.31:0/3225446488 shutdown_connections 2024-04-26T18:02:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.918+0000 7fcec4845740 1 --2- 172.21.15.31:0/3225446488 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560ee9593230 0x560ee95956a0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.918+0000 7fcec4845740 1 --2- 172.21.15.31:0/3225446488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee95830e0 0x560ee9580b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.918+0000 7fcec4845740 1 --2- 172.21.15.31:0/3225446488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9578820 0x560ee95805d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.918+0000 7fcec4845740 1 --2- 172.21.15.31:0/3225446488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9577f00 0x560ee9580090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.918+0000 7fcec4845740 1 -- 172.21.15.31:0/3225446488 >> 172.21.15.31:0/3225446488 conn(0x560ee9574410 msgr2=0x560ee957a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.918+0000 7fce9affd640 0 client.25634 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.918+0000 7fcec4845740 1 -- 172.21.15.31:0/3225446488 shutdown_connections 2024-04-26T18:02:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.918+0000 7fcec4845740 1 -- 172.21.15.31:0/3225446488 wait complete. 2024-04-26T18:02:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.918+0000 7fcec4845740 20 client.25634 trim_cache size 0 max 16384 2024-04-26T18:02:28.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.942+0000 7fcec4845740 1 Processor -- start 2024-04-26T18:02:28.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.942+0000 7fcec4845740 1 -- start start 2024-04-26T18:02:28.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.943+0000 7fcec4845740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee9577f00 0x560ee957e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.943+0000 7fcec4845740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9578820 0x560ee957a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.943+0000 7fcec4845740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee95830e0 0x560ee9580930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.943+0000 7fcec4845740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc1b7340 con 0x560ee9578820 2024-04-26T18:02:28.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.943+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee95830e0 0x560ee9580930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.943+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee95830e0 0x560ee9580930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:33548/0 (socket says 172.21.15.31:33548) 2024-04-26T18:02:28.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.943+0000 7fcebbfff640 1 -- 172.21.15.31:0/498270229 learned_addr learned my addr 172.21.15.31:0/498270229 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:28.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.943+0000 7fcec4845740 1 -- 172.21.15.31:0/498270229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc1b4bb0 con 0x560ee95830e0 2024-04-26T18:02:28.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.943+0000 7fcec4845740 1 -- 172.21.15.31:0/498270229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc1b2a40 con 0x560ee9577f00 2024-04-26T18:02:28.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.944+0000 7fcebaffd640 1 --2- 172.21.15.31:0/498270229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9578820 0x560ee957a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.944+0000 7fcebb7fe640 1 --2- 172.21.15.31:0/498270229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee9577f00 0x560ee957e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.944+0000 7fcebbfff640 1 -- 172.21.15.31:0/498270229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee9577f00 msgr2=0x560ee957e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:28.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.944+0000 7fcebbfff640 1 --2- 172.21.15.31:0/498270229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee9577f00 0x560ee957e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.944+0000 7fcebbfff640 1 -- 172.21.15.31:0/498270229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9578820 msgr2=0x560ee957a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:28.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.944+0000 7fcebbfff640 1 --2- 172.21.15.31:0/498270229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9578820 0x560ee957a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.944+0000 7fcebbfff640 1 -- 172.21.15.31:0/498270229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ee957f040 con 0x560ee95830e0 2024-04-26T18:02:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.944+0000 7fcebbfff640 1 --2- 172.21.15.31:0/498270229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee95830e0 0x560ee9580930 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009bf0 tx=0x7fceb4003e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c50c4d4f41212072 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.944+0000 7fcea17fa640 1 -- 172.21.15.31:0/498270229 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4003970 con 0x560ee95830e0 2024-04-26T18:02:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.945+0000 7fcea17fa640 1 -- 172.21.15.31:0/498270229 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceb4053070 con 0x560ee95830e0 2024-04-26T18:02:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.945+0000 7fcea17fa640 1 -- 172.21.15.31:0/498270229 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4004360 con 0x560ee95830e0 2024-04-26T18:02:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.945+0000 7fcec4845740 1 -- 172.21.15.31:0/498270229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee95830e0 msgr2=0x560ee9580930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.945+0000 7fcec4845740 1 --2- 172.21.15.31:0/498270229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee95830e0 0x560ee9580930 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009bf0 tx=0x7fceb4003e20 comp rx=0 tx=0).stop 2024-04-26T18:02:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.945+0000 7fcec4845740 1 -- 172.21.15.31:0/498270229 shutdown_connections 2024-04-26T18:02:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.945+0000 7fcec4845740 1 --2- 172.21.15.31:0/498270229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee95830e0 0x560ee9580930 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.945+0000 7fcec4845740 1 --2- 172.21.15.31:0/498270229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee9578820 0x560ee957a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.945+0000 7fcec4845740 1 --2- 172.21.15.31:0/498270229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee9577f00 0x560ee957e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.945+0000 7fcec4845740 1 -- 172.21.15.31:0/498270229 >> 172.21.15.31:0/498270229 conn(0x560ee9574410 msgr2=0x560ee95735b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.946+0000 7fcec4845740 1 -- 172.21.15.31:0/498270229 shutdown_connections 2024-04-26T18:02:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.946+0000 7fcec4845740 1 -- 172.21.15.31:0/498270229 wait complete. 2024-04-26T18:02:28.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.947+0000 7fcec4845740 1 -- start start 2024-04-26T18:02:28.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.947+0000 7fcec4845740 1 Processor -- start 2024-04-26T18:02:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.947+0000 7fcec4845740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9577f00 0x560ee957cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.947+0000 7fcec4845740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee9578820 0x560ee957d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.947+0000 7fcec4845740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee95830e0 0x560ee957d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.947+0000 7fcec4845740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc1b3090 con 0x560ee95830e0 2024-04-26T18:02:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.947+0000 7fcec4845740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc1b44e0 con 0x560ee9577f00 2024-04-26T18:02:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.947+0000 7fcec4845740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc1b3740 con 0x560ee9578820 2024-04-26T18:02:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.947+0000 7fcec4845740 10 client.0 _ll_register_callbacks cb 0x7fceb0009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:02:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fcebb7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9577f00 0x560ee957cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee95830e0 0x560ee957d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fcebb7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9577f00 0x560ee957cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:33556/0 (socket says 172.21.15.31:33556) 2024-04-26T18:02:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fcebb7fe640 1 -- 172.21.15.31:0/2937080677 learned_addr learned my addr 172.21.15.31:0/2937080677 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fcebaffd640 1 --2- 172.21.15.31:0/2937080677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee9578820 0x560ee957d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fcebbfff640 1 -- 172.21.15.31:0/2937080677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee9578820 msgr2=0x560ee957d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fcebbfff640 1 --2- 172.21.15.31:0/2937080677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee9578820 0x560ee957d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fcebbfff640 1 -- 172.21.15.31:0/2937080677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9577f00 msgr2=0x560ee957cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fcebbfff640 1 --2- 172.21.15.31:0/2937080677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9577f00 0x560ee957cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fcebbfff640 1 -- 172.21.15.31:0/2937080677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ee95f8820 con 0x560ee95830e0 2024-04-26T18:02:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fcebaffd640 1 --2- 172.21.15.31:0/2937080677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee9578820 0x560ee957d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:02:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fcebb7fe640 1 --2- 172.21.15.31:0/2937080677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9577f00 0x560ee957cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:02:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fcebbfff640 1 --2- 172.21.15.31:0/2937080677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee95830e0 0x560ee957d690 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fceb4004af0 tx=0x7fceb4009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96932fe658113220 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fce9affd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:02:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fce9affd640 1 -- 172.21.15.31:0/2937080677 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb405d070 con 0x560ee95830e0 2024-04-26T18:02:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fce9affd640 1 -- 172.21.15.31:0/2937080677 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fceb4030c10 con 0x560ee95830e0 2024-04-26T18:02:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.948+0000 7fcec4845740 1 -- 172.21.15.31:0/2937080677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fceb4053070 con 0x560ee95830e0 2024-04-26T18:02:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.949+0000 7fce9affd640 1 -- 172.21.15.31:0/2937080677 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fceb4003e90 con 0x560ee95830e0 2024-04-26T18:02:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.949+0000 7fcec4845740 10 client.15954 fetch_fsmap learned FSMap version 32 2024-04-26T18:02:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.949+0000 7fcec4845740 1 -- 172.21.15.31:0/2937080677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560ee95f99f0 con 0x560ee95830e0 2024-04-26T18:02:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.949+0000 7fce9affd640 1 -- 172.21.15.31:0/2937080677 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceb4053070 con 0x560ee95830e0 2024-04-26T18:02:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.949+0000 7fce9affd640 1 -- 172.21.15.31:0/2937080677 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fceb4044710 con 0x560ee95830e0 2024-04-26T18:02:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.949+0000 7fcec4845740 10 client.15954 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:02:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.949+0000 7fcec4845740 10 client.15954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:02:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.949+0000 7fcec4845740 20 client.15954 populate_metadata read hostname 'smithi031' 2024-04-26T18:02:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.949+0000 7fcec4845740 1 --2- 172.21.15.31:0/2937080677 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560ee95f9fe0 0x560ee95fc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.949+0000 7fcec4845740 4 client.15954 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:02:28.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.949+0000 7fcec4845740 1 -- 172.21.15.31:0/2937080677 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560ee94ec070 con 0x560ee95f9fe0 2024-04-26T18:02:28.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.950+0000 7fcebb7fe640 1 --2- 172.21.15.31:0/2937080677 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560ee95f9fe0 0x560ee95fc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:28.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.950+0000 7fcebb7fe640 1 --2- 172.21.15.31:0/2937080677 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560ee95f9fe0 0x560ee95fc400 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2230e06264f3a8b2 server_cookie=d9f3c996c0a3fab4 in_seq=0 out_seq=0 2024-04-26T18:02:28.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.950+0000 7fce9affd640 10 client.15954 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:28.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.969+0000 7fce9affd640 1 -- 172.21.15.31:0/2937080677 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560ee94ec070 con 0x560ee95f9fe0 2024-04-26T18:02:28.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.969+0000 7fce9affd640 10 client.15954 handle_command_reply: tid=0 2024-04-26T18:02:28.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.969+0000 7fcec4845740 1 client.15954 shutdown 2024-04-26T18:02:28.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec4845740 1 -- 172.21.15.31:0/2937080677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee95830e0 msgr2=0x560ee957d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:28.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec4845740 1 --2- 172.21.15.31:0/2937080677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee95830e0 0x560ee957d690 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fceb4004af0 tx=0x7fceb4009a20 comp rx=0 tx=0).stop 2024-04-26T18:02:28.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec4845740 1 -- 172.21.15.31:0/2937080677 shutdown_connections 2024-04-26T18:02:28.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec4845740 1 --2- 172.21.15.31:0/2937080677 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x560ee95f9fe0 0x560ee95fc400 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec4845740 1 --2- 172.21.15.31:0/2937080677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ee95830e0 0x560ee957d690 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec4845740 1 --2- 172.21.15.31:0/2937080677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ee9578820 0x560ee957d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fce9affd640 0 client.15954 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:28.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec4845740 1 --2- 172.21.15.31:0/2937080677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560ee9577f00 0x560ee957cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec4845740 1 -- 172.21.15.31:0/2937080677 >> 172.21.15.31:0/2937080677 conn(0x560ee9574410 msgr2=0x560ee95818c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:28.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec4845740 1 -- 172.21.15.31:0/2937080677 shutdown_connections 2024-04-26T18:02:28.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec4845740 1 -- 172.21.15.31:0/2937080677 wait complete. 2024-04-26T18:02:28.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec4845740 20 client.15954 trim_cache size 0 max 16384 2024-04-26T18:02:28.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:02:28.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:02:28.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:02:28.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:02:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec1b2c640 1 -- 172.21.15.31:0/1460513719 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fcea40a3e90 msgr2=0x7fcea40a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec1b2c640 1 --2- 172.21.15.31:0/1460513719 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fcea40a3e90 0x7fcea40a6350 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcebc19c7c0 tx=0x7fceb0005e50 comp rx=0 tx=0).stop 2024-04-26T18:02:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec1b2c640 1 -- 172.21.15.31:0/1460513719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcebc102930 msgr2=0x7fcebc1a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec1b2c640 1 --2- 172.21.15.31:0/1460513719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcebc102930 0x7fcebc1a0ec0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fceac00e9f0 tx=0x7fceac00eeb0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec1b2c640 1 -- 172.21.15.31:0/1460513719 shutdown_connections 2024-04-26T18:02:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec1b2c640 1 --2- 172.21.15.31:0/1460513719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcebc108940 0x7fcebc19b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec1b2c640 1 --2- 172.21.15.31:0/1460513719 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fcea40a3e90 0x7fcea40a6350 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec1b2c640 1 --2- 172.21.15.31:0/1460513719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcebc1032f0 0x7fcebc1a1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.970+0000 7fcec1b2c640 1 --2- 172.21.15.31:0/1460513719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcebc102930 0x7fcebc1a0ec0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.971+0000 7fcec1b2c640 1 -- 172.21.15.31:0/1460513719 >> 172.21.15.31:0/1460513719 conn(0x7fcebc0fe3e0 msgr2=0x7fcebc075d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.971+0000 7fcec1b2c640 1 -- 172.21.15.31:0/1460513719 shutdown_connections 2024-04-26T18:02:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:28.971+0000 7fcec1b2c640 1 -- 172.21.15.31:0/1460513719 wait complete. 2024-04-26T18:02:29.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:29 smithi186 ceph-mon[32797]: pgmap v182: 97 pgs: 97 active+clean; 3.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 40 MiB/s wr, 1.95k op/s 2024-04-26T18:02:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:29 smithi031 ceph-mon[29164]: pgmap v182: 97 pgs: 97 active+clean; 3.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 40 MiB/s wr, 1.95k op/s 2024-04-26T18:02:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:29 smithi097 ceph-mon[32419]: pgmap v182: 97 pgs: 97 active+clean; 3.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 40 MiB/s wr, 1.95k op/s 2024-04-26T18:02:31.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:31 smithi186 ceph-mon[32797]: pgmap v183: 97 pgs: 97 active+clean; 3.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 38 MiB/s wr, 1.87k op/s 2024-04-26T18:02:31.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:31 smithi031 ceph-mon[29164]: pgmap v183: 97 pgs: 97 active+clean; 3.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 38 MiB/s wr, 1.87k op/s 2024-04-26T18:02:31.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:31 smithi097 ceph-mon[32419]: pgmap v183: 97 pgs: 97 active+clean; 3.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 38 MiB/s wr, 1.87k op/s 2024-04-26T18:02:33.096 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 310.0 s, 2875.5 tps, lat 26.179 ms stddev 27.770 2024-04-26T18:02:33.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:33 smithi186 ceph-mon[32797]: pgmap v184: 97 pgs: 97 active+clean; 3.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 43 MiB/s wr, 2.19k op/s 2024-04-26T18:02:33.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:33 smithi031 ceph-mon[29164]: pgmap v184: 97 pgs: 97 active+clean; 3.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 43 MiB/s wr, 2.19k op/s 2024-04-26T18:02:33.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:33 smithi097 ceph-mon[32419]: pgmap v184: 97 pgs: 97 active+clean; 3.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 43 MiB/s wr, 2.19k op/s 2024-04-26T18:02:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:35 smithi186 ceph-mon[32797]: pgmap v185: 97 pgs: 97 active+clean; 3.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 11 MiB/s wr, 643 op/s 2024-04-26T18:02:35.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:35 smithi031 ceph-mon[29164]: pgmap v185: 97 pgs: 97 active+clean; 3.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 11 MiB/s wr, 643 op/s 2024-04-26T18:02:35.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:35 smithi097 ceph-mon[32419]: pgmap v185: 97 pgs: 97 active+clean; 3.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 11 MiB/s wr, 643 op/s 2024-04-26T18:02:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:37 smithi186 ceph-mon[32797]: pgmap v186: 97 pgs: 97 active+clean; 3.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 13 MiB/s wr, 814 op/s 2024-04-26T18:02:37.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:37 smithi031 ceph-mon[29164]: pgmap v186: 97 pgs: 97 active+clean; 3.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 13 MiB/s wr, 814 op/s 2024-04-26T18:02:37.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:37 smithi097 ceph-mon[32419]: pgmap v186: 97 pgs: 97 active+clean; 3.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 13 MiB/s wr, 814 op/s 2024-04-26T18:02:38.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 315.0 s, 2370.2 tps, lat 52.644 ms stddev 290.608 2024-04-26T18:02:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:39 smithi186 ceph-mon[32797]: pgmap v187: 97 pgs: 97 active+clean; 3.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 9.6 MiB/s wr, 619 op/s 2024-04-26T18:02:39.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:39 smithi031 ceph-mon[29164]: pgmap v187: 97 pgs: 97 active+clean; 3.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 9.6 MiB/s wr, 619 op/s 2024-04-26T18:02:39.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:39 smithi097 ceph-mon[32419]: pgmap v187: 97 pgs: 97 active+clean; 3.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 9.6 MiB/s wr, 619 op/s 2024-04-26T18:02:41.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:41 smithi186 ceph-mon[32797]: pgmap v188: 97 pgs: 97 active+clean; 3.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 7.8 MiB/s wr, 541 op/s 2024-04-26T18:02:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:41 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:02:41.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:41 smithi031 ceph-mon[29164]: pgmap v188: 97 pgs: 97 active+clean; 3.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 7.8 MiB/s wr, 541 op/s 2024-04-26T18:02:41.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:41 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:02:41.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:41 smithi097 ceph-mon[32419]: pgmap v188: 97 pgs: 97 active+clean; 3.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 7.8 MiB/s wr, 541 op/s 2024-04-26T18:02:41.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:41 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:02:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.113+0000 7ff184293640 1 -- 172.21.15.31:0/549767786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c109930 msgr2=0x7ff17c109db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.113+0000 7ff184293640 1 --2- 172.21.15.31:0/549767786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c109930 0x7ff17c109db0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009370 tx=0x7ff16c02a420 comp rx=0 tx=0).stop 2024-04-26T18:02:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.114+0000 7ff184293640 1 -- 172.21.15.31:0/549767786 shutdown_connections 2024-04-26T18:02:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.114+0000 7ff184293640 1 --2- 172.21.15.31:0/549767786 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff17c10a2f0 0x7ff17c116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.114+0000 7ff184293640 1 --2- 172.21.15.31:0/549767786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c109930 0x7ff17c109db0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.114+0000 7ff184293640 1 --2- 172.21.15.31:0/549767786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff17c10e390 0x7ff17c10e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.114+0000 7ff184293640 1 -- 172.21.15.31:0/549767786 >> 172.21.15.31:0/549767786 conn(0x7ff17c06de90 msgr2=0x7ff17c06e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.114+0000 7ff184293640 1 -- 172.21.15.31:0/549767786 shutdown_connections 2024-04-26T18:02:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.114+0000 7ff184293640 1 -- 172.21.15.31:0/549767786 wait complete. 2024-04-26T18:02:42.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.115+0000 7ff184293640 1 Processor -- start 2024-04-26T18:02:42.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.115+0000 7ff184293640 1 -- start start 2024-04-26T18:02:42.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.115+0000 7ff184293640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff17c109930 0x7ff17c1a7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.115+0000 7ff184293640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c10a2f0 0x7ff17c1a8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.116+0000 7ff18202f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff17c109930 0x7ff17c1a7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.116+0000 7ff184293640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff17c10e390 0x7ff17c1ac6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.116+0000 7ff184293640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c1191a0 con 0x7ff17c10a2f0 2024-04-26T18:02:42.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.116+0000 7ff184293640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c119020 con 0x7ff17c10e390 2024-04-26T18:02:42.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.116+0000 7ff184293640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c119320 con 0x7ff17c109930 2024-04-26T18:02:42.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.116+0000 7ff18202f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff17c109930 0x7ff17c1a7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52302/0 (socket says 172.21.15.31:52302) 2024-04-26T18:02:42.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.116+0000 7ff18202f640 1 -- 172.21.15.31:0/3878707895 learned_addr learned my addr 172.21.15.31:0/3878707895 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:42.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.116+0000 7ff18182e640 1 --2- 172.21.15.31:0/3878707895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c10a2f0 0x7ff17c1a8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.116+0000 7ff182830640 1 --2- 172.21.15.31:0/3878707895 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff17c10e390 0x7ff17c1ac6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.116+0000 7ff18202f640 1 -- 172.21.15.31:0/3878707895 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff17c10e390 msgr2=0x7ff17c1ac6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.116+0000 7ff18202f640 1 --2- 172.21.15.31:0/3878707895 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff17c10e390 0x7ff17c1ac6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.116+0000 7ff18202f640 1 -- 172.21.15.31:0/3878707895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c10a2f0 msgr2=0x7ff17c1a8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.116+0000 7ff18202f640 1 --2- 172.21.15.31:0/3878707895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c10a2f0 0x7ff17c1a8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.116+0000 7ff18202f640 1 -- 172.21.15.31:0/3878707895 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff17c1acdf0 con 0x7ff17c109930 2024-04-26T18:02:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.117+0000 7ff18202f640 1 --2- 172.21.15.31:0/3878707895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff17c109930 0x7ff17c1a7e50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff17800bfd0 tx=0x7ff17800ad50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.117+0000 7ff1737fe640 1 -- 172.21.15.31:0/3878707895 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff178019070 con 0x7ff17c109930 2024-04-26T18:02:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.117+0000 7ff1737fe640 1 -- 172.21.15.31:0/3878707895 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff178014410 con 0x7ff17c109930 2024-04-26T18:02:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.117+0000 7ff184293640 1 -- 172.21.15.31:0/3878707895 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff17c1ad0e0 con 0x7ff17c109930 2024-04-26T18:02:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.117+0000 7ff1737fe640 1 -- 172.21.15.31:0/3878707895 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff178007e60 con 0x7ff17c109930 2024-04-26T18:02:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.117+0000 7ff184293640 1 -- 172.21.15.31:0/3878707895 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff17c1b4970 con 0x7ff17c109930 2024-04-26T18:02:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.120+0000 7ff1737fe640 1 -- 172.21.15.31:0/3878707895 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff17801c070 con 0x7ff17c109930 2024-04-26T18:02:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.121+0000 7ff186fad740 1 Processor -- start 2024-04-26T18:02:42.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.121+0000 7ff186fad740 1 -- start start 2024-04-26T18:02:42.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.121+0000 7ff186fad740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b6f00 0x55b5e62b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.121+0000 7ff186fad740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b7820 0x55b5e62c1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.121+0000 7ff186fad740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 0x55b5e62c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.121+0000 7ff186fad740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b5e621d460 con 0x55b5e62b7820 2024-04-26T18:02:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.121+0000 7ff186fad740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b5e61d7590 con 0x55b5e62c20e0 2024-04-26T18:02:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.121+0000 7ff186fad740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b5e6219420 con 0x55b5e62b6f00 2024-04-26T18:02:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.122+0000 7ff182830640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b6f00 0x55b5e62b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.122+0000 7ff182830640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b6f00 0x55b5e62b72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52308/0 (socket says 172.21.15.31:52308) 2024-04-26T18:02:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.122+0000 7ff182830640 1 -- 172.21.15.31:0/3151672633 learned_addr learned my addr 172.21.15.31:0/3151672633 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.122+0000 7ff18182e640 1 --2- 172.21.15.31:0/3151672633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 0x55b5e62c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.122+0000 7ff182830640 1 -- 172.21.15.31:0/3151672633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 msgr2=0x55b5e62c44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.122+0000 7ff182830640 1 --2- 172.21.15.31:0/3151672633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 0x55b5e62c44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.122+0000 7ff182830640 1 -- 172.21.15.31:0/3151672633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b7820 msgr2=0x55b5e62c1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.122+0000 7ff18202f640 1 --2- 172.21.15.31:0/3151672633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b7820 0x55b5e62c1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.123+0000 7ff1737fe640 1 --2- 172.21.15.31:0/3878707895 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ff1680a3eb0 0x7ff1680a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.123+0000 7ff182830640 1 --2- 172.21.15.31:0/3151672633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b7820 0x55b5e62c1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.123+0000 7ff182830640 1 -- 172.21.15.31:0/3151672633 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5e61e40d0 con 0x55b5e62b6f00 2024-04-26T18:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.123+0000 7ff18182e640 1 --2- 172.21.15.31:0/3878707895 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ff1680a3eb0 0x7ff1680a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.123+0000 7ff1737fe640 1 -- 172.21.15.31:0/3878707895 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7ff1780cf950 con 0x7ff17c109930 2024-04-26T18:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.123+0000 7ff182830640 1 --2- 172.21.15.31:0/3151672633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b6f00 0x55b5e62b72e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff17400aa30 tx=0x7ff17400a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da78142e190d1187 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.123+0000 7ff18202f640 1 --2- 172.21.15.31:0/3151672633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b7820 0x55b5e62c1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.123+0000 7ff170bfe640 1 -- 172.21.15.31:0/3151672633 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff174004b20 con 0x55b5e62b6f00 2024-04-26T18:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.123+0000 7ff170bfe640 1 -- 172.21.15.31:0/3151672633 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff174004cc0 con 0x55b5e62b6f00 2024-04-26T18:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.123+0000 7ff170bfe640 1 -- 172.21.15.31:0/3151672633 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17400d840 con 0x55b5e62b6f00 2024-04-26T18:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.124+0000 7ff186fad740 1 -- 172.21.15.31:0/3151672633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b6f00 msgr2=0x55b5e62b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.124+0000 7ff18182e640 1 --2- 172.21.15.31:0/3878707895 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ff1680a3eb0 0x7ff1680a6370 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff16c006010 tx=0x7ff16c000bc0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.124+0000 7ff186fad740 1 --2- 172.21.15.31:0/3151672633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b6f00 0x55b5e62b72e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff17400aa30 tx=0x7ff17400a560 comp rx=0 tx=0).stop 2024-04-26T18:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.124+0000 7ff186fad740 1 -- 172.21.15.31:0/3151672633 shutdown_connections 2024-04-26T18:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.124+0000 7ff186fad740 1 --2- 172.21.15.31:0/3151672633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 0x55b5e62c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.124+0000 7ff186fad740 1 --2- 172.21.15.31:0/3151672633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b7820 0x55b5e62c1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.124+0000 7ff186fad740 1 --2- 172.21.15.31:0/3151672633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b6f00 0x55b5e62b72e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.124+0000 7ff186fad740 1 -- 172.21.15.31:0/3151672633 >> 172.21.15.31:0/3151672633 conn(0x55b5e62b3410 msgr2=0x55b5e62b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.124+0000 7ff186fad740 1 -- 172.21.15.31:0/3151672633 shutdown_connections 2024-04-26T18:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.124+0000 7ff186fad740 1 -- 172.21.15.31:0/3151672633 wait complete. 2024-04-26T18:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.125+0000 7ff186fad740 1 -- start start 2024-04-26T18:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.125+0000 7ff186fad740 1 Processor -- start 2024-04-26T18:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.125+0000 7ff186fad740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b6f00 0x55b5e62bf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.125+0000 7ff186fad740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b7820 0x55b5e62bf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.125+0000 7ff186fad740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 0x55b5e62bfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.125+0000 7ff186fad740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b5e62c4d10 con 0x55b5e62b6f00 2024-04-26T18:02:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.125+0000 7ff186fad740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55b5e61a4b60 con 0x55b5e62c20e0 2024-04-26T18:02:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.125+0000 7ff186fad740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b5e62c4e90 con 0x55b5e62b7820 2024-04-26T18:02:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.125+0000 7ff186fad740 10 client.0 _ll_register_callbacks cb 0x55b5e61c92b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:02:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.125+0000 7ff18202f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b6f00 0x55b5e62bf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.125+0000 7ff182830640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 0x55b5e62bfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.125+0000 7ff18202f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b6f00 0x55b5e62bf060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35944/0 (socket says 172.21.15.31:35944) 2024-04-26T18:02:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.125+0000 7ff18202f640 1 -- 172.21.15.31:0/2767703616 learned_addr learned my addr 172.21.15.31:0/2767703616 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.125+0000 7ff18182e640 1 --2- 172.21.15.31:0/2767703616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b7820 0x55b5e62bf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.126+0000 7ff182830640 1 -- 172.21.15.31:0/2767703616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b7820 msgr2=0x55b5e62bf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.126+0000 7ff182830640 1 --2- 172.21.15.31:0/2767703616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b7820 0x55b5e62bf5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.126+0000 7ff182830640 1 -- 172.21.15.31:0/2767703616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b6f00 msgr2=0x55b5e62bf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.126+0000 7ff182830640 1 --2- 172.21.15.31:0/2767703616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b6f00 0x55b5e62bf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.126+0000 7ff182830640 1 -- 172.21.15.31:0/2767703616 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5e5f2f210 con 0x55b5e62c20e0 2024-04-26T18:02:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.126+0000 7ff18202f640 1 --2- 172.21.15.31:0/2767703616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b6f00 0x55b5e62bf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:02:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.126+0000 7ff182830640 1 --2- 172.21.15.31:0/2767703616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 0x55b5e62bfae0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff174006010 tx=0x7ff174005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a0e9ab0c1620904 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.126+0000 7ff165ffb640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T18:02:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.126+0000 7ff165ffb640 1 -- 172.21.15.31:0/2767703616 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff174005210 con 0x55b5e62c20e0 2024-04-26T18:02:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.126+0000 7ff165ffb640 1 -- 172.21.15.31:0/2767703616 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7ff1740053b0 con 0x55b5e62c20e0 2024-04-26T18:02:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.126+0000 7ff186fad740 1 -- 172.21.15.31:0/2767703616 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff174004cc0 con 0x55b5e62c20e0 2024-04-26T18:02:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.126+0000 7ff165ffb640 1 -- 172.21.15.31:0/2767703616 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff17402d020 con 0x55b5e62c20e0 2024-04-26T18:02:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.126+0000 7ff186fad740 10 client.25652 fetch_fsmap learned FSMap version 32 2024-04-26T18:02:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.126+0000 7ff186fad740 1 -- 172.21.15.31:0/2767703616 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b5e5ffb620 con 0x55b5e62c20e0 2024-04-26T18:02:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.127+0000 7ff165ffb640 1 -- 172.21.15.31:0/2767703616 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1740055d0 con 0x55b5e62c20e0 2024-04-26T18:02:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.127+0000 7ff165ffb640 1 -- 172.21.15.31:0/2767703616 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ff17400b040 con 0x55b5e62c20e0 2024-04-26T18:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.127+0000 7ff186fad740 10 client.25652 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.127+0000 7ff186fad740 10 client.25652 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.127+0000 7ff186fad740 20 client.25652 populate_metadata read hostname 'smithi031' 2024-04-26T18:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.127+0000 7ff186fad740 1 --2- 172.21.15.31:0/2767703616 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b5e62d2530 0x55b5e62d4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.127+0000 7ff186fad740 4 client.25652 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.127+0000 7ff186fad740 1 -- 172.21.15.31:0/2767703616 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff174004cc0 con 0x55b5e62d2530 2024-04-26T18:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.128+0000 7ff18202f640 1 --2- 172.21.15.31:0/2767703616 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b5e62d2530 0x55b5e62d4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.129+0000 7ff165ffb640 10 client.25652 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.129+0000 7ff18202f640 1 --2- 172.21.15.31:0/2767703616 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b5e62d2530 0x55b5e62d4990 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72bac7be3e35f67f server_cookie=4e4e4d4b34310f99 in_seq=0 out_seq=0 2024-04-26T18:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.130+0000 7ff165ffb640 1 -- 172.21.15.31:0/2767703616 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7ff174004cc0 con 0x55b5e62d2530 2024-04-26T18:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.130+0000 7ff165ffb640 10 client.25652 handle_command_reply: tid=0 2024-04-26T18:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.130+0000 7ff186fad740 1 client.25652 shutdown 2024-04-26T18:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.130+0000 7ff186fad740 1 -- 172.21.15.31:0/2767703616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 msgr2=0x55b5e62bfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.130+0000 7ff186fad740 1 --2- 172.21.15.31:0/2767703616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 0x55b5e62bfae0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff174006010 tx=0x7ff174005830 comp rx=0 tx=0).stop 2024-04-26T18:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.130+0000 7ff186fad740 1 -- 172.21.15.31:0/2767703616 shutdown_connections 2024-04-26T18:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.130+0000 7ff186fad740 1 --2- 172.21.15.31:0/2767703616 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b5e62d2530 0x55b5e62d4990 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.130+0000 7ff186fad740 1 --2- 172.21.15.31:0/2767703616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 0x55b5e62bfae0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.130+0000 7ff186fad740 1 --2- 172.21.15.31:0/2767703616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b7820 0x55b5e62bf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.130+0000 7ff186fad740 1 --2- 172.21.15.31:0/2767703616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b6f00 0x55b5e62bf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.130+0000 7ff186fad740 1 -- 172.21.15.31:0/2767703616 >> 172.21.15.31:0/2767703616 conn(0x55b5e62b3410 msgr2=0x55b5e62b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.130+0000 7ff165ffb640 0 client.25652 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.130+0000 7ff186fad740 1 -- 172.21.15.31:0/2767703616 shutdown_connections 2024-04-26T18:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.130+0000 7ff186fad740 1 -- 172.21.15.31:0/2767703616 wait complete. 2024-04-26T18:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.131+0000 7ff186fad740 20 client.25652 trim_cache size 0 max 16384 2024-04-26T18:02:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.152+0000 7ff186fad740 1 Processor -- start 2024-04-26T18:02:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.152+0000 7ff186fad740 1 -- start start 2024-04-26T18:02:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.152+0000 7ff186fad740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b6f00 0x55b5e62cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.152+0000 7ff186fad740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b7820 0x55b5e62da260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff186fad740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 0x55b5e62da7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff186fad740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c1bfb00 con 0x55b5e62b6f00 2024-04-26T18:02:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff186fad740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c1bd370 con 0x55b5e62c20e0 2024-04-26T18:02:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff186fad740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c1bb1c0 con 0x55b5e62b7820 2024-04-26T18:02:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff18182e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b7820 0x55b5e62da260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff18202f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b6f00 0x55b5e62cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff18182e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b7820 0x55b5e62da260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52324/0 (socket says 172.21.15.31:52324) 2024-04-26T18:02:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff18182e640 1 -- 172.21.15.31:0/3045890669 learned_addr learned my addr 172.21.15.31:0/3045890669 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff182830640 1 --2- 172.21.15.31:0/3045890669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 0x55b5e62da7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff18182e640 1 -- 172.21.15.31:0/3045890669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 msgr2=0x55b5e62da7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff18182e640 1 --2- 172.21.15.31:0/3045890669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 0x55b5e62da7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff18182e640 1 -- 172.21.15.31:0/3045890669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b6f00 msgr2=0x55b5e62cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff18182e640 1 --2- 172.21.15.31:0/3045890669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b6f00 0x55b5e62cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff18182e640 1 -- 172.21.15.31:0/3045890669 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5e62bd5b0 con 0x55b5e62b7820 2024-04-26T18:02:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.153+0000 7ff18182e640 1 --2- 172.21.15.31:0/3045890669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b7820 0x55b5e62da260 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009340 tx=0x7ff16c03f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f00fcb80f9b79eab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.154+0000 7ff167fff640 1 -- 172.21.15.31:0/3045890669 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c035040 con 0x55b5e62b7820 2024-04-26T18:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.154+0000 7ff167fff640 1 -- 172.21.15.31:0/3045890669 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff16c04f030 con 0x55b5e62b7820 2024-04-26T18:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.154+0000 7ff167fff640 1 -- 172.21.15.31:0/3045890669 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c003de0 con 0x55b5e62b7820 2024-04-26T18:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.155+0000 7ff186fad740 1 -- 172.21.15.31:0/3045890669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b7820 msgr2=0x55b5e62da260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.155+0000 7ff186fad740 1 --2- 172.21.15.31:0/3045890669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b7820 0x55b5e62da260 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009340 tx=0x7ff16c03f000 comp rx=0 tx=0).stop 2024-04-26T18:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.155+0000 7ff186fad740 1 -- 172.21.15.31:0/3045890669 shutdown_connections 2024-04-26T18:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.155+0000 7ff186fad740 1 --2- 172.21.15.31:0/3045890669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62c20e0 0x55b5e62da7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.155+0000 7ff186fad740 1 --2- 172.21.15.31:0/3045890669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62b7820 0x55b5e62da260 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.155+0000 7ff186fad740 1 --2- 172.21.15.31:0/3045890669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b6f00 0x55b5e62cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.155+0000 7ff186fad740 1 -- 172.21.15.31:0/3045890669 >> 172.21.15.31:0/3045890669 conn(0x55b5e62b3410 msgr2=0x55b5e62b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.155+0000 7ff186fad740 1 -- 172.21.15.31:0/3045890669 shutdown_connections 2024-04-26T18:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.155+0000 7ff186fad740 1 -- 172.21.15.31:0/3045890669 wait complete. 2024-04-26T18:02:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.155+0000 7ff186fad740 1 -- start start 2024-04-26T18:02:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.155+0000 7ff186fad740 1 Processor -- start 2024-04-26T18:02:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.156+0000 7ff186fad740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62b6f00 0x55b5e62c5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.156+0000 7ff186fad740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b7820 0x55b5e62c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.156+0000 7ff186fad740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62c20e0 0x55b5e6336ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.156+0000 7ff186fad740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c1bb830 con 0x55b5e62b7820 2024-04-26T18:02:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.156+0000 7ff186fad740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c1bcca0 con 0x55b5e62b6f00 2024-04-26T18:02:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.156+0000 7ff186fad740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c1bbf00 con 0x55b5e62c20e0 2024-04-26T18:02:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.156+0000 7ff186fad740 10 client.0 _ll_register_callbacks cb 0x7ff17400a900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:02:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.156+0000 7ff182830640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62c20e0 0x55b5e6336ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.156+0000 7ff18202f640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62b6f00 0x55b5e62c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.156+0000 7ff182830640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62c20e0 0x55b5e6336ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52330/0 (socket says 172.21.15.31:52330) 2024-04-26T18:02:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.156+0000 7ff182830640 1 -- 172.21.15.31:0/2691498045 learned_addr learned my addr 172.21.15.31:0/2691498045 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.156+0000 7ff18202f640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62b6f00 0x55b5e62c5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:37768/0 (socket says 172.21.15.31:37768) 2024-04-26T18:02:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.157+0000 7ff18202f640 1 -- 172.21.15.31:0/2691498045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62c20e0 msgr2=0x55b5e6336ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.157+0000 7ff18202f640 1 --2- 172.21.15.31:0/2691498045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62c20e0 0x55b5e6336ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.157+0000 7ff18202f640 1 -- 172.21.15.31:0/2691498045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b7820 msgr2=0x55b5e62c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.157+0000 7ff18202f640 1 --2- 172.21.15.31:0/2691498045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b7820 0x55b5e62c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.157+0000 7ff18202f640 1 -- 172.21.15.31:0/2691498045 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5e6337000 con 0x55b5e62b6f00 2024-04-26T18:02:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.157+0000 7ff18202f640 1 --2- 172.21.15.31:0/2691498045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62b6f00 0x55b5e62c5290 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff17809b690 tx=0x7ff178086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d16f3f00776a274 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.157+0000 7ff182830640 1 --2- 172.21.15.31:0/2691498045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62c20e0 0x55b5e6336ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:02:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.157+0000 7ff165ffb640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T18:02:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.157+0000 7ff165ffb640 1 -- 172.21.15.31:0/2691498045 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff178099d50 con 0x55b5e62b6f00 2024-04-26T18:02:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.157+0000 7ff165ffb640 1 -- 172.21.15.31:0/2691498045 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7ff1780cd310 con 0x55b5e62b6f00 2024-04-26T18:02:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.158+0000 7ff186fad740 1 -- 172.21.15.31:0/2691498045 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff16c04f030 con 0x55b5e62b6f00 2024-04-26T18:02:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.158+0000 7ff165ffb640 1 -- 172.21.15.31:0/2691498045 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff1780cd510 con 0x55b5e62b6f00 2024-04-26T18:02:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.158+0000 7ff186fad740 10 client.25658 fetch_fsmap learned FSMap version 32 2024-04-26T18:02:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.158+0000 7ff186fad740 1 -- 172.21.15.31:0/2691498045 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b5e6338320 con 0x55b5e62b6f00 2024-04-26T18:02:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.158+0000 7ff165ffb640 1 -- 172.21.15.31:0/2691498045 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1780860b0 con 0x55b5e62b6f00 2024-04-26T18:02:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.159+0000 7ff165ffb640 1 -- 172.21.15.31:0/2691498045 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7ff1780976e0 con 0x55b5e62b6f00 2024-04-26T18:02:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.159+0000 7ff186fad740 10 client.25658 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:02:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.159+0000 7ff186fad740 10 client.25658 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:02:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.159+0000 7ff186fad740 20 client.25658 populate_metadata read hostname 'smithi031' 2024-04-26T18:02:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.159+0000 7ff186fad740 1 --2- 172.21.15.31:0/2691498045 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b5e6338a50 0x55b5e633aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.159+0000 7ff186fad740 4 client.25658 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:02:42.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.159+0000 7ff186fad740 1 -- 172.21.15.31:0/2691498045 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7ff16c04f030 con 0x55b5e6338a50 2024-04-26T18:02:42.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.159+0000 7ff18182e640 1 --2- 172.21.15.31:0/2691498045 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b5e6338a50 0x55b5e633aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:42.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.159+0000 7ff18182e640 1 --2- 172.21.15.31:0/2691498045 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b5e6338a50 0x55b5e633aeb0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e126a21cda232ad2 server_cookie=d98559e89d91a7f7 in_seq=0 out_seq=0 2024-04-26T18:02:42.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.159+0000 7ff165ffb640 10 client.25658 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.170+0000 7ff165ffb640 1 -- 172.21.15.31:0/2691498045 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7ff16c04f030 con 0x55b5e6338a50 2024-04-26T18:02:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.170+0000 7ff165ffb640 10 client.25658 handle_command_reply: tid=0 2024-04-26T18:02:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.171+0000 7ff186fad740 1 client.25658 shutdown 2024-04-26T18:02:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.171+0000 7ff186fad740 1 -- 172.21.15.31:0/2691498045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62b6f00 msgr2=0x55b5e62c5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.171+0000 7ff186fad740 1 --2- 172.21.15.31:0/2691498045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62b6f00 0x55b5e62c5290 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff17809b690 tx=0x7ff178086040 comp rx=0 tx=0).stop 2024-04-26T18:02:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.172+0000 7ff186fad740 1 -- 172.21.15.31:0/2691498045 shutdown_connections 2024-04-26T18:02:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.172+0000 7ff186fad740 1 --2- 172.21.15.31:0/2691498045 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55b5e6338a50 0x55b5e633aeb0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.172+0000 7ff186fad740 1 --2- 172.21.15.31:0/2691498045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5e62c20e0 0x55b5e6336ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.172+0000 7ff165ffb640 0 client.25658 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.172+0000 7ff186fad740 1 --2- 172.21.15.31:0/2691498045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5e62b7820 0x55b5e62c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.172+0000 7ff186fad740 1 --2- 172.21.15.31:0/2691498045 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b5e62b6f00 0x55b5e62c5290 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.172+0000 7ff186fad740 1 -- 172.21.15.31:0/2691498045 >> 172.21.15.31:0/2691498045 conn(0x55b5e62b3410 msgr2=0x55b5e62bd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.172+0000 7ff186fad740 1 -- 172.21.15.31:0/2691498045 shutdown_connections 2024-04-26T18:02:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.172+0000 7ff186fad740 1 -- 172.21.15.31:0/2691498045 wait complete. 2024-04-26T18:02:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.172+0000 7ff186fad740 20 client.25658 trim_cache size 0 max 16384 2024-04-26T18:02:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:02:42.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:02:42.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:02:42.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:02:42.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.173+0000 7ff184293640 1 -- 172.21.15.31:0/3878707895 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ff1680a3eb0 msgr2=0x7ff1680a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:42.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.173+0000 7ff184293640 1 --2- 172.21.15.31:0/3878707895 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ff1680a3eb0 0x7ff1680a6370 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff16c006010 tx=0x7ff16c000bc0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.173+0000 7ff184293640 1 -- 172.21.15.31:0/3878707895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff17c109930 msgr2=0x7ff17c1a7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.173+0000 7ff184293640 1 --2- 172.21.15.31:0/3878707895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff17c109930 0x7ff17c1a7e50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff17800bfd0 tx=0x7ff17800ad50 comp rx=0 tx=0).stop 2024-04-26T18:02:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.173+0000 7ff184293640 1 -- 172.21.15.31:0/3878707895 shutdown_connections 2024-04-26T18:02:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.173+0000 7ff184293640 1 --2- 172.21.15.31:0/3878707895 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff17c10e390 0x7ff17c1ac6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.173+0000 7ff184293640 1 --2- 172.21.15.31:0/3878707895 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ff1680a3eb0 0x7ff1680a6370 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.173+0000 7ff184293640 1 --2- 172.21.15.31:0/3878707895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff17c10a2f0 0x7ff17c1a8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.173+0000 7ff184293640 1 --2- 172.21.15.31:0/3878707895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff17c109930 0x7ff17c1a7e50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.173+0000 7ff184293640 1 -- 172.21.15.31:0/3878707895 >> 172.21.15.31:0/3878707895 conn(0x7ff17c06de90 msgr2=0x7ff17c0720e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.174+0000 7ff184293640 1 -- 172.21.15.31:0/3878707895 shutdown_connections 2024-04-26T18:02:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:42.174+0000 7ff184293640 1 -- 172.21.15.31:0/3878707895 wait complete. 2024-04-26T18:02:42.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:42 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:02:42.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:42 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:02:42.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:42 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:02:42.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:42 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:02:42.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:42 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:02:42.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:42 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:02:42.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:42 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:02:42.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:42 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:02:42.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:42 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:02:43.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 320.0 s, 3278.8 tps, lat 30.532 ms stddev 33.922 2024-04-26T18:02:43.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:43 smithi186 ceph-mon[32797]: pgmap v189: 97 pgs: 97 active+clean; 3.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 12 MiB/s wr, 858 op/s 2024-04-26T18:02:43.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:43 smithi031 ceph-mon[29164]: pgmap v189: 97 pgs: 97 active+clean; 3.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 12 MiB/s wr, 858 op/s 2024-04-26T18:02:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:43 smithi097 ceph-mon[32419]: pgmap v189: 97 pgs: 97 active+clean; 3.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 12 MiB/s wr, 858 op/s 2024-04-26T18:02:45.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:45 smithi186 ceph-mon[32797]: pgmap v190: 97 pgs: 97 active+clean; 3.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.1 MiB/s wr, 541 op/s 2024-04-26T18:02:45.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:45 smithi031 ceph-mon[29164]: pgmap v190: 97 pgs: 97 active+clean; 3.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.1 MiB/s wr, 541 op/s 2024-04-26T18:02:45.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:45 smithi097 ceph-mon[32419]: pgmap v190: 97 pgs: 97 active+clean; 3.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.1 MiB/s wr, 541 op/s 2024-04-26T18:02:47.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:47 smithi186 ceph-mon[32797]: pgmap v191: 97 pgs: 97 active+clean; 3.8 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 9.6 MiB/s wr, 761 op/s 2024-04-26T18:02:47.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:47 smithi031 ceph-mon[29164]: pgmap v191: 97 pgs: 97 active+clean; 3.8 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 9.6 MiB/s wr, 761 op/s 2024-04-26T18:02:47.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:47 smithi097 ceph-mon[32419]: pgmap v191: 97 pgs: 97 active+clean; 3.8 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 9.6 MiB/s wr, 761 op/s 2024-04-26T18:02:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 325.0 s, 3042.4 tps, lat 32.639 ms stddev 36.646 2024-04-26T18:02:49.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:49 smithi186 ceph-mon[32797]: pgmap v192: 97 pgs: 97 active+clean; 3.8 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.5 MiB/s wr, 665 op/s 2024-04-26T18:02:49.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:49 smithi031 ceph-mon[29164]: pgmap v192: 97 pgs: 97 active+clean; 3.8 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.5 MiB/s wr, 665 op/s 2024-04-26T18:02:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:49 smithi097 ceph-mon[32419]: pgmap v192: 97 pgs: 97 active+clean; 3.8 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.5 MiB/s wr, 665 op/s 2024-04-26T18:02:51.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:51 smithi186 ceph-mon[32797]: pgmap v193: 97 pgs: 97 active+clean; 3.8 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.8 MiB/s wr, 612 op/s 2024-04-26T18:02:51.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:51 smithi031 ceph-mon[29164]: pgmap v193: 97 pgs: 97 active+clean; 3.8 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.8 MiB/s wr, 612 op/s 2024-04-26T18:02:51.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:51 smithi097 ceph-mon[32419]: pgmap v193: 97 pgs: 97 active+clean; 3.8 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.8 MiB/s wr, 612 op/s 2024-04-26T18:02:53.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 330.0 s, 3570.0 tps, lat 28.180 ms stddev 30.725 2024-04-26T18:02:53.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:53 smithi186 ceph-mon[32797]: pgmap v194: 97 pgs: 97 active+clean; 3.8 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 12 MiB/s wr, 1.00k op/s 2024-04-26T18:02:53.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:53 smithi031 ceph-mon[29164]: pgmap v194: 97 pgs: 97 active+clean; 3.8 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 12 MiB/s wr, 1.00k op/s 2024-04-26T18:02:53.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:53 smithi097 ceph-mon[32419]: pgmap v194: 97 pgs: 97 active+clean; 3.8 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 12 MiB/s wr, 1.00k op/s 2024-04-26T18:02:55.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.294+0000 7f3b35a64640 1 -- 172.21.15.31:0/2558905963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b301022d0 msgr2=0x7f3b30106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:55.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.294+0000 7f3b35a64640 1 --2- 172.21.15.31:0/2558905963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b301022d0 0x7f3b30106790 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009860 tx=0x7f3b1802a460 comp rx=0 tx=0).stop 2024-04-26T18:02:55.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.294+0000 7f3b35a64640 1 -- 172.21.15.31:0/2558905963 shutdown_connections 2024-04-26T18:02:55.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.294+0000 7f3b35a64640 1 --2- 172.21.15.31:0/2558905963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b30106cd0 0x7f3b30107160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.294+0000 7f3b35a64640 1 --2- 172.21.15.31:0/2558905963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b301022d0 0x7f3b30106790 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.294+0000 7f3b35a64640 1 --2- 172.21.15.31:0/2558905963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b301019b0 0x7f3b30101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.294+0000 7f3b35a64640 1 -- 172.21.15.31:0/2558905963 >> 172.21.15.31:0/2558905963 conn(0x7f3b300789b0 msgr2=0x7f3b30078dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.295+0000 7f3b35a64640 1 -- 172.21.15.31:0/2558905963 shutdown_connections 2024-04-26T18:02:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.295+0000 7f3b35a64640 1 -- 172.21.15.31:0/2558905963 wait complete. 2024-04-26T18:02:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.296+0000 7f3b35a64640 1 Processor -- start 2024-04-26T18:02:55.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.297+0000 7f3b35a64640 1 -- start start 2024-04-26T18:02:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.298+0000 7f3b35a64640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b301019b0 0x7f3b3019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.298+0000 7f3b35a64640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b301022d0 0x7f3b3019fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.298+0000 7f3b35a64640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b30106cd0 0x7f3b301a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.298+0000 7f3b35a64640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301191d0 con 0x7f3b301019b0 2024-04-26T18:02:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.298+0000 7f3b35a64640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3b30119050 con 0x7f3b301022d0 2024-04-26T18:02:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.298+0000 7f3b35a64640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b30119350 con 0x7f3b30106cd0 2024-04-26T18:02:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.298+0000 7f3b26ffd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b301022d0 0x7f3b3019fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.298+0000 7f3b2ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b30106cd0 0x7f3b301a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.298+0000 7f3b2f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b301019b0 0x7f3b3019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.298+0000 7f3b26ffd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b301022d0 0x7f3b3019fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:49984/0 (socket says 172.21.15.31:49984) 2024-04-26T18:02:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.298+0000 7f3b26ffd640 1 -- 172.21.15.31:0/227149742 learned_addr learned my addr 172.21.15.31:0/227149742 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.299+0000 7f3b2ffff640 1 -- 172.21.15.31:0/227149742 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b301022d0 msgr2=0x7f3b3019fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.299+0000 7f3b2ffff640 1 --2- 172.21.15.31:0/227149742 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b301022d0 0x7f3b3019fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.299+0000 7f3b2ffff640 1 -- 172.21.15.31:0/227149742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b301019b0 msgr2=0x7f3b3019f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.299+0000 7f3b2ffff640 1 --2- 172.21.15.31:0/227149742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b301019b0 0x7f3b3019f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.299+0000 7f3b2ffff640 1 -- 172.21.15.31:0/227149742 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b301a46b0 con 0x7f3b30106cd0 2024-04-26T18:02:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.299+0000 7f3b26ffd640 1 --2- 172.21.15.31:0/227149742 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b301022d0 0x7f3b3019fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:02:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.299+0000 7f3b2f7fe640 1 --2- 172.21.15.31:0/227149742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b301019b0 0x7f3b3019f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:02:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.299+0000 7f3b2ffff640 1 --2- 172.21.15.31:0/227149742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b30106cd0 0x7f3b301a3f80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800ef00 tx=0x7f3b2800c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.300+0000 7f3b2d7fa640 1 -- 172.21.15.31:0/227149742 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b28004b20 con 0x7f3b30106cd0 2024-04-26T18:02:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.300+0000 7f3b2d7fa640 1 -- 172.21.15.31:0/227149742 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b28003e20 con 0x7f3b30106cd0 2024-04-26T18:02:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.300+0000 7f3b2d7fa640 1 -- 172.21.15.31:0/227149742 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2800b570 con 0x7f3b30106cd0 2024-04-26T18:02:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.300+0000 7f3b35a64640 1 -- 172.21.15.31:0/227149742 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b301a49a0 con 0x7f3b30106cd0 2024-04-26T18:02:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.300+0000 7f3b35a64640 1 -- 172.21.15.31:0/227149742 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b301ac230 con 0x7f3b30106cd0 2024-04-26T18:02:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.302+0000 7f3b2d7fa640 1 -- 172.21.15.31:0/227149742 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3b2801d020 con 0x7f3b30106cd0 2024-04-26T18:02:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.302+0000 7f3b38780740 1 Processor -- start 2024-04-26T18:02:55.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.303+0000 7f3b2d7fa640 1 --2- 172.21.15.31:0/227149742 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3b1c0a3e40 0x7f3b1c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.303+0000 7f3b26ffd640 1 --2- 172.21.15.31:0/227149742 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3b1c0a3e40 0x7f3b1c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.303+0000 7f3b38780740 1 -- start start 2024-04-26T18:02:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.304+0000 7f3b2d7fa640 1 -- 172.21.15.31:0/227149742 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f3b280d0530 con 0x7f3b30106cd0 2024-04-26T18:02:55.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.304+0000 7f3b26ffd640 1 --2- 172.21.15.31:0/227149742 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3b1c0a3e40 0x7f3b1c0a6300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3b180095f0 tx=0x7f3b18005cb0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.304+0000 7f3b38780740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08dff00 0x5582b08e02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.304+0000 7f3b38780740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08e0820 0x5582b08eaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.305+0000 7f3b2f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08dff00 0x5582b08e02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.305+0000 7f3b2f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08dff00 0x5582b08e02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53378/0 (socket says 172.21.15.31:53378) 2024-04-26T18:02:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.305+0000 7f3b2f7fe640 1 -- 172.21.15.31:0/3483549001 learned_addr learned my addr 172.21.15.31:0/3483549001 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.305+0000 7f3b38780740 1 --2- 172.21.15.31:0/3483549001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b08ed4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.305+0000 7f3b26ffd640 1 --2- 172.21.15.31:0/3483549001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08e0820 0x5582b08eaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.305+0000 7f3b38780740 1 -- 172.21.15.31:0/3483549001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5582b0846460 con 0x5582b08eb0e0 2024-04-26T18:02:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.305+0000 7f3b38780740 1 -- 172.21.15.31:0/3483549001 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5582b0800590 con 0x5582b08e0820 2024-04-26T18:02:55.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.305+0000 7f3b38780740 1 -- 172.21.15.31:0/3483549001 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5582b0842420 con 0x5582b08dff00 2024-04-26T18:02:55.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.306+0000 7f3b2f7fe640 1 -- 172.21.15.31:0/3483549001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08e0820 msgr2=0x5582b08eaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:55.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.306+0000 7f3b2f7fe640 1 --2- 172.21.15.31:0/3483549001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08e0820 0x5582b08eaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.306+0000 7f3b2ffff640 1 --2- 172.21.15.31:0/3483549001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b08ed4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.306+0000 7f3b2f7fe640 1 -- 172.21.15.31:0/3483549001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 msgr2=0x5582b08ed4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.306+0000 7f3b2f7fe640 1 --2- 172.21.15.31:0/3483549001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b08ed4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.306+0000 7f3b2f7fe640 1 -- 172.21.15.31:0/3483549001 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582b080d0d0 con 0x5582b08dff00 2024-04-26T18:02:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.306+0000 7f3b2ffff640 1 --2- 172.21.15.31:0/3483549001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b08ed4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:02:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.306+0000 7f3b2f7fe640 1 --2- 172.21.15.31:0/3483549001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08dff00 0x5582b08e02e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3b200115f0 tx=0x7f3b20011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd40bee4d6692c83 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.306+0000 7f3b25ffb640 1 -- 172.21.15.31:0/3483549001 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2001e070 con 0x5582b08dff00 2024-04-26T18:02:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.306+0000 7f3b25ffb640 1 -- 172.21.15.31:0/3483549001 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b2000faf0 con 0x5582b08dff00 2024-04-26T18:02:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.306+0000 7f3b25ffb640 1 -- 172.21.15.31:0/3483549001 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b20019b10 con 0x5582b08dff00 2024-04-26T18:02:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.307+0000 7f3b38780740 1 -- 172.21.15.31:0/3483549001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08dff00 msgr2=0x5582b08e02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.307+0000 7f3b38780740 1 --2- 172.21.15.31:0/3483549001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08dff00 0x5582b08e02e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3b200115f0 tx=0x7f3b20011ab0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.307+0000 7f3b38780740 1 -- 172.21.15.31:0/3483549001 shutdown_connections 2024-04-26T18:02:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.307+0000 7f3b38780740 1 --2- 172.21.15.31:0/3483549001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b08ed4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.307+0000 7f3b38780740 1 --2- 172.21.15.31:0/3483549001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08e0820 0x5582b08eaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.307+0000 7f3b38780740 1 --2- 172.21.15.31:0/3483549001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08dff00 0x5582b08e02e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.307+0000 7f3b38780740 1 -- 172.21.15.31:0/3483549001 >> 172.21.15.31:0/3483549001 conn(0x5582b08dc410 msgr2=0x5582b08dc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:55.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.307+0000 7f3b38780740 1 -- 172.21.15.31:0/3483549001 shutdown_connections 2024-04-26T18:02:55.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.307+0000 7f3b38780740 1 -- 172.21.15.31:0/3483549001 wait complete. 2024-04-26T18:02:55.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.307+0000 7f3b38780740 1 -- start start 2024-04-26T18:02:55.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b38780740 1 Processor -- start 2024-04-26T18:02:55.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b38780740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08e8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b38780740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08e0820 0x5582b08e8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b38780740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08eb0e0 0x5582b08e8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b38780740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5582b08edd10 con 0x5582b08e0820 2024-04-26T18:02:55.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b38780740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x5582b07cdb60 con 0x5582b08dff00 2024-04-26T18:02:55.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b38780740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5582b08ede90 con 0x5582b08eb0e0 2024-04-26T18:02:55.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b38780740 10 client.0 _ll_register_callbacks cb 0x5582b07f22b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:02:55.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b26ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08e0820 0x5582b08e8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b26ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08e0820 0x5582b08e8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50588/0 (socket says 172.21.15.31:50588) 2024-04-26T18:02:55.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b26ffd640 1 -- 172.21.15.31:0/2085491352 learned_addr learned my addr 172.21.15.31:0/2085491352 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:55.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b2f7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08e8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b2ffff640 1 --2- 172.21.15.31:0/2085491352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08eb0e0 0x5582b08e8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b2f7fe640 1 -- 172.21.15.31:0/2085491352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08eb0e0 msgr2=0x5582b08e8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:55.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b2f7fe640 1 --2- 172.21.15.31:0/2085491352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08eb0e0 0x5582b08e8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b2f7fe640 1 -- 172.21.15.31:0/2085491352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08e0820 msgr2=0x5582b08e8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b2f7fe640 1 --2- 172.21.15.31:0/2085491352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08e0820 0x5582b08e8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b2f7fe640 1 -- 172.21.15.31:0/2085491352 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582b0558210 con 0x5582b08dff00 2024-04-26T18:02:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b26ffd640 1 --2- 172.21.15.31:0/2085491352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08e0820 0x5582b08e8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:02:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b2f7fe640 1 --2- 172.21.15.31:0/2085491352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08e8040 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3b200115c0 tx=0x7f3b20009d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59dac10c6a0695e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b0effd640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T18:02:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b0effd640 1 -- 172.21.15.31:0/2085491352 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2001e040 con 0x5582b08dff00 2024-04-26T18:02:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b38780740 1 -- 172.21.15.31:0/2085491352 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b2000faf0 con 0x5582b08dff00 2024-04-26T18:02:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.309+0000 7f3b0effd640 1 -- 172.21.15.31:0/2085491352 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f3b20016040 con 0x5582b08dff00 2024-04-26T18:02:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.310+0000 7f3b0effd640 1 -- 172.21.15.31:0/2085491352 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3b2000f6c0 con 0x5582b08dff00 2024-04-26T18:02:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.310+0000 7f3b38780740 10 client.25670 fetch_fsmap learned FSMap version 32 2024-04-26T18:02:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.310+0000 7f3b38780740 1 -- 172.21.15.31:0/2085491352 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5582b0624620 con 0x5582b08dff00 2024-04-26T18:02:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.310+0000 7f3b0effd640 1 -- 172.21.15.31:0/2085491352 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b2000faf0 con 0x5582b08dff00 2024-04-26T18:02:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.310+0000 7f3b0effd640 1 -- 172.21.15.31:0/2085491352 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f3b20021070 con 0x5582b08dff00 2024-04-26T18:02:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.310+0000 7f3b38780740 10 client.25670 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:02:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.310+0000 7f3b38780740 10 client.25670 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:02:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.310+0000 7f3b38780740 20 client.25670 populate_metadata read hostname 'smithi031' 2024-04-26T18:02:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.310+0000 7f3b38780740 1 --2- 172.21.15.31:0/2085491352 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5582b08fb040 0x5582b08fd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.310+0000 7f3b38780740 4 client.25670 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:02:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.310+0000 7f3b38780740 1 -- 172.21.15.31:0/2085491352 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582b0854070 con 0x5582b08fb040 2024-04-26T18:02:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.311+0000 7f3b26ffd640 1 --2- 172.21.15.31:0/2085491352 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5582b08fb040 0x5582b08fd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.311+0000 7f3b0effd640 10 client.25670 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.311+0000 7f3b26ffd640 1 --2- 172.21.15.31:0/2085491352 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5582b08fb040 0x5582b08fd450 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60c37b9700c1aa26 server_cookie=e7f3488d0a76dfd1 in_seq=0 out_seq=0 2024-04-26T18:02:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b0effd640 1 -- 172.21.15.31:0/2085491352 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582b0854070 con 0x5582b08fb040 2024-04-26T18:02:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b0effd640 10 client.25670 handle_command_reply: tid=0 2024-04-26T18:02:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b38780740 1 client.25670 shutdown 2024-04-26T18:02:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b38780740 1 -- 172.21.15.31:0/2085491352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 msgr2=0x5582b08e8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b38780740 1 --2- 172.21.15.31:0/2085491352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08e8040 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3b200115c0 tx=0x7f3b20009d40 comp rx=0 tx=0).stop 2024-04-26T18:02:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b38780740 1 -- 172.21.15.31:0/2085491352 shutdown_connections 2024-04-26T18:02:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b38780740 1 --2- 172.21.15.31:0/2085491352 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5582b08fb040 0x5582b08fd450 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b38780740 1 --2- 172.21.15.31:0/2085491352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08eb0e0 0x5582b08e8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b38780740 1 --2- 172.21.15.31:0/2085491352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08e0820 0x5582b08e8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b38780740 1 --2- 172.21.15.31:0/2085491352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08e8040 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b0effd640 0 client.25670 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b38780740 1 -- 172.21.15.31:0/2085491352 >> 172.21.15.31:0/2085491352 conn(0x5582b08dc410 msgr2=0x5582b08e2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b38780740 1 -- 172.21.15.31:0/2085491352 shutdown_connections 2024-04-26T18:02:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b38780740 1 -- 172.21.15.31:0/2085491352 wait complete. 2024-04-26T18:02:55.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.312+0000 7f3b38780740 20 client.25670 trim_cache size 0 max 16384 2024-04-26T18:02:55.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.332+0000 7f3b38780740 1 Processor -- start 2024-04-26T18:02:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.332+0000 7f3b38780740 1 -- start start 2024-04-26T18:02:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.333+0000 7f3b38780740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.333+0000 7f3b38780740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08e0820 0x5582b08e71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.333+0000 7f3b38780740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b0902d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.333+0000 7f3b38780740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301b7440 con 0x5582b08eb0e0 2024-04-26T18:02:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.333+0000 7f3b38780740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301b4cb0 con 0x5582b08dff00 2024-04-26T18:02:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.333+0000 7f3b38780740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301b2b00 con 0x5582b08e0820 2024-04-26T18:02:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.333+0000 7f3b2f7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.333+0000 7f3b2f7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:50024/0 (socket says 172.21.15.31:50024) 2024-04-26T18:02:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.333+0000 7f3b2f7fe640 1 -- 172.21.15.31:0/2973064295 learned_addr learned my addr 172.21.15.31:0/2973064295 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.333+0000 7f3b2ffff640 1 --2- 172.21.15.31:0/2973064295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b0902d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.333+0000 7f3b26ffd640 1 --2- 172.21.15.31:0/2973064295 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08e0820 0x5582b08e71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.334+0000 7f3b26ffd640 1 -- 172.21.15.31:0/2973064295 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 msgr2=0x5582b08f6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:55.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.334+0000 7f3b26ffd640 1 --2- 172.21.15.31:0/2973064295 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08f6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.334+0000 7f3b26ffd640 1 -- 172.21.15.31:0/2973064295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 msgr2=0x5582b0902d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:55.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.334+0000 7f3b26ffd640 1 --2- 172.21.15.31:0/2973064295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b0902d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.334+0000 7f3b26ffd640 1 -- 172.21.15.31:0/2973064295 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582b08e65b0 con 0x5582b08e0820 2024-04-26T18:02:55.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.334+0000 7f3b2ffff640 1 --2- 172.21.15.31:0/2973064295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b0902d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:02:55.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.334+0000 7f3b2f7fe640 1 --2- 172.21.15.31:0/2973064295 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:02:55.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.335+0000 7f3b26ffd640 1 --2- 172.21.15.31:0/2973064295 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08e0820 0x5582b08e71c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x5582b08e78a0 tx=0x7f3b1803f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14c23882e30fb5d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:55.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.335+0000 7f3b257fa640 1 -- 172.21.15.31:0/2973064295 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b180556c0 con 0x5582b08e0820 2024-04-26T18:02:55.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.335+0000 7f3b257fa640 1 -- 172.21.15.31:0/2973064295 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b1805b070 con 0x5582b08e0820 2024-04-26T18:02:55.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.335+0000 7f3b257fa640 1 -- 172.21.15.31:0/2973064295 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18032770 con 0x5582b08e0820 2024-04-26T18:02:55.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.336+0000 7f3b38780740 1 -- 172.21.15.31:0/2973064295 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08e0820 msgr2=0x5582b08e71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:55.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.336+0000 7f3b38780740 1 --2- 172.21.15.31:0/2973064295 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08e0820 0x5582b08e71c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x5582b08e78a0 tx=0x7f3b1803f000 comp rx=0 tx=0).stop 2024-04-26T18:02:55.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.336+0000 7f3b38780740 1 -- 172.21.15.31:0/2973064295 shutdown_connections 2024-04-26T18:02:55.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.336+0000 7f3b38780740 1 --2- 172.21.15.31:0/2973064295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b0902d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.336+0000 7f3b38780740 1 --2- 172.21.15.31:0/2973064295 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08e0820 0x5582b08e71c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.336+0000 7f3b38780740 1 --2- 172.21.15.31:0/2973064295 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.336+0000 7f3b38780740 1 -- 172.21.15.31:0/2973064295 >> 172.21.15.31:0/2973064295 conn(0x5582b08dc410 msgr2=0x5582b08dc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.336+0000 7f3b38780740 1 -- 172.21.15.31:0/2973064295 shutdown_connections 2024-04-26T18:02:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.336+0000 7f3b38780740 1 -- 172.21.15.31:0/2973064295 wait complete. 2024-04-26T18:02:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.336+0000 7f3b38780740 1 -- start start 2024-04-26T18:02:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.336+0000 7f3b38780740 1 Processor -- start 2024-04-26T18:02:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b38780740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08f30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b38780740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08e0820 0x5582b08f3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b38780740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b08e4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b38780740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301b3170 con 0x5582b08eb0e0 2024-04-26T18:02:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b38780740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301b45e0 con 0x5582b08dff00 2024-04-26T18:02:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b38780740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301b3840 con 0x5582b08e0820 2024-04-26T18:02:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b38780740 10 client.0 _ll_register_callbacks cb 0x7f3b2001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:02:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b2f7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08f30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b2f7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08f30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:50034/0 (socket says 172.21.15.31:50034) 2024-04-26T18:02:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b2f7fe640 1 -- 172.21.15.31:0/3141254155 learned_addr learned my addr 172.21.15.31:0/3141254155 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:02:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b2ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b08e4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b26ffd640 1 --2- 172.21.15.31:0/3141254155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08e0820 0x5582b08f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b2f7fe640 1 -- 172.21.15.31:0/3141254155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08e0820 msgr2=0x5582b08f3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b2f7fe640 1 --2- 172.21.15.31:0/3141254155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08e0820 0x5582b08f3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b2f7fe640 1 -- 172.21.15.31:0/3141254155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 msgr2=0x5582b08e4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:02:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b2f7fe640 1 --2- 172.21.15.31:0/3141254155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b08e4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b2f7fe640 1 -- 172.21.15.31:0/3141254155 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582b08e4eb0 con 0x5582b08dff00 2024-04-26T18:02:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b2ffff640 1 --2- 172.21.15.31:0/3141254155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b08e4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:02:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b2f7fe640 1 --2- 172.21.15.31:0/3141254155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08f30e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3b200055f0 tx=0x7f3b20005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25cd50f8fdc97d0d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:02:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b0effd640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T18:02:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.337+0000 7f3b0effd640 1 -- 172.21.15.31:0/3141254155 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2001e040 con 0x5582b08dff00 2024-04-26T18:02:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.338+0000 7f3b0effd640 1 -- 172.21.15.31:0/3141254155 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f3b20011280 con 0x5582b08dff00 2024-04-26T18:02:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.338+0000 7f3b38780740 1 -- 172.21.15.31:0/3141254155 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b1805b070 con 0x5582b08dff00 2024-04-26T18:02:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.338+0000 7f3b0effd640 1 -- 172.21.15.31:0/3141254155 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3b20011480 con 0x5582b08dff00 2024-04-26T18:02:55.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.338+0000 7f3b38780740 10 client.25676 fetch_fsmap learned FSMap version 32 2024-04-26T18:02:55.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.338+0000 7f3b38780740 1 -- 172.21.15.31:0/3141254155 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5582b0972110 con 0x5582b08dff00 2024-04-26T18:02:55.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.338+0000 7f3b0effd640 1 -- 172.21.15.31:0/3141254155 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b1805b070 con 0x5582b08dff00 2024-04-26T18:02:55.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.338+0000 7f3b0effd640 1 -- 172.21.15.31:0/3141254155 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f3b20033070 con 0x5582b08dff00 2024-04-26T18:02:55.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.340+0000 7f3b38780740 10 client.25676 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:02:55.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.340+0000 7f3b38780740 10 client.25676 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:02:55.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.340+0000 7f3b38780740 20 client.25676 populate_metadata read hostname 'smithi031' 2024-04-26T18:02:55.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.340+0000 7f3b38780740 1 --2- 172.21.15.31:0/3141254155 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5582b0972890 0x5582b0974cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:02:55.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.340+0000 7f3b38780740 4 client.25676 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:02:55.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.340+0000 7f3b38780740 1 -- 172.21.15.31:0/3141254155 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5582b084dee0 con 0x5582b0972890 2024-04-26T18:02:55.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.340+0000 7f3b26ffd640 1 --2- 172.21.15.31:0/3141254155 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5582b0972890 0x5582b0974cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:02:55.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.340+0000 7f3b26ffd640 1 --2- 172.21.15.31:0/3141254155 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5582b0972890 0x5582b0974cb0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d57cbaa809955809 server_cookie=d1f7bfa0f8b02f67 in_seq=0 out_seq=0 2024-04-26T18:02:55.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.340+0000 7f3b0effd640 10 client.25676 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:55.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.383+0000 7f3b0effd640 1 -- 172.21.15.31:0/3141254155 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5582b084dee0 con 0x5582b0972890 2024-04-26T18:02:55.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.383+0000 7f3b0effd640 10 client.25676 handle_command_reply: tid=0 2024-04-26T18:02:55.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.384+0000 7f3b38780740 1 client.25676 shutdown 2024-04-26T18:02:55.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.384+0000 7f3b38780740 1 -- 172.21.15.31:0/3141254155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 msgr2=0x5582b08f30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:55.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.384+0000 7f3b38780740 1 --2- 172.21.15.31:0/3141254155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08f30e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3b200055f0 tx=0x7f3b20005280 comp rx=0 tx=0).stop 2024-04-26T18:02:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.386+0000 7f3b38780740 1 -- 172.21.15.31:0/3141254155 shutdown_connections 2024-04-26T18:02:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.386+0000 7f3b38780740 1 --2- 172.21.15.31:0/3141254155 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x5582b0972890 0x5582b0974cb0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.386+0000 7f3b38780740 1 --2- 172.21.15.31:0/3141254155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5582b08eb0e0 0x5582b08e4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.386+0000 7f3b38780740 1 --2- 172.21.15.31:0/3141254155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5582b08e0820 0x5582b08f3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.386+0000 7f3b38780740 1 --2- 172.21.15.31:0/3141254155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5582b08dff00 0x5582b08f30e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.386+0000 7f3b0effd640 0 client.25676 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:02:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.386+0000 7f3b38780740 1 -- 172.21.15.31:0/3141254155 >> 172.21.15.31:0/3141254155 conn(0x5582b08dc410 msgr2=0x5582b08e18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.386+0000 7f3b38780740 1 -- 172.21.15.31:0/3141254155 shutdown_connections 2024-04-26T18:02:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.387+0000 7f3b38780740 1 -- 172.21.15.31:0/3141254155 wait complete. 2024-04-26T18:02:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.387+0000 7f3b38780740 20 client.25676 trim_cache size 0 max 16384 2024-04-26T18:02:55.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:02:55.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:02:55.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:02:55.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:02:55.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.390+0000 7f3b35a64640 1 -- 172.21.15.31:0/227149742 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3b1c0a3e40 msgr2=0x7f3b1c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:55.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.390+0000 7f3b35a64640 1 --2- 172.21.15.31:0/227149742 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3b1c0a3e40 0x7f3b1c0a6300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3b180095f0 tx=0x7f3b18005cb0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.390+0000 7f3b35a64640 1 -- 172.21.15.31:0/227149742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b30106cd0 msgr2=0x7f3b301a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:02:55.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.390+0000 7f3b35a64640 1 --2- 172.21.15.31:0/227149742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b30106cd0 0x7f3b301a3f80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800ef00 tx=0x7f3b2800c740 comp rx=0 tx=0).stop 2024-04-26T18:02:55.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.391+0000 7f3b35a64640 1 -- 172.21.15.31:0/227149742 shutdown_connections 2024-04-26T18:02:55.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.391+0000 7f3b35a64640 1 --2- 172.21.15.31:0/227149742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b30106cd0 0x7f3b301a3f80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.391+0000 7f3b35a64640 1 --2- 172.21.15.31:0/227149742 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b301022d0 0x7f3b3019fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.391+0000 7f3b35a64640 1 --2- 172.21.15.31:0/227149742 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3b1c0a3e40 0x7f3b1c0a6300 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.391+0000 7f3b35a64640 1 --2- 172.21.15.31:0/227149742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b301019b0 0x7f3b3019f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:02:55.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.391+0000 7f3b35a64640 1 -- 172.21.15.31:0/227149742 >> 172.21.15.31:0/227149742 conn(0x7f3b300789b0 msgr2=0x7f3b30104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:02:55.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.394+0000 7f3b35a64640 1 -- 172.21.15.31:0/227149742 shutdown_connections 2024-04-26T18:02:55.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:02:55.395+0000 7f3b35a64640 1 -- 172.21.15.31:0/227149742 wait complete. 2024-04-26T18:02:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:55 smithi186 ceph-mon[32797]: pgmap v195: 97 pgs: 97 active+clean; 3.8 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.1 MiB/s wr, 684 op/s 2024-04-26T18:02:55.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:55 smithi031 ceph-mon[29164]: pgmap v195: 97 pgs: 97 active+clean; 3.8 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.1 MiB/s wr, 684 op/s 2024-04-26T18:02:55.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:55 smithi097 ceph-mon[32419]: pgmap v195: 97 pgs: 97 active+clean; 3.8 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.1 MiB/s wr, 684 op/s 2024-04-26T18:02:57.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:57 smithi186 ceph-mon[32797]: pgmap v196: 97 pgs: 97 active+clean; 3.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 11 MiB/s wr, 897 op/s 2024-04-26T18:02:57.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:57 smithi031 ceph-mon[29164]: pgmap v196: 97 pgs: 97 active+clean; 3.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 11 MiB/s wr, 897 op/s 2024-04-26T18:02:57.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:57 smithi097 ceph-mon[32419]: pgmap v196: 97 pgs: 97 active+clean; 3.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 11 MiB/s wr, 897 op/s 2024-04-26T18:02:58.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 335.0 s, 3263.8 tps, lat 30.544 ms stddev 33.613 2024-04-26T18:02:58.481 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 31.7 2024-04-26T18:02:58.482 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T18:02:58.882 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:02:59.274 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:02:59 smithi031 ceph-mon[29164]: pgmap v197: 97 pgs: 97 active+clean; 3.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 809 op/s 2024-04-26T18:02:59.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:02:59 smithi186 ceph-mon[32797]: pgmap v197: 97 pgs: 97 active+clean; 3.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 809 op/s 2024-04-26T18:02:59.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:02:59 smithi097 ceph-mon[32419]: pgmap v197: 97 pgs: 97 active+clean; 3.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 809 op/s 2024-04-26T18:03:00.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.981+0000 7f3da3448700 1 -- 172.21.15.31:0/3663524804 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d9c06d6b0 msgr2=0x7f3d9c117fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:00.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.981+0000 7f3da3448700 1 --2- 172.21.15.31:0/3663524804 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d9c06d6b0 0x7f3d9c117fb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3d900098f0 tx=0x7f3d90009c00 comp rx=0 tx=0).stop 2024-04-26T18:03:00.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.981+0000 7f3da3448700 1 -- 172.21.15.31:0/3663524804 shutdown_connections 2024-04-26T18:03:00.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.981+0000 7f3da3448700 1 --2- 172.21.15.31:0/3663524804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d9c1184f0 0x7f3d9c11a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:00.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.981+0000 7f3da3448700 1 --2- 172.21.15.31:0/3663524804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d9c06cd90 0x7f3d9c06d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:00.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.981+0000 7f3da3448700 1 --2- 172.21.15.31:0/3663524804 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d9c06d6b0 0x7f3d9c117fb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:00.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.981+0000 7f3da3448700 1 -- 172.21.15.31:0/3663524804 >> 172.21.15.31:0/3663524804 conn(0x7f3d9c0705e0 msgr2=0x7f3d9c10a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:00.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.982+0000 7f3da3448700 1 -- 172.21.15.31:0/3663524804 shutdown_connections 2024-04-26T18:03:00.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.982+0000 7f3da3448700 1 -- 172.21.15.31:0/3663524804 wait complete. 2024-04-26T18:03:00.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.983+0000 7f3da3448700 1 Processor -- start 2024-04-26T18:03:00.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.983+0000 7f3da3448700 1 -- start start 2024-04-26T18:03:00.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.983+0000 7f3da3448700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d9c06cd90 0x7f3d9c1a4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:00.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.983+0000 7f3da3448700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d9c06d6b0 0x7f3d9c1a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:00.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.983+0000 7f3da3448700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d9c1184f0 0x7f3d9c1a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:00.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.983+0000 7f3da3448700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c11dfc0 con 0x7f3d9c06d6b0 2024-04-26T18:03:00.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.983+0000 7f3da3448700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c11de40 con 0x7f3d9c1184f0 2024-04-26T18:03:00.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.983+0000 7f3da3448700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c11e140 con 0x7f3d9c06cd90 2024-04-26T18:03:00.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.983+0000 7f3da09e3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d9c06d6b0 0x7f3d9c1a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:00.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.983+0000 7f3da19e5700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d9c1184f0 0x7f3d9c1a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:00.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.983+0000 7f3da09e3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d9c06d6b0 0x7f3d9c1a4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50608/0 (socket says 172.21.15.31:50608) 2024-04-26T18:03:00.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.983+0000 7f3da09e3700 1 -- 172.21.15.31:0/3878851118 learned_addr learned my addr 172.21.15.31:0/3878851118 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:00.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.983+0000 7f3da11e4700 1 --2- 172.21.15.31:0/3878851118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d9c06cd90 0x7f3d9c1a4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:00.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.984+0000 7f3da19e5700 1 -- 172.21.15.31:0/3878851118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d9c06cd90 msgr2=0x7f3d9c1a4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:00.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.984+0000 7f3da19e5700 1 --2- 172.21.15.31:0/3878851118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d9c06cd90 0x7f3d9c1a4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:00.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.984+0000 7f3da19e5700 1 -- 172.21.15.31:0/3878851118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d9c06d6b0 msgr2=0x7f3d9c1a4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:00.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.984+0000 7f3da19e5700 1 --2- 172.21.15.31:0/3878851118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d9c06d6b0 0x7f3d9c1a4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:00.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.984+0000 7f3da19e5700 1 -- 172.21.15.31:0/3878851118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d9c1a9540 con 0x7f3d9c1184f0 2024-04-26T18:03:00.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.984+0000 7f3da19e5700 1 --2- 172.21.15.31:0/3878851118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d9c1184f0 0x7f3d9c1a8e00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400d720 tx=0x7f3d9400da30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:00.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.984+0000 7f3d9a7fc700 1 -- 172.21.15.31:0/3878851118 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94016070 con 0x7f3d9c1184f0 2024-04-26T18:03:00.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.984+0000 7f3d9a7fc700 1 -- 172.21.15.31:0/3878851118 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d9400f690 con 0x7f3d9c1184f0 2024-04-26T18:03:00.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.984+0000 7f3da3448700 1 -- 172.21.15.31:0/3878851118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d9c1a9830 con 0x7f3d9c1184f0 2024-04-26T18:03:00.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.984+0000 7f3d9a7fc700 1 -- 172.21.15.31:0/3878851118 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94003a40 con 0x7f3d9c1184f0 2024-04-26T18:03:00.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.984+0000 7f3da3448700 1 -- 172.21.15.31:0/3878851118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d9c1b10b0 con 0x7f3d9c1184f0 2024-04-26T18:03:00.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.985+0000 7f3da3448700 1 -- 172.21.15.31:0/3878851118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d9c06a110 con 0x7f3d9c1184f0 2024-04-26T18:03:00.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.987+0000 7f3d9a7fc700 1 -- 172.21.15.31:0/3878851118 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3d94003be0 con 0x7f3d9c1184f0 2024-04-26T18:03:00.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.988+0000 7f3d9a7fc700 1 --2- 172.21.15.31:0/3878851118 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3d840a3f00 0x7f3d840a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:00.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.988+0000 7f3d9a7fc700 1 -- 172.21.15.31:0/3878851118 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f3d940cfd10 con 0x7f3d9c1184f0 2024-04-26T18:03:00.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.988+0000 7f3da11e4700 1 --2- 172.21.15.31:0/3878851118 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3d840a3f00 0x7f3d840a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:00.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.989+0000 7f3da11e4700 1 --2- 172.21.15.31:0/3878851118 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3d840a3f00 0x7f3d840a63c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c00f540 tx=0x7f3d8c00b040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:00.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:00.993+0000 7f3d9a7fc700 1 -- 172.21.15.31:0/3878851118 <== mon.2 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d9401f070 con 0x7f3d9c1184f0 2024-04-26T18:03:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:01.315+0000 7f3da3448700 1 -- 172.21.15.31:0/3878851118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d9c02e490 con 0x7f3d9c1184f0 2024-04-26T18:03:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:01.316+0000 7f3d9a7fc700 1 -- 172.21.15.31:0/3878851118 <== mon.2 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11877 (secure 0 0 0) 0x7f3d9409c400 con 0x7f3d9c1184f0 2024-04-26T18:03:01.316 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:03:01.316 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T18:00:06.847155+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24446,24457,24473]},"id":1}]} 2024-04-26T18:03:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:01.319+0000 7f3da3448700 1 -- 172.21.15.31:0/3878851118 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3d840a3f00 msgr2=0x7f3d840a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:01.319+0000 7f3da3448700 1 --2- 172.21.15.31:0/3878851118 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3d840a3f00 0x7f3d840a63c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c00f540 tx=0x7f3d8c00b040 comp rx=0 tx=0).stop 2024-04-26T18:03:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:01.319+0000 7f3da3448700 1 -- 172.21.15.31:0/3878851118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d9c1184f0 msgr2=0x7f3d9c1a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:01.319+0000 7f3da3448700 1 --2- 172.21.15.31:0/3878851118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d9c1184f0 0x7f3d9c1a8e00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400d720 tx=0x7f3d9400da30 comp rx=0 tx=0).stop 2024-04-26T18:03:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:01.319+0000 7f3da3448700 1 -- 172.21.15.31:0/3878851118 shutdown_connections 2024-04-26T18:03:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:01.319+0000 7f3da3448700 1 --2- 172.21.15.31:0/3878851118 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3d840a3f00 0x7f3d840a63c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:01.319+0000 7f3da3448700 1 --2- 172.21.15.31:0/3878851118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d9c1184f0 0x7f3d9c1a8e00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:01.319+0000 7f3da3448700 1 --2- 172.21.15.31:0/3878851118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d9c06cd90 0x7f3d9c1a4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:01.319+0000 7f3da3448700 1 --2- 172.21.15.31:0/3878851118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d9c06d6b0 0x7f3d9c1a4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:01.319+0000 7f3da3448700 1 -- 172.21.15.31:0/3878851118 >> 172.21.15.31:0/3878851118 conn(0x7f3d9c0705e0 msgr2=0x7f3d9c071c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:01.319+0000 7f3da3448700 1 -- 172.21.15.31:0/3878851118 shutdown_connections 2024-04-26T18:03:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:01.319+0000 7f3da3448700 1 -- 172.21.15.31:0/3878851118 wait complete. 2024-04-26T18:03:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 32 2024-04-26T18:03:01.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:01 smithi186 ceph-mon[32797]: pgmap v198: 97 pgs: 97 active+clean; 3.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.3 MiB/s wr, 734 op/s 2024-04-26T18:03:01.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:01 smithi031 ceph-mon[29164]: pgmap v198: 97 pgs: 97 active+clean; 3.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.3 MiB/s wr, 734 op/s 2024-04-26T18:03:01.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:01 smithi097 ceph-mon[32419]: pgmap v198: 97 pgs: 97 active+clean; 3.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.3 MiB/s wr, 734 op/s 2024-04-26T18:03:02.253 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24473 quiesce db / --timeout 60 --expiration 91.7 --await-for 42.0' 2024-04-26T18:03:02.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24473 quiesce db / --timeout 60 --expiration 91.7 --await-for 42.0 2024-04-26T18:03:02.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:02 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3878851118' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:03:02.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:02 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3878851118' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:03:02.561 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:03:02.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:02 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3878851118' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:03:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 340.0 s, 1860.7 tps, lat 51.365 ms stddev 89.846 2024-04-26T18:03:03.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:03 smithi186 ceph-mon[32797]: pgmap v199: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-26T18:03:03.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:03 smithi031 ceph-mon[29164]: pgmap v199: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-26T18:03:03.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:03 smithi097 ceph-mon[32419]: pgmap v199: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-26T18:03:04.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.498+0000 7f025b17b700 1 -- 172.21.15.31:0/3318780669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02541094c0 msgr2=0x7f02541098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:04.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.498+0000 7f025b17b700 1 --2- 172.21.15.31:0/3318780669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02541094c0 0x7f02541098a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f0248009990 tx=0x7f0248009ca0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.498+0000 7f025b17b700 1 -- 172.21.15.31:0/3318780669 shutdown_connections 2024-04-26T18:03:04.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.498+0000 7f025b17b700 1 --2- 172.21.15.31:0/3318780669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f025406c370 0x7f0254110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.498+0000 7f025b17b700 1 --2- 172.21.15.31:0/3318780669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02541094c0 0x7f02541098a0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.498+0000 7f025b17b700 1 --2- 172.21.15.31:0/3318780669 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f025406b9d0 0x7f025406be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.498+0000 7f025b17b700 1 -- 172.21.15.31:0/3318780669 >> 172.21.15.31:0/3318780669 conn(0x7f0254079360 msgr2=0x7f0254079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.498+0000 7f025b17b700 1 -- 172.21.15.31:0/3318780669 shutdown_connections 2024-04-26T18:03:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.498+0000 7f025b17b700 1 -- 172.21.15.31:0/3318780669 wait complete. 2024-04-26T18:03:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.499+0000 7f025b17b700 1 Processor -- start 2024-04-26T18:03:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.499+0000 7f025b17b700 1 -- start start 2024-04-26T18:03:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.499+0000 7f025b17b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f025406b9d0 0x7f02541023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.499+0000 7f025b17b700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f025406c370 0x7f0254102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.499+0000 7f025b17b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02541094c0 0x7f0254102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.499+0000 7f025b17b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f02541135d0 con 0x7f02541094c0 2024-04-26T18:03:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.499+0000 7f025b17b700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0254113450 con 0x7f025406c370 2024-04-26T18:03:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.499+0000 7f025b17b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0254113750 con 0x7f025406b9d0 2024-04-26T18:03:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.499+0000 7f0259718700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02541094c0 0x7f0254102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.499+0000 7f0253fff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f025406c370 0x7f0254102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.499+0000 7f0253fff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f025406c370 0x7f0254102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:50062/0 (socket says 172.21.15.31:50062) 2024-04-26T18:03:04.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.499+0000 7f0258f17700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f025406b9d0 0x7f02541023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.499+0000 7f0259718700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02541094c0 0x7f0254102e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50622/0 (socket says 172.21.15.31:50622) 2024-04-26T18:03:04.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.499+0000 7f0253fff700 1 -- 172.21.15.31:0/3065698175 learned_addr learned my addr 172.21.15.31:0/3065698175 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:04.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.500+0000 7f0253fff700 1 -- 172.21.15.31:0/3065698175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f025406b9d0 msgr2=0x7f02541023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:04.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.500+0000 7f0253fff700 1 --2- 172.21.15.31:0/3065698175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f025406b9d0 0x7f02541023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.500+0000 7f0253fff700 1 -- 172.21.15.31:0/3065698175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02541094c0 msgr2=0x7f0254102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:04.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.500+0000 7f0253fff700 1 --2- 172.21.15.31:0/3065698175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02541094c0 0x7f0254102e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.500+0000 7f0253fff700 1 -- 172.21.15.31:0/3065698175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02541a7f60 con 0x7f025406c370 2024-04-26T18:03:04.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.500+0000 7f0258f17700 1 --2- 172.21.15.31:0/3065698175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f025406b9d0 0x7f02541023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:03:04.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.500+0000 7f0253fff700 1 --2- 172.21.15.31:0/3065698175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f025406c370 0x7f0254102920 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f024000e9a0 tx=0x7f024000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:04.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.500+0000 7f0251ffb700 1 -- 172.21.15.31:0/3065698175 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024000ca10 con 0x7f025406c370 2024-04-26T18:03:04.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.500+0000 7f025b17b700 1 -- 172.21.15.31:0/3065698175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02541a8250 con 0x7f025406c370 2024-04-26T18:03:04.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.500+0000 7f0251ffb700 1 -- 172.21.15.31:0/3065698175 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f024000cbb0 con 0x7f025406c370 2024-04-26T18:03:04.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.500+0000 7f0251ffb700 1 -- 172.21.15.31:0/3065698175 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0240018680 con 0x7f025406c370 2024-04-26T18:03:04.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.500+0000 7f025b17b700 1 -- 172.21.15.31:0/3065698175 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02541a87e0 con 0x7f025406c370 2024-04-26T18:03:04.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.503+0000 7f0251ffb700 1 -- 172.21.15.31:0/3065698175 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0240010430 con 0x7f025406c370 2024-04-26T18:03:04.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.503+0000 7f0251ffb700 1 --2- 172.21.15.31:0/3065698175 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f02440a3ed0 0x7f02440a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.504+0000 7f0258f17700 1 --2- 172.21.15.31:0/3065698175 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f02440a3ed0 0x7f02440a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.504+0000 7f0251ffb700 1 -- 172.21.15.31:0/3065698175 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f02400d08f0 con 0x7f025406c370 2024-04-26T18:03:04.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.504+0000 7f0258f17700 1 --2- 172.21.15.31:0/3065698175 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f02440a3ed0 0x7f02440a6390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f024800b2e0 tx=0x7f024801a040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:04.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.504+0000 7f0265a0db80 1 Processor -- start 2024-04-26T18:03:04.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.504+0000 7f0265a0db80 1 -- start start 2024-04-26T18:03:04.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0265a0db80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x562055070610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0265a0db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 0x56205507b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0265a0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x56205507ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0265a0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562054fc38d0 con 0x56205507ba00 2024-04-26T18:03:04.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0265a0db80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x562054f62770 con 0x562055070230 2024-04-26T18:03:04.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0265a0db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562054fbf890 con 0x562055070b50 2024-04-26T18:03:04.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0259718700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x56205507ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0259718700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x56205507ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50636/0 (socket says 172.21.15.31:50636) 2024-04-26T18:03:04.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0253fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 0x56205507b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0259718700 1 -- 172.21.15.31:0/2009397801 learned_addr learned my addr 172.21.15.31:0/2009397801 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:04.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0258f17700 1 --2- 172.21.15.31:0/2009397801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x562055070610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0253fff700 1 -- 172.21.15.31:0/2009397801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 msgr2=0x562055070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:04.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0253fff700 1 --2- 172.21.15.31:0/2009397801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x562055070610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0253fff700 1 -- 172.21.15.31:0/2009397801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 msgr2=0x56205507ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:04.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0253fff700 1 --2- 172.21.15.31:0/2009397801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x56205507ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0253fff700 1 -- 172.21.15.31:0/2009397801 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562054f10a60 con 0x562055070b50 2024-04-26T18:03:04.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.505+0000 7f0253fff700 1 --2- 172.21.15.31:0/2009397801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 0x56205507b4c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f02400ce090 tx=0x7f024009d130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2573340cc6f01ad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:04.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.506+0000 7f023effd700 1 -- 172.21.15.31:0/2009397801 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02400a5550 con 0x562055070b50 2024-04-26T18:03:04.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.506+0000 7f023effd700 1 -- 172.21.15.31:0/2009397801 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f024009abe0 con 0x562055070b50 2024-04-26T18:03:04.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.506+0000 7f023effd700 1 -- 172.21.15.31:0/2009397801 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02400a5320 con 0x562055070b50 2024-04-26T18:03:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.506+0000 7f0265a0db80 1 -- 172.21.15.31:0/2009397801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 msgr2=0x56205507b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.506+0000 7f0265a0db80 1 --2- 172.21.15.31:0/2009397801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 0x56205507b4c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f02400ce090 tx=0x7f024009d130 comp rx=0 tx=0).stop 2024-04-26T18:03:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.506+0000 7f0265a0db80 1 -- 172.21.15.31:0/2009397801 shutdown_connections 2024-04-26T18:03:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.506+0000 7f0265a0db80 1 --2- 172.21.15.31:0/2009397801 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x562055070610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.506+0000 7f0265a0db80 1 --2- 172.21.15.31:0/2009397801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x56205507ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.506+0000 7f0265a0db80 1 --2- 172.21.15.31:0/2009397801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 0x56205507b4c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.506+0000 7f0265a0db80 1 -- 172.21.15.31:0/2009397801 >> 172.21.15.31:0/2009397801 conn(0x562054f9ded0 msgr2=0x56205506c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.507+0000 7f0265a0db80 1 -- 172.21.15.31:0/2009397801 shutdown_connections 2024-04-26T18:03:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.507+0000 7f0265a0db80 1 -- 172.21.15.31:0/2009397801 wait complete. 2024-04-26T18:03:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.507+0000 7f0265a0db80 1 -- start start 2024-04-26T18:03:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.507+0000 7f0265a0db80 1 Processor -- start 2024-04-26T18:03:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.507+0000 7f0265a0db80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x56205507b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.507+0000 7f0265a0db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 0x562055078540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.507+0000 7f0265a0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x562055078a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.507+0000 7f0265a0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56205507e5d0 con 0x56205507ba00 2024-04-26T18:03:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.507+0000 7f0265a0db80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x562054f29ae0 con 0x562055070230 2024-04-26T18:03:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.507+0000 7f0265a0db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56205507e750 con 0x562055070b50 2024-04-26T18:03:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.508+0000 7f0265a0db80 10 client.0 _ll_register_callbacks cb 0x562054f57710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:03:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.508+0000 7f0258f17700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x56205507b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.508+0000 7f0258f17700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x56205507b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:50084/0 (socket says 172.21.15.31:50084) 2024-04-26T18:03:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.508+0000 7f0259718700 1 --2- 172.21.15.31:0/615346494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x562055078a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.508+0000 7f0258f17700 1 -- 172.21.15.31:0/615346494 learned_addr learned my addr 172.21.15.31:0/615346494 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.508+0000 7f0253fff700 1 --2- 172.21.15.31:0/615346494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 0x562055078540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.508+0000 7f0259718700 1 -- 172.21.15.31:0/615346494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 msgr2=0x562055078540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.508+0000 7f0259718700 1 --2- 172.21.15.31:0/615346494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 0x562055078540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.508+0000 7f0259718700 1 -- 172.21.15.31:0/615346494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 msgr2=0x56205507b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.508+0000 7f0259718700 1 --2- 172.21.15.31:0/615346494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x56205507b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.508+0000 7f0259718700 1 -- 172.21.15.31:0/615346494 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620550792c0 con 0x56205507ba00 2024-04-26T18:03:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.508+0000 7f0253fff700 1 --2- 172.21.15.31:0/615346494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 0x562055078540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:03:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.508+0000 7f0259718700 1 --2- 172.21.15.31:0/615346494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x562055078a80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x56205506db90 tx=0x7f024c004910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe125153d0ed5508 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.508+0000 7f0233fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:03:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.509+0000 7f0233fff700 1 -- 172.21.15.31:0/615346494 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c0147c0 con 0x56205507ba00 2024-04-26T18:03:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.509+0000 7f0233fff700 1 -- 172.21.15.31:0/615346494 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f024c014960 con 0x56205507ba00 2024-04-26T18:03:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.509+0000 7f0265a0db80 1 -- 172.21.15.31:0/615346494 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f024009abe0 con 0x56205507ba00 2024-04-26T18:03:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.509+0000 7f0233fff700 1 -- 172.21.15.31:0/615346494 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f024c014b80 con 0x56205507ba00 2024-04-26T18:03:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.509+0000 7f0265a0db80 10 client.15999 fetch_fsmap learned FSMap version 32 2024-04-26T18:03:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.509+0000 7f0265a0db80 1 -- 172.21.15.31:0/615346494 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562055089180 con 0x56205507ba00 2024-04-26T18:03:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.509+0000 7f0233fff700 1 -- 172.21.15.31:0/615346494 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f024009abe0 con 0x56205507ba00 2024-04-26T18:03:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.509+0000 7f0233fff700 1 -- 172.21.15.31:0/615346494 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f024c028480 con 0x56205507ba00 2024-04-26T18:03:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.510+0000 7f0265a0db80 10 client.15999 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:03:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.510+0000 7f0265a0db80 10 client.15999 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T18:03:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.510+0000 7f0265a0db80 20 client.15999 populate_metadata read hostname 'smithi031' 2024-04-26T18:03:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.510+0000 7f0265a0db80 1 --2- 172.21.15.31:0/615346494 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56205508b760 0x56205508db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.510+0000 7f0265a0db80 4 client.15999 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:03:04.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.510+0000 7f0265a0db80 1 -- 172.21.15.31:0/615346494 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562054fd12b0 con 0x56205508b760 2024-04-26T18:03:04.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.510+0000 7f0258f17700 1 --2- 172.21.15.31:0/615346494 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56205508b760 0x56205508db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.510+0000 7f0258f17700 1 --2- 172.21.15.31:0/615346494 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56205508b760 0x56205508db50 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89b425a0fd569c55 server_cookie=307227f86b746c53 in_seq=0 out_seq=0 2024-04-26T18:03:04.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.510+0000 7f0233fff700 10 client.15999 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:04.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.511+0000 7f0233fff700 1 -- 172.21.15.31:0/615346494 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562054fd12b0 con 0x56205508b760 2024-04-26T18:03:04.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.511+0000 7f0233fff700 10 client.15999 handle_command_reply: tid=0 2024-04-26T18:03:04.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.511+0000 7f0265a0db80 1 client.15999 shutdown 2024-04-26T18:03:04.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.511+0000 7f0265a0db80 1 -- 172.21.15.31:0/615346494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 msgr2=0x562055078a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:04.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.511+0000 7f0265a0db80 1 --2- 172.21.15.31:0/615346494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x562055078a80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x56205506db90 tx=0x7f024c004910 comp rx=0 tx=0).stop 2024-04-26T18:03:04.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.511+0000 7f0265a0db80 1 -- 172.21.15.31:0/615346494 shutdown_connections 2024-04-26T18:03:04.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.511+0000 7f0265a0db80 1 --2- 172.21.15.31:0/615346494 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56205508b760 0x56205508db50 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.511+0000 7f0233fff700 0 client.15999 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:04.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.511+0000 7f0265a0db80 1 --2- 172.21.15.31:0/615346494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x56205507b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.511+0000 7f0265a0db80 1 --2- 172.21.15.31:0/615346494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x562055078a80 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.511+0000 7f0265a0db80 1 --2- 172.21.15.31:0/615346494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 0x562055078540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.511+0000 7f0265a0db80 1 -- 172.21.15.31:0/615346494 >> 172.21.15.31:0/615346494 conn(0x562054f9ded0 msgr2=0x56205506c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:04.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.512+0000 7f0265a0db80 1 -- 172.21.15.31:0/615346494 shutdown_connections 2024-04-26T18:03:04.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.512+0000 7f0265a0db80 1 -- 172.21.15.31:0/615346494 wait complete. 2024-04-26T18:03:04.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.512+0000 7f0265a0db80 20 client.15999 trim_cache size 0 max 16384 2024-04-26T18:03:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.535+0000 7f0265a0db80 1 Processor -- start 2024-04-26T18:03:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.535+0000 7f0265a0db80 1 -- start start 2024-04-26T18:03:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.535+0000 7f0265a0db80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x562055073da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.535+0000 7f0265a0db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 0x562055077e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.535+0000 7f0265a0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x5620550874d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.535+0000 7f0265a0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f02541b26f0 con 0x56205507ba00 2024-04-26T18:03:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.535+0000 7f0265a0db80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f02541aff20 con 0x562055070230 2024-04-26T18:03:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.535+0000 7f0265a0db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f02541add70 con 0x562055070b50 2024-04-26T18:03:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.535+0000 7f0259718700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x5620550874d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.535+0000 7f0259718700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x5620550874d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50652/0 (socket says 172.21.15.31:50652) 2024-04-26T18:03:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.535+0000 7f0259718700 1 -- 172.21.15.31:0/2849328879 learned_addr learned my addr 172.21.15.31:0/2849328879 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.535+0000 7f0258f17700 1 --2- 172.21.15.31:0/2849328879 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x562055073da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.535+0000 7f0253fff700 1 --2- 172.21.15.31:0/2849328879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 0x562055077e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.536+0000 7f0258f17700 1 -- 172.21.15.31:0/2849328879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 msgr2=0x562055077e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.536+0000 7f0258f17700 1 --2- 172.21.15.31:0/2849328879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 0x562055077e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.536+0000 7f0258f17700 1 -- 172.21.15.31:0/2849328879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 msgr2=0x5620550874d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.536+0000 7f0258f17700 1 --2- 172.21.15.31:0/2849328879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x5620550874d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.536+0000 7f0258f17700 1 -- 172.21.15.31:0/2849328879 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562055076af0 con 0x562055070230 2024-04-26T18:03:04.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.536+0000 7f0259718700 1 --2- 172.21.15.31:0/2849328879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x5620550874d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:03:04.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.536+0000 7f0258f17700 1 --2- 172.21.15.31:0/2849328879 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x562055073da0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x56205508e390 tx=0x7f024801f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bacd458a89eec656 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:04.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.536+0000 7f023e7fc700 1 -- 172.21.15.31:0/2849328879 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02480049d0 con 0x562055070230 2024-04-26T18:03:04.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.536+0000 7f023e7fc700 1 -- 172.21.15.31:0/2849328879 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0248039070 con 0x562055070230 2024-04-26T18:03:04.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.536+0000 7f023e7fc700 1 -- 172.21.15.31:0/2849328879 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0248026430 con 0x562055070230 2024-04-26T18:03:04.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.537+0000 7f0265a0db80 1 -- 172.21.15.31:0/2849328879 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 msgr2=0x562055073da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:04.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.537+0000 7f0265a0db80 1 --2- 172.21.15.31:0/2849328879 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x562055073da0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x56205508e390 tx=0x7f024801f000 comp rx=0 tx=0).stop 2024-04-26T18:03:04.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.537+0000 7f0265a0db80 1 -- 172.21.15.31:0/2849328879 shutdown_connections 2024-04-26T18:03:04.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.537+0000 7f0265a0db80 1 --2- 172.21.15.31:0/2849328879 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x562055073da0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.537+0000 7f0265a0db80 1 --2- 172.21.15.31:0/2849328879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56205507ba00 0x5620550874d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.537+0000 7f0265a0db80 1 --2- 172.21.15.31:0/2849328879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562055070b50 0x562055077e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.537+0000 7f0265a0db80 1 -- 172.21.15.31:0/2849328879 >> 172.21.15.31:0/2849328879 conn(0x562054f9ded0 msgr2=0x5620550729c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.537+0000 7f0265a0db80 1 -- 172.21.15.31:0/2849328879 shutdown_connections 2024-04-26T18:03:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.537+0000 7f0265a0db80 1 -- 172.21.15.31:0/2849328879 wait complete. 2024-04-26T18:03:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.537+0000 7f0265a0db80 1 -- start start 2024-04-26T18:03:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0265a0db80 1 Processor -- start 2024-04-26T18:03:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0265a0db80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x562055083bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0265a0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562055070b50 0x5620550809b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0265a0db80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56205507ba00 0x562055080ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0265a0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f02541ae3e0 con 0x562055070b50 2024-04-26T18:03:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0265a0db80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f02541af850 con 0x562055070230 2024-04-26T18:03:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0265a0db80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f02541aeab0 con 0x56205507ba00 2024-04-26T18:03:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0265a0db80 10 client.0 _ll_register_callbacks cb 0x562054f57710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:03:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0253fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562055070b50 0x5620550809b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0253fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562055070b50 0x5620550809b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50664/0 (socket says 172.21.15.31:50664) 2024-04-26T18:03:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0253fff700 1 -- 172.21.15.31:0/2839677792 learned_addr learned my addr 172.21.15.31:0/2839677792 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0259718700 1 --2- 172.21.15.31:0/2839677792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56205507ba00 0x562055080ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0258f17700 1 --2- 172.21.15.31:0/2839677792 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x562055083bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0253fff700 1 -- 172.21.15.31:0/2839677792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56205507ba00 msgr2=0x562055080ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0253fff700 1 --2- 172.21.15.31:0/2839677792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56205507ba00 0x562055080ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0253fff700 1 -- 172.21.15.31:0/2839677792 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 msgr2=0x562055083bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0253fff700 1 --2- 172.21.15.31:0/2839677792 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x562055083bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.538+0000 7f0253fff700 1 -- 172.21.15.31:0/2839677792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562055081460 con 0x562055070b50 2024-04-26T18:03:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.539+0000 7f0253fff700 1 --2- 172.21.15.31:0/2839677792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562055070b50 0x5620550809b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0240098d40 tx=0x7f02400a0db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ff9f9cd891dde2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.539+0000 7f0233fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:03:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.539+0000 7f0233fff700 1 -- 172.21.15.31:0/2839677792 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024009cd40 con 0x562055070b50 2024-04-26T18:03:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.539+0000 7f0233fff700 1 -- 172.21.15.31:0/2839677792 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f02400a50c0 con 0x562055070b50 2024-04-26T18:03:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.539+0000 7f0265a0db80 1 -- 172.21.15.31:0/2839677792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0248039070 con 0x562055070b50 2024-04-26T18:03:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.539+0000 7f0233fff700 1 -- 172.21.15.31:0/2839677792 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f02400ba030 con 0x562055070b50 2024-04-26T18:03:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.539+0000 7f0265a0db80 10 client.16005 fetch_fsmap learned FSMap version 32 2024-04-26T18:03:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.539+0000 7f0265a0db80 1 -- 172.21.15.31:0/2839677792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5620550f2a20 con 0x562055070b50 2024-04-26T18:03:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.539+0000 7f0233fff700 1 -- 172.21.15.31:0/2839677792 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0248039070 con 0x562055070b50 2024-04-26T18:03:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.540+0000 7f0233fff700 1 -- 172.21.15.31:0/2839677792 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f0240098630 con 0x562055070b50 2024-04-26T18:03:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.540+0000 7f0265a0db80 10 client.16005 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T18:03:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.540+0000 7f0265a0db80 10 client.16005 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T18:03:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.540+0000 7f0265a0db80 20 client.16005 populate_metadata read hostname 'smithi031' 2024-04-26T18:03:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.540+0000 7f0265a0db80 1 --2- 172.21.15.31:0/2839677792 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56205508b760 0x5620550f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.540+0000 7f0265a0db80 4 client.16005 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 91.7, "await_for": 42.0}] 2024-04-26T18:03:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.540+0000 7f0265a0db80 1 -- 172.21.15.31:0/2839677792 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 91.7, "await_for": 42.0}) -- 0x562054fcb350 con 0x56205508b760 2024-04-26T18:03:04.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.540+0000 7f0258f17700 1 --2- 172.21.15.31:0/2839677792 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56205508b760 0x5620550f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:04.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.540+0000 7f0233fff700 10 client.16005 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:04.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.540+0000 7f0258f17700 1 --2- 172.21.15.31:0/2839677792 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56205508b760 0x5620550f4dd0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38bf35aebfbecf18 server_cookie=b9527192c1f65c2 in_seq=0 out_seq=0 2024-04-26T18:03:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.777+0000 7f0233fff700 1 -- 172.21.15.31:0/2839677792 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x562054fcb350 con 0x56205508b760 2024-04-26T18:03:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.777+0000 7f0233fff700 10 client.16005 handle_command_reply: tid=0 2024-04-26T18:03:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.777+0000 7f0265a0db80 1 client.16005 shutdown 2024-04-26T18:03:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.777+0000 7f0265a0db80 1 -- 172.21.15.31:0/2839677792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562055070b50 msgr2=0x5620550809b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.777+0000 7f0265a0db80 1 --2- 172.21.15.31:0/2839677792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562055070b50 0x5620550809b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0240098d40 tx=0x7f02400a0db0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.777+0000 7f0265a0db80 1 -- 172.21.15.31:0/2839677792 shutdown_connections 2024-04-26T18:03:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.777+0000 7f0265a0db80 1 --2- 172.21.15.31:0/2839677792 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56205508b760 0x5620550f4dd0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.777+0000 7f0233fff700 0 client.16005 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.777+0000 7f0265a0db80 1 --2- 172.21.15.31:0/2839677792 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562055070230 0x562055083bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.777+0000 7f0265a0db80 1 --2- 172.21.15.31:0/2839677792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56205507ba00 0x562055080ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.777+0000 7f0265a0db80 1 --2- 172.21.15.31:0/2839677792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562055070b50 0x5620550809b0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.777+0000 7f0265a0db80 1 -- 172.21.15.31:0/2839677792 >> 172.21.15.31:0/2839677792 conn(0x562054f9ded0 msgr2=0x562055076e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.778+0000 7f0265a0db80 1 -- 172.21.15.31:0/2839677792 shutdown_connections 2024-04-26T18:03:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.778+0000 7f0265a0db80 1 -- 172.21.15.31:0/2839677792 wait complete. 2024-04-26T18:03:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.778+0000 7f0265a0db80 20 client.16005 trim_cache size 0 max 16384 2024-04-26T18:03:04.778 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T18:03:04.778 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 32, 2024-04-26T18:03:04.778 INFO:teuthology.orchestra.run.smithi031.stdout: "leader": 24473, 2024-04-26T18:03:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "set_version": 18, 2024-04-26T18:03:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "sets": { 2024-04-26T18:03:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "13a7907": { 2024-04-26T18:03:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "version": 18, 2024-04-26T18:03:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "age_ref": 0.0, 2024-04-26T18:03:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T18:03:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "QUIESCED", 2024-04-26T18:03:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T18:03:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:03:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "timeout": 60.0, 2024-04-26T18:03:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "expiration": 91.7, 2024-04-26T18:03:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "members": { 2024-04-26T18:03:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "file:/": { 2024-04-26T18:03:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "excluded": false, 2024-04-26T18:03:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T18:03:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "QUIESCED", 2024-04-26T18:03:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T18:03:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:03:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:03:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:03:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:03:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:03:04.780 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T18:03:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.778+0000 7f025b17b700 1 -- 172.21.15.31:0/3065698175 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f02440a3ed0 msgr2=0x7f02440a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.778+0000 7f025b17b700 1 --2- 172.21.15.31:0/3065698175 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f02440a3ed0 0x7f02440a6390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f024800b2e0 tx=0x7f024801a040 comp rx=0 tx=0).stop 2024-04-26T18:03:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.778+0000 7f025b17b700 1 -- 172.21.15.31:0/3065698175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f025406c370 msgr2=0x7f0254102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.778+0000 7f025b17b700 1 --2- 172.21.15.31:0/3065698175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f025406c370 0x7f0254102920 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f024000e9a0 tx=0x7f024000ecb0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.778+0000 7f025b17b700 1 -- 172.21.15.31:0/3065698175 shutdown_connections 2024-04-26T18:03:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.778+0000 7f025b17b700 1 --2- 172.21.15.31:0/3065698175 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f02440a3ed0 0x7f02440a6390 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.778+0000 7f025b17b700 1 --2- 172.21.15.31:0/3065698175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02541094c0 0x7f0254102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.778+0000 7f025b17b700 1 --2- 172.21.15.31:0/3065698175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f025406b9d0 0x7f02541023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.778+0000 7f025b17b700 1 --2- 172.21.15.31:0/3065698175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f025406c370 0x7f0254102920 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.778+0000 7f025b17b700 1 -- 172.21.15.31:0/3065698175 >> 172.21.15.31:0/3065698175 conn(0x7f0254079360 msgr2=0x7f02541003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.779+0000 7f025b17b700 1 -- 172.21.15.31:0/3065698175 shutdown_connections 2024-04-26T18:03:04.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:04.779+0000 7f025b17b700 1 -- 172.21.15.31:0/3065698175 wait complete. 2024-04-26T18:03:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:05 smithi186 ceph-mon[32797]: pgmap v200: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 7.9 MiB/s wr, 637 op/s 2024-04-26T18:03:05.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:05 smithi031 ceph-mon[29164]: pgmap v200: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 7.9 MiB/s wr, 637 op/s 2024-04-26T18:03:05.597 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '13a7907', quiesce took 7.1 seconds. Will release after: 31.7 2024-04-26T18:03:05.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:05 smithi097 ceph-mon[32419]: pgmap v200: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 7.9 MiB/s wr, 637 op/s 2024-04-26T18:03:07.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:07 smithi186 ceph-mon[32797]: pgmap v201: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 8.5 MiB/s wr, 740 op/s 2024-04-26T18:03:07.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:07 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:03:07.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:07 smithi186 ceph-mon[32797]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T18:03:07.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:07 smithi031 ceph-mon[29164]: pgmap v201: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 8.5 MiB/s wr, 740 op/s 2024-04-26T18:03:07.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:07 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:03:07.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:07 smithi031 ceph-mon[29164]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T18:03:07.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:07 smithi097 ceph-mon[32419]: pgmap v201: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 8.5 MiB/s wr, 740 op/s 2024-04-26T18:03:07.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:07 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:03:07.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:07 smithi097 ceph-mon[32419]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T18:03:08.098 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 345.0 s, 103.1 tps, lat 327.989 ms stddev 187.024 2024-04-26T18:03:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.505+0000 7f31b60dc640 1 -- 172.21.15.31:0/2497966277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31b01044e0 msgr2=0x7f31b010c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.505+0000 7f31b60dc640 1 --2- 172.21.15.31:0/2497966277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31b01044e0 0x7f31b010c9c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f31a4009900 tx=0x7f31a402a510 comp rx=0 tx=0).stop 2024-04-26T18:03:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.505+0000 7f31b60dc640 1 -- 172.21.15.31:0/2497966277 shutdown_connections 2024-04-26T18:03:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.505+0000 7f31b60dc640 1 --2- 172.21.15.31:0/2497966277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31b010cf00 0x7f31b010f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.505+0000 7f31b60dc640 1 --2- 172.21.15.31:0/2497966277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31b01044e0 0x7f31b010c9c0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.505+0000 7f31b60dc640 1 --2- 172.21.15.31:0/2497966277 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31b0103bc0 0x7f31b0103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.505+0000 7f31b60dc640 1 -- 172.21.15.31:0/2497966277 >> 172.21.15.31:0/2497966277 conn(0x7f31b00789b0 msgr2=0x7f31b0078dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.505+0000 7f31b60dc640 1 -- 172.21.15.31:0/2497966277 shutdown_connections 2024-04-26T18:03:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.505+0000 7f31b60dc640 1 -- 172.21.15.31:0/2497966277 wait complete. 2024-04-26T18:03:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.506+0000 7f31b60dc640 1 Processor -- start 2024-04-26T18:03:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.506+0000 7f31b60dc640 1 -- start start 2024-04-26T18:03:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.506+0000 7f31b60dc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31b0103bc0 0x7f31b0199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.506+0000 7f31b60dc640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31b01044e0 0x7f31b0199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.506+0000 7f31b60dc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31b010cf00 0x7f31b019d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.506+0000 7f31b60dc640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f31b0112bb0 con 0x7f31b0103bc0 2024-04-26T18:03:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.506+0000 7f31b60dc640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f31b0112a30 con 0x7f31b01044e0 2024-04-26T18:03:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.506+0000 7f31b60dc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f31b0112d30 con 0x7f31b010cf00 2024-04-26T18:03:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.506+0000 7f31af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31b0103bc0 0x7f31b0199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.506+0000 7f31af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31b0103bc0 0x7f31b0199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40390/0 (socket says 172.21.15.31:40390) 2024-04-26T18:03:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.506+0000 7f31af7fe640 1 -- 172.21.15.31:0/543858990 learned_addr learned my addr 172.21.15.31:0/543858990 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.506+0000 7f31aeffd640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31b01044e0 0x7f31b0199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.506+0000 7f31affff640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31b010cf00 0x7f31b019d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.506+0000 7f31af7fe640 1 -- 172.21.15.31:0/543858990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31b010cf00 msgr2=0x7f31b019d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.507+0000 7f31af7fe640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31b010cf00 0x7f31b019d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.507+0000 7f31af7fe640 1 -- 172.21.15.31:0/543858990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31b01044e0 msgr2=0x7f31b0199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.507+0000 7f31af7fe640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31b01044e0 0x7f31b0199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.507+0000 7f31af7fe640 1 -- 172.21.15.31:0/543858990 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31b019e020 con 0x7f31b0103bc0 2024-04-26T18:03:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.507+0000 7f31aeffd640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31b01044e0 0x7f31b0199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:03:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.507+0000 7f31affff640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31b010cf00 0x7f31b019d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:03:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.507+0000 7f31af7fe640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31b0103bc0 0x7f31b0199030 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f31a000c920 tx=0x7f31a000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.507+0000 7f31acff9640 1 -- 172.21.15.31:0/543858990 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a0007c70 con 0x7f31b0103bc0 2024-04-26T18:03:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.507+0000 7f31acff9640 1 -- 172.21.15.31:0/543858990 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31a0007e10 con 0x7f31b0103bc0 2024-04-26T18:03:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.507+0000 7f31acff9640 1 -- 172.21.15.31:0/543858990 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a0018820 con 0x7f31b0103bc0 2024-04-26T18:03:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.507+0000 7f31b60dc640 1 -- 172.21.15.31:0/543858990 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31b019e310 con 0x7f31b0103bc0 2024-04-26T18:03:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.507+0000 7f31b60dc640 1 -- 172.21.15.31:0/543858990 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31b01a5ba0 con 0x7f31b0103bc0 2024-04-26T18:03:08.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.509+0000 7f31b8df4740 1 Processor -- start 2024-04-26T18:03:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31acff9640 1 -- 172.21.15.31:0/543858990 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f31a000f450 con 0x7f31b0103bc0 2024-04-26T18:03:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31b8df4740 1 -- start start 2024-04-26T18:03:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31b8df4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2598ff00 0x555c259902e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31b8df4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 0x555c2599aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31b8df4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2599b0e0 0x555c2599d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31b8df4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555c258f6460 con 0x555c2599b0e0 2024-04-26T18:03:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31b8df4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x555c258b0590 con 0x555c2598ff00 2024-04-26T18:03:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31b8df4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555c258f2420 con 0x555c25990820 2024-04-26T18:03:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31aeffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2599b0e0 0x555c2599d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31aeffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2599b0e0 0x555c2599d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40406/0 (socket says 172.21.15.31:40406) 2024-04-26T18:03:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31aeffd640 1 -- 172.21.15.31:0/3837319136 learned_addr learned my addr 172.21.15.31:0/3837319136 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31affff640 1 --2- 172.21.15.31:0/3837319136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2598ff00 0x555c259902e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31af7fe640 1 --2- 172.21.15.31:0/3837319136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 0x555c2599aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31aeffd640 1 -- 172.21.15.31:0/3837319136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 msgr2=0x555c2599aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31aeffd640 1 --2- 172.21.15.31:0/3837319136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 0x555c2599aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31aeffd640 1 -- 172.21.15.31:0/3837319136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2598ff00 msgr2=0x555c259902e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31aeffd640 1 --2- 172.21.15.31:0/3837319136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2598ff00 0x555c259902e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.510+0000 7f31aeffd640 1 -- 172.21.15.31:0/3837319136 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c258bd0d0 con 0x555c2599b0e0 2024-04-26T18:03:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.511+0000 7f31acff9640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f31980a3dc0 0x7f31980a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.511+0000 7f31aeffd640 1 --2- 172.21.15.31:0/3837319136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2599b0e0 0x555c2599d4d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f31a402aa30 tx=0x7f31a4032dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768be0b8586f20ad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.511+0000 7f31affff640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f31980a3dc0 0x7f31980a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.511+0000 7f31acff9640 1 -- 172.21.15.31:0/543858990 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8528+0+0 (secure 0 0 0) 0x7f31a00d0a10 con 0x7f31b0103bc0 2024-04-26T18:03:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.511+0000 7f3195ffb640 1 -- 172.21.15.31:0/3837319136 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a403f070 con 0x555c2599b0e0 2024-04-26T18:03:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.511+0000 7f3195ffb640 1 -- 172.21.15.31:0/3837319136 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31a40316a0 con 0x555c2599b0e0 2024-04-26T18:03:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.511+0000 7f3195ffb640 1 -- 172.21.15.31:0/3837319136 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4035070 con 0x555c2599b0e0 2024-04-26T18:03:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.511+0000 7f31b8df4740 1 -- 172.21.15.31:0/3837319136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2599b0e0 msgr2=0x555c2599d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.511+0000 7f31affff640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f31980a3dc0 0x7f31980a6280 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f31a8005fd0 tx=0x7f31a8009500 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.511+0000 7f31b8df4740 1 --2- 172.21.15.31:0/3837319136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2599b0e0 0x555c2599d4d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f31a402aa30 tx=0x7f31a4032dc0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 -- 172.21.15.31:0/3837319136 shutdown_connections 2024-04-26T18:03:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 --2- 172.21.15.31:0/3837319136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2599b0e0 0x555c2599d4d0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 --2- 172.21.15.31:0/3837319136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 0x555c2599aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 --2- 172.21.15.31:0/3837319136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2598ff00 0x555c259902e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 -- 172.21.15.31:0/3837319136 >> 172.21.15.31:0/3837319136 conn(0x555c2598c410 msgr2=0x555c2598c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 -- 172.21.15.31:0/3837319136 shutdown_connections 2024-04-26T18:03:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 -- 172.21.15.31:0/3837319136 wait complete. 2024-04-26T18:03:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 -- start start 2024-04-26T18:03:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 Processor -- start 2024-04-26T18:03:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2598ff00 0x555c25998040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 0x555c25998580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2599b0e0 0x555c25998ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555c2599dcb0 con 0x555c2599b0e0 2024-04-26T18:03:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x555c2587db60 con 0x555c2598ff00 2024-04-26T18:03:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.512+0000 7f31b8df4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555c2599de30 con 0x555c25990820 2024-04-26T18:03:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f31b8df4740 10 client.0 _ll_register_callbacks cb 0x555c258a22b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:03:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f31affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2599b0e0 0x555c25998ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f31affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2599b0e0 0x555c25998ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40416/0 (socket says 172.21.15.31:40416) 2024-04-26T18:03:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f31affff640 1 -- 172.21.15.31:0/4066118928 learned_addr learned my addr 172.21.15.31:0/4066118928 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f31af7fe640 1 --2- 172.21.15.31:0/4066118928 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2598ff00 0x555c25998040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f31aeffd640 1 --2- 172.21.15.31:0/4066118928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 0x555c25998580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f31affff640 1 -- 172.21.15.31:0/4066118928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 msgr2=0x555c25998580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f31affff640 1 --2- 172.21.15.31:0/4066118928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 0x555c25998580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f31affff640 1 -- 172.21.15.31:0/4066118928 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2598ff00 msgr2=0x555c25998040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f31affff640 1 --2- 172.21.15.31:0/4066118928 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2598ff00 0x555c25998040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f31affff640 1 -- 172.21.15.31:0/4066118928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c25608210 con 0x555c2599b0e0 2024-04-26T18:03:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f31af7fe640 1 --2- 172.21.15.31:0/4066118928 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2598ff00 0x555c25998040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:03:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f31affff640 1 --2- 172.21.15.31:0/4066118928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2599b0e0 0x555c25998ac0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f31a8010960 tx=0x7f31a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea6171fd9da7d87 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f318effd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:03:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f318effd640 1 -- 172.21.15.31:0/4066118928 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a801ed40 con 0x555c2599b0e0 2024-04-26T18:03:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f318effd640 1 -- 172.21.15.31:0/4066118928 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8528+0+0 (secure 0 0 0) 0x7f31a801f420 con 0x555c2599b0e0 2024-04-26T18:03:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.513+0000 7f31b8df4740 1 -- 172.21.15.31:0/4066118928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31a40316a0 con 0x555c2599b0e0 2024-04-26T18:03:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.514+0000 7f318effd640 1 -- 172.21.15.31:0/4066118928 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f31a801f620 con 0x555c2599b0e0 2024-04-26T18:03:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.514+0000 7f31b8df4740 10 client.16017 fetch_fsmap learned FSMap version 33 2024-04-26T18:03:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.514+0000 7f31b8df4740 1 -- 172.21.15.31:0/4066118928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555c256d4620 con 0x555c2599b0e0 2024-04-26T18:03:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.514+0000 7f318effd640 1 -- 172.21.15.31:0/4066118928 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31a40316a0 con 0x555c2599b0e0 2024-04-26T18:03:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.514+0000 7f318effd640 1 -- 172.21.15.31:0/4066118928 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f31a801c6e0 con 0x555c2599b0e0 2024-04-26T18:03:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.514+0000 7f31b8df4740 10 client.16017 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T18:03:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.514+0000 7f31b8df4740 10 client.16017 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:03:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.514+0000 7f31b8df4740 20 client.16017 populate_metadata read hostname 'smithi031' 2024-04-26T18:03:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.514+0000 7f31b8df4740 1 --2- 172.21.15.31:0/4066118928 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555c259ab230 0x555c259ad680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.514+0000 7f31b8df4740 4 client.16017 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:03:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.514+0000 7f31b8df4740 1 -- 172.21.15.31:0/4066118928 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c25904070 con 0x555c259ab230 2024-04-26T18:03:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.514+0000 7f31af7fe640 1 --2- 172.21.15.31:0/4066118928 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555c259ab230 0x555c259ad680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.515+0000 7f31af7fe640 1 --2- 172.21.15.31:0/4066118928 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555c259ab230 0x555c259ad680 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb9bc962f319b3d7 server_cookie=48be51e9eaefbcdb in_seq=0 out_seq=0 2024-04-26T18:03:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.515+0000 7f318effd640 10 client.16017 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.515+0000 7f318effd640 1 -- 172.21.15.31:0/4066118928 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555c25904070 con 0x555c259ab230 2024-04-26T18:03:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.516+0000 7f318effd640 10 client.16017 handle_command_reply: tid=0 2024-04-26T18:03:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.516+0000 7f31b8df4740 1 client.16017 shutdown 2024-04-26T18:03:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.516+0000 7f31b8df4740 1 -- 172.21.15.31:0/4066118928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2599b0e0 msgr2=0x555c25998ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.516+0000 7f31b8df4740 1 --2- 172.21.15.31:0/4066118928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2599b0e0 0x555c25998ac0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f31a8010960 tx=0x7f31a8004a40 comp rx=0 tx=0).stop 2024-04-26T18:03:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.516+0000 7f31b8df4740 1 -- 172.21.15.31:0/4066118928 shutdown_connections 2024-04-26T18:03:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.516+0000 7f31b8df4740 1 --2- 172.21.15.31:0/4066118928 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555c259ab230 0x555c259ad680 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.516+0000 7f31b8df4740 1 --2- 172.21.15.31:0/4066118928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2599b0e0 0x555c25998ac0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.516+0000 7f31b8df4740 1 --2- 172.21.15.31:0/4066118928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 0x555c25998580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.516+0000 7f31b8df4740 1 --2- 172.21.15.31:0/4066118928 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2598ff00 0x555c25998040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.516+0000 7f31b8df4740 1 -- 172.21.15.31:0/4066118928 >> 172.21.15.31:0/4066118928 conn(0x555c2598c410 msgr2=0x555c25992140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.516+0000 7f318effd640 0 client.16017 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.516+0000 7f31b8df4740 1 -- 172.21.15.31:0/4066118928 shutdown_connections 2024-04-26T18:03:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.516+0000 7f31b8df4740 1 -- 172.21.15.31:0/4066118928 wait complete. 2024-04-26T18:03:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.516+0000 7f31b8df4740 20 client.16017 trim_cache size 0 max 16384 2024-04-26T18:03:08.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.531+0000 7f31b8df4740 1 Processor -- start 2024-04-26T18:03:08.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.531+0000 7f31b8df4740 1 -- start start 2024-04-26T18:03:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31b8df4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2598ff00 0x555c259a6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31b8df4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c25990820 0x555c25993360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31b8df4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c2599b0e0 0x555c259928c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31b8df4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f31b01b0dc0 con 0x555c2598ff00 2024-04-26T18:03:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31b8df4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f31b01ae630 con 0x555c25990820 2024-04-26T18:03:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31b8df4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f31b01ac480 con 0x555c2599b0e0 2024-04-26T18:03:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2598ff00 0x555c259a6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2598ff00 0x555c259a6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40420/0 (socket says 172.21.15.31:40420) 2024-04-26T18:03:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31af7fe640 1 -- 172.21.15.31:0/2927244773 learned_addr learned my addr 172.21.15.31:0/2927244773 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31affff640 1 --2- 172.21.15.31:0/2927244773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c2599b0e0 0x555c259928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31aeffd640 1 --2- 172.21.15.31:0/2927244773 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c25990820 0x555c25993360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31af7fe640 1 -- 172.21.15.31:0/2927244773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c2599b0e0 msgr2=0x555c259928c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31af7fe640 1 --2- 172.21.15.31:0/2927244773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c2599b0e0 0x555c259928c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31af7fe640 1 -- 172.21.15.31:0/2927244773 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c25990820 msgr2=0x555c25993360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31af7fe640 1 --2- 172.21.15.31:0/2927244773 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c25990820 0x555c25993360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31af7fe640 1 -- 172.21.15.31:0/2927244773 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c25987d60 con 0x555c2598ff00 2024-04-26T18:03:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31affff640 1 --2- 172.21.15.31:0/2927244773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c2599b0e0 0x555c259928c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:03:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31aeffd640 1 --2- 172.21.15.31:0/2927244773 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c25990820 0x555c25993360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:03:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31af7fe640 1 --2- 172.21.15.31:0/2927244773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2598ff00 0x555c259a6b70 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f31a0000c00 tx=0x7f31a00d2e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eefe5d893f8983e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31957fa640 1 -- 172.21.15.31:0/2927244773 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a000de20 con 0x555c2598ff00 2024-04-26T18:03:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31957fa640 1 -- 172.21.15.31:0/2927244773 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31a009d070 con 0x555c2598ff00 2024-04-26T18:03:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.532+0000 7f31957fa640 1 -- 172.21.15.31:0/2927244773 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a000fa90 con 0x555c2598ff00 2024-04-26T18:03:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.533+0000 7f31b8df4740 1 -- 172.21.15.31:0/2927244773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2598ff00 msgr2=0x555c259a6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.533+0000 7f31b8df4740 1 --2- 172.21.15.31:0/2927244773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2598ff00 0x555c259a6b70 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f31a0000c00 tx=0x7f31a00d2e60 comp rx=0 tx=0).stop 2024-04-26T18:03:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.533+0000 7f31b8df4740 1 -- 172.21.15.31:0/2927244773 shutdown_connections 2024-04-26T18:03:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.533+0000 7f31b8df4740 1 --2- 172.21.15.31:0/2927244773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c2599b0e0 0x555c259928c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.533+0000 7f31b8df4740 1 --2- 172.21.15.31:0/2927244773 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c25990820 0x555c25993360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.533+0000 7f31b8df4740 1 --2- 172.21.15.31:0/2927244773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2598ff00 0x555c259a6b70 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.533+0000 7f31b8df4740 1 -- 172.21.15.31:0/2927244773 >> 172.21.15.31:0/2927244773 conn(0x555c2598c410 msgr2=0x555c2599a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.533+0000 7f31b8df4740 1 -- 172.21.15.31:0/2927244773 shutdown_connections 2024-04-26T18:03:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.533+0000 7f31b8df4740 1 -- 172.21.15.31:0/2927244773 wait complete. 2024-04-26T18:03:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.533+0000 7f31b8df4740 1 -- start start 2024-04-26T18:03:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31b8df4740 1 Processor -- start 2024-04-26T18:03:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31b8df4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2598ff00 0x555c25995130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31b8df4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 0x555c25995670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31b8df4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2599b0e0 0x555c25a10800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31b8df4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f31b01acaf0 con 0x555c2598ff00 2024-04-26T18:03:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31b8df4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f31b01adf60 con 0x555c2599b0e0 2024-04-26T18:03:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31b8df4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f31b01ad1c0 con 0x555c25990820 2024-04-26T18:03:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31b8df4740 10 client.0 _ll_register_callbacks cb 0x7f31a8009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:03:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2598ff00 0x555c25995130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2598ff00 0x555c25995130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40434/0 (socket says 172.21.15.31:40434) 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31af7fe640 1 -- 172.21.15.31:0/1244629843 learned_addr learned my addr 172.21.15.31:0/1244629843 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31affff640 1 --2- 172.21.15.31:0/1244629843 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2599b0e0 0x555c25a10800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31aeffd640 1 --2- 172.21.15.31:0/1244629843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 0x555c25995670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31af7fe640 1 -- 172.21.15.31:0/1244629843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 msgr2=0x555c25995670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31af7fe640 1 --2- 172.21.15.31:0/1244629843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 0x555c25995670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31af7fe640 1 -- 172.21.15.31:0/1244629843 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2599b0e0 msgr2=0x555c25a10800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31af7fe640 1 --2- 172.21.15.31:0/1244629843 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2599b0e0 0x555c25a10800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31af7fe640 1 -- 172.21.15.31:0/1244629843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c25a10d40 con 0x555c2598ff00 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31affff640 1 --2- 172.21.15.31:0/1244629843 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2599b0e0 0x555c25a10800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f31af7fe640 1 --2- 172.21.15.31:0/1244629843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2598ff00 0x555c25995130 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f31a0020040 tx=0x7f31a009aa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479d0db4916aa3bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.534+0000 7f318effd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.535+0000 7f318effd640 1 -- 172.21.15.31:0/1244629843 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a00b5070 con 0x555c2598ff00 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.535+0000 7f318effd640 1 -- 172.21.15.31:0/1244629843 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8528+0+0 (secure 0 0 0) 0x7f31a0098fe0 con 0x555c2598ff00 2024-04-26T18:03:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.535+0000 7f31b8df4740 1 -- 172.21.15.31:0/1244629843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31a009d070 con 0x555c2598ff00 2024-04-26T18:03:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.535+0000 7f318effd640 1 -- 172.21.15.31:0/1244629843 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f31a009c970 con 0x555c2598ff00 2024-04-26T18:03:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.535+0000 7f31b8df4740 10 client.16023 fetch_fsmap learned FSMap version 33 2024-04-26T18:03:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.535+0000 7f31b8df4740 1 -- 172.21.15.31:0/1244629843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555c25a11e80 con 0x555c2598ff00 2024-04-26T18:03:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.535+0000 7f318effd640 1 -- 172.21.15.31:0/1244629843 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31a009d070 con 0x555c2598ff00 2024-04-26T18:03:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.535+0000 7f318effd640 1 -- 172.21.15.31:0/1244629843 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f31a00672a0 con 0x555c2598ff00 2024-04-26T18:03:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.535+0000 7f31b8df4740 10 client.16023 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T18:03:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.535+0000 7f31b8df4740 10 client.16023 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:03:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.535+0000 7f31b8df4740 20 client.16023 populate_metadata read hostname 'smithi031' 2024-04-26T18:03:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.535+0000 7f31b8df4740 1 --2- 172.21.15.31:0/1244629843 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555c25a12630 0x555c25a14a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.535+0000 7f31b8df4740 4 client.16023 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:03:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.535+0000 7f31b8df4740 1 -- 172.21.15.31:0/1244629843 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555c258fdee0 con 0x555c25a12630 2024-04-26T18:03:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.536+0000 7f31aeffd640 1 --2- 172.21.15.31:0/1244629843 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555c25a12630 0x555c25a14a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.536+0000 7f31aeffd640 1 --2- 172.21.15.31:0/1244629843 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555c25a12630 0x555c25a14a20 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cfdc26a9bd90e73 server_cookie=8f0920cd046ec085 in_seq=0 out_seq=0 2024-04-26T18:03:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.536+0000 7f318effd640 10 client.16023 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:08.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.555+0000 7f318effd640 1 -- 172.21.15.31:0/1244629843 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555c258fdee0 con 0x555c25a12630 2024-04-26T18:03:08.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.556+0000 7f318effd640 10 client.16023 handle_command_reply: tid=0 2024-04-26T18:03:08.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.556+0000 7f31b8df4740 1 client.16023 shutdown 2024-04-26T18:03:08.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.556+0000 7f31b8df4740 1 -- 172.21.15.31:0/1244629843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2598ff00 msgr2=0x555c25995130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.556+0000 7f31b8df4740 1 --2- 172.21.15.31:0/1244629843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2598ff00 0x555c25995130 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f31a0020040 tx=0x7f31a009aa80 comp rx=0 tx=0).stop 2024-04-26T18:03:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.556+0000 7f31b8df4740 1 -- 172.21.15.31:0/1244629843 shutdown_connections 2024-04-26T18:03:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.556+0000 7f31b8df4740 1 --2- 172.21.15.31:0/1244629843 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x555c25a12630 0x555c25a14a20 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.556+0000 7f31b8df4740 1 --2- 172.21.15.31:0/1244629843 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x555c2599b0e0 0x555c25a10800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.556+0000 7f318effd640 0 client.16023 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.556+0000 7f31b8df4740 1 --2- 172.21.15.31:0/1244629843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c25990820 0x555c25995670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.556+0000 7f31b8df4740 1 --2- 172.21.15.31:0/1244629843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555c2598ff00 0x555c25995130 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.556+0000 7f31b8df4740 1 -- 172.21.15.31:0/1244629843 >> 172.21.15.31:0/1244629843 conn(0x555c2598c410 msgr2=0x555c2598ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.556+0000 7f31b8df4740 1 -- 172.21.15.31:0/1244629843 shutdown_connections 2024-04-26T18:03:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.556+0000 7f31b8df4740 1 -- 172.21.15.31:0/1244629843 wait complete. 2024-04-26T18:03:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.556+0000 7f31b8df4740 20 client.16023 trim_cache size 0 max 16384 2024-04-26T18:03:08.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:03:08.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:03:08.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:03:08.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:03:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.557+0000 7f31b60dc640 1 -- 172.21.15.31:0/543858990 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f31980a3dc0 msgr2=0x7f31980a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.557+0000 7f31b60dc640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f31980a3dc0 0x7f31980a6280 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f31a8005fd0 tx=0x7f31a8009500 comp rx=0 tx=0).stop 2024-04-26T18:03:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.557+0000 7f31b60dc640 1 -- 172.21.15.31:0/543858990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31b0103bc0 msgr2=0x7f31b0199030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.557+0000 7f31b60dc640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31b0103bc0 0x7f31b0199030 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f31a000c920 tx=0x7f31a000cde0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.557+0000 7f31b60dc640 1 -- 172.21.15.31:0/543858990 shutdown_connections 2024-04-26T18:03:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.557+0000 7f31b60dc640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31b010cf00 0x7f31b019d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.557+0000 7f31b60dc640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31b01044e0 0x7f31b0199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.557+0000 7f31b60dc640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f31980a3dc0 0x7f31980a6280 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.557+0000 7f31b60dc640 1 --2- 172.21.15.31:0/543858990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31b0103bc0 0x7f31b0199030 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.557+0000 7f31b60dc640 1 -- 172.21.15.31:0/543858990 >> 172.21.15.31:0/543858990 conn(0x7f31b00789b0 msgr2=0x7f31b00fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:08.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.557+0000 7f31b60dc640 1 -- 172.21.15.31:0/543858990 shutdown_connections 2024-04-26T18:03:08.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:08.557+0000 7f31b60dc640 1 -- 172.21.15.31:0/543858990 wait complete. 2024-04-26T18:03:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:09 smithi186 ceph-mon[32797]: pgmap v203: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 4.8 MiB/s wr, 573 op/s 2024-04-26T18:03:09.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:09 smithi031 ceph-mon[29164]: pgmap v203: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 4.8 MiB/s wr, 573 op/s 2024-04-26T18:03:09.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:09 smithi097 ceph-mon[32419]: pgmap v203: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 4.8 MiB/s wr, 573 op/s 2024-04-26T18:03:11.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:11 smithi031 ceph-mon[29164]: pgmap v204: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 4.8 MiB/s wr, 573 op/s 2024-04-26T18:03:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:11 smithi097 ceph-mon[32419]: pgmap v204: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 4.8 MiB/s wr, 573 op/s 2024-04-26T18:03:11.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:11 smithi186 ceph-mon[32797]: pgmap v204: 97 pgs: 97 active+clean; 3.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 4.8 MiB/s wr, 573 op/s 2024-04-26T18:03:13.099 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 350.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:03:13.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:13 smithi031 ceph-mon[29164]: pgmap v205: 97 pgs: 97 active+clean; 3.7 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 1.1 MiB/s wr, 233 op/s 2024-04-26T18:03:13.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:13 smithi097 ceph-mon[32419]: pgmap v205: 97 pgs: 97 active+clean; 3.7 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 1.1 MiB/s wr, 233 op/s 2024-04-26T18:03:13.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:13 smithi186 ceph-mon[32797]: pgmap v205: 97 pgs: 97 active+clean; 3.7 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 1.1 MiB/s wr, 233 op/s 2024-04-26T18:03:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:14 smithi186 ceph-mon[32797]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T18:03:15.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:14 smithi186 ceph-mon[32797]: pgmap v207: 97 pgs: 97 active+clean; 3.7 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 462 KiB/s wr, 136 op/s 2024-04-26T18:03:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:14 smithi031 ceph-mon[29164]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T18:03:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:14 smithi031 ceph-mon[29164]: pgmap v207: 97 pgs: 97 active+clean; 3.7 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 462 KiB/s wr, 136 op/s 2024-04-26T18:03:15.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:14 smithi097 ceph-mon[32419]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T18:03:15.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:14 smithi097 ceph-mon[32419]: pgmap v207: 97 pgs: 97 active+clean; 3.7 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 462 KiB/s wr, 136 op/s 2024-04-26T18:03:17.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:16 smithi186 ceph-mon[32797]: pgmap v208: 97 pgs: 97 active+clean; 3.7 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 MiB/s wr, 429 op/s 2024-04-26T18:03:17.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:16 smithi031 ceph-mon[29164]: pgmap v208: 97 pgs: 97 active+clean; 3.7 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 MiB/s wr, 429 op/s 2024-04-26T18:03:17.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:16 smithi097 ceph-mon[32419]: pgmap v208: 97 pgs: 97 active+clean; 3.7 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 MiB/s wr, 429 op/s 2024-04-26T18:03:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 355.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:03:19.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:18 smithi186 ceph-mon[32797]: pgmap v209: 97 pgs: 97 active+clean; 3.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 MiB/s wr, 1.19k op/s 2024-04-26T18:03:19.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:18 smithi031 ceph-mon[29164]: pgmap v209: 97 pgs: 97 active+clean; 3.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 MiB/s wr, 1.19k op/s 2024-04-26T18:03:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:18 smithi097 ceph-mon[32419]: pgmap v209: 97 pgs: 97 active+clean; 3.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 MiB/s wr, 1.19k op/s 2024-04-26T18:03:21.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:20 smithi186 ceph-mon[32797]: pgmap v210: 97 pgs: 97 active+clean; 3.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 MiB/s wr, 1.19k op/s 2024-04-26T18:03:21.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:20 smithi031 ceph-mon[29164]: pgmap v210: 97 pgs: 97 active+clean; 3.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 MiB/s wr, 1.19k op/s 2024-04-26T18:03:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:20 smithi097 ceph-mon[32419]: pgmap v210: 97 pgs: 97 active+clean; 3.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 MiB/s wr, 1.19k op/s 2024-04-26T18:03:21.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.657+0000 7fc28e719640 1 -- 172.21.15.31:0/1328988205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288108a30 msgr2=0x7fc288108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:21.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.657+0000 7fc28e719640 1 --2- 172.21.15.31:0/1328988205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288108a30 0x7fc288108e10 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc278009930 tx=0x7fc27802a420 comp rx=0 tx=0).stop 2024-04-26T18:03:21.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.657+0000 7fc28e719640 1 -- 172.21.15.31:0/1328988205 shutdown_connections 2024-04-26T18:03:21.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.657+0000 7fc28e719640 1 --2- 172.21.15.31:0/1328988205 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc28806bb00 0x7fc28810f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.657+0000 7fc28e719640 1 --2- 172.21.15.31:0/1328988205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc28806b160 0x7fc28806b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.657+0000 7fc28e719640 1 --2- 172.21.15.31:0/1328988205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288108a30 0x7fc288108e10 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.657+0000 7fc28e719640 1 -- 172.21.15.31:0/1328988205 >> 172.21.15.31:0/1328988205 conn(0x7fc288077630 msgr2=0x7fc288077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:21.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.657+0000 7fc28e719640 1 -- 172.21.15.31:0/1328988205 shutdown_connections 2024-04-26T18:03:21.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.657+0000 7fc28e719640 1 -- 172.21.15.31:0/1328988205 wait complete. 2024-04-26T18:03:21.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.658+0000 7fc28e719640 1 Processor -- start 2024-04-26T18:03:21.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.658+0000 7fc28e719640 1 -- start start 2024-04-26T18:03:21.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.658+0000 7fc28e719640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc28806b160 0x7fc2881990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.658+0000 7fc28e719640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc28806bb00 0x7fc2881995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.658+0000 7fc28e719640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288108a30 0x7fc28819d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.658+0000 7fc28e719640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc288112bb0 con 0x7fc288108a30 2024-04-26T18:03:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.658+0000 7fc28e719640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc288112a30 con 0x7fc28806bb00 2024-04-26T18:03:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.658+0000 7fc28e719640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc288112d30 con 0x7fc28806b160 2024-04-26T18:03:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.658+0000 7fc28ccb6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288108a30 0x7fc28819d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.658+0000 7fc28ccb6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288108a30 0x7fc28819d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43958/0 (socket says 172.21.15.31:43958) 2024-04-26T18:03:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.658+0000 7fc28ccb6640 1 -- 172.21.15.31:0/3313968018 learned_addr learned my addr 172.21.15.31:0/3313968018 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.658+0000 7fc287fff640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc28806b160 0x7fc2881990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.659+0000 7fc2877fe640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc28806bb00 0x7fc2881995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.659+0000 7fc28ccb6640 1 -- 172.21.15.31:0/3313968018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc28806b160 msgr2=0x7fc2881990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.659+0000 7fc28ccb6640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc28806b160 0x7fc2881990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.659+0000 7fc28ccb6640 1 -- 172.21.15.31:0/3313968018 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc28806bb00 msgr2=0x7fc2881995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.659+0000 7fc28ccb6640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc28806bb00 0x7fc2881995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.659+0000 7fc28ccb6640 1 -- 172.21.15.31:0/3313968018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc28819e0a0 con 0x7fc288108a30 2024-04-26T18:03:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.659+0000 7fc2877fe640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc28806bb00 0x7fc2881995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:03:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.659+0000 7fc28ccb6640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288108a30 0x7fc28819d970 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc28000eaa0 tx=0x7fc28000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.659+0000 7fc2857fa640 1 -- 172.21.15.31:0/3313968018 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28000cdb0 con 0x7fc288108a30 2024-04-26T18:03:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.659+0000 7fc28e719640 1 -- 172.21.15.31:0/3313968018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc28819e390 con 0x7fc288108a30 2024-04-26T18:03:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.659+0000 7fc2857fa640 1 -- 172.21.15.31:0/3313968018 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc280005020 con 0x7fc288108a30 2024-04-26T18:03:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.659+0000 7fc2857fa640 1 -- 172.21.15.31:0/3313968018 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2800116d0 con 0x7fc288108a30 2024-04-26T18:03:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.659+0000 7fc28e719640 1 -- 172.21.15.31:0/3313968018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2881a5c20 con 0x7fc288108a30 2024-04-26T18:03:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.661+0000 7fc2857fa640 1 -- 172.21.15.31:0/3313968018 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc2800051d0 con 0x7fc288108a30 2024-04-26T18:03:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.662+0000 7fc291438740 1 Processor -- start 2024-04-26T18:03:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.662+0000 7fc291438740 1 -- start start 2024-04-26T18:03:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.662+0000 7fc291438740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598aae00 0x55e3598ab1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.662+0000 7fc291438740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598ab720 0x55e3598b9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.662+0000 7fc291438740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ba4f0 0x55e3598bc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.662+0000 7fc291438740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e359811460 con 0x55e3598ab720 2024-04-26T18:03:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.662+0000 7fc291438740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55e3597cb590 con 0x55e3598ba4f0 2024-04-26T18:03:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.662+0000 7fc291438740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e35980d420 con 0x55e3598aae00 2024-04-26T18:03:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.662+0000 7fc2857fa640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc2700a3dc0 0x7fc2700a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.662+0000 7fc287fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598ab720 0x55e3598b9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.662+0000 7fc287fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598ab720 0x55e3598b9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43970/0 (socket says 172.21.15.31:43970) 2024-04-26T18:03:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.662+0000 7fc287fff640 1 -- 172.21.15.31:0/171681945 learned_addr learned my addr 172.21.15.31:0/171681945 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.662+0000 7fc28ccb6640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc2700a3dc0 0x7fc2700a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.662+0000 7fc2877fe640 1 --2- 172.21.15.31:0/171681945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598aae00 0x55e3598ab1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.663+0000 7fc2877fe640 1 --2- 172.21.15.31:0/171681945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ba4f0 0x55e3598bc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.663+0000 7fc2857fa640 1 -- 172.21.15.31:0/3313968018 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7fc2800d09d0 con 0x7fc288108a30 2024-04-26T18:03:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.663+0000 7fc287fff640 1 -- 172.21.15.31:0/171681945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598aae00 msgr2=0x55e3598ab1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.663+0000 7fc287fff640 1 --2- 172.21.15.31:0/171681945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598aae00 0x55e3598ab1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.663+0000 7fc287fff640 1 -- 172.21.15.31:0/171681945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ba4f0 msgr2=0x55e3598bc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.663+0000 7fc287fff640 1 --2- 172.21.15.31:0/171681945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ba4f0 0x55e3598bc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.663+0000 7fc287fff640 1 -- 172.21.15.31:0/171681945 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3597d80d0 con 0x55e3598ab720 2024-04-26T18:03:21.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.663+0000 7fc287fff640 1 --2- 172.21.15.31:0/171681945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598ab720 0x55e3598b9fb0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc27802a940 tx=0x7fc278005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8944315193ce41 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.663+0000 7fc28ccb6640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc2700a3dc0 0x7fc2700a6280 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc28009c8c0 tx=0x7fc280005890 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.663+0000 7fc26e7fc640 1 -- 172.21.15.31:0/171681945 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc278040070 con 0x55e3598ab720 2024-04-26T18:03:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.663+0000 7fc26e7fc640 1 -- 172.21.15.31:0/171681945 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2780316a0 con 0x55e3598ab720 2024-04-26T18:03:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.663+0000 7fc26e7fc640 1 -- 172.21.15.31:0/171681945 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc278035040 con 0x55e3598ab720 2024-04-26T18:03:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.664+0000 7fc291438740 1 -- 172.21.15.31:0/171681945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598ab720 msgr2=0x55e3598b9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.664+0000 7fc291438740 1 --2- 172.21.15.31:0/171681945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598ab720 0x55e3598b9fb0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc27802a940 tx=0x7fc278005c00 comp rx=0 tx=0).stop 2024-04-26T18:03:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.664+0000 7fc291438740 1 -- 172.21.15.31:0/171681945 shutdown_connections 2024-04-26T18:03:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.664+0000 7fc291438740 1 --2- 172.21.15.31:0/171681945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ba4f0 0x55e3598bc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.664+0000 7fc291438740 1 --2- 172.21.15.31:0/171681945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598ab720 0x55e3598b9fb0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.664+0000 7fc291438740 1 --2- 172.21.15.31:0/171681945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598aae00 0x55e3598ab1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.664+0000 7fc291438740 1 -- 172.21.15.31:0/171681945 >> 172.21.15.31:0/171681945 conn(0x55e3598a7410 msgr2=0x55e3598a77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.664+0000 7fc291438740 1 -- 172.21.15.31:0/171681945 shutdown_connections 2024-04-26T18:03:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.664+0000 7fc291438740 1 -- 172.21.15.31:0/171681945 wait complete. 2024-04-26T18:03:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.664+0000 7fc291438740 1 -- start start 2024-04-26T18:03:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc291438740 1 Processor -- start 2024-04-26T18:03:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc291438740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598aae00 0x55e3598b7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc291438740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ab720 0x55e3598b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc291438740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598ba4f0 0x55e3598b7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc291438740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e3598bd0c0 con 0x55e3598aae00 2024-04-26T18:03:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc291438740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55e359798b60 con 0x55e3598ab720 2024-04-26T18:03:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc291438740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e3598bd240 con 0x55e3598ba4f0 2024-04-26T18:03:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc291438740 10 client.0 _ll_register_callbacks cb 0x55e3597bd2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:03:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc2877fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598aae00 0x55e3598b7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc2877fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598aae00 0x55e3598b7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43986/0 (socket says 172.21.15.31:43986) 2024-04-26T18:03:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc2877fe640 1 -- 172.21.15.31:0/1216221082 learned_addr learned my addr 172.21.15.31:0/1216221082 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc287fff640 1 --2- 172.21.15.31:0/1216221082 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ab720 0x55e3598b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc2877fe640 1 -- 172.21.15.31:0/1216221082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598ba4f0 msgr2=0x55e3598b7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc2877fe640 1 --2- 172.21.15.31:0/1216221082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598ba4f0 0x55e3598b7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc2877fe640 1 -- 172.21.15.31:0/1216221082 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ab720 msgr2=0x55e3598b78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc2877fe640 1 --2- 172.21.15.31:0/1216221082 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ab720 0x55e3598b78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.665+0000 7fc2877fe640 1 -- 172.21.15.31:0/1216221082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e359523210 con 0x55e3598aae00 2024-04-26T18:03:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc2877fe640 1 --2- 172.21.15.31:0/1216221082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598aae00 0x55e3598b7390 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc27c00ce30 tx=0x7fc27c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c40d4583767be40 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc2677fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:03:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc2677fe640 1 -- 172.21.15.31:0/1216221082 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c011cb0 con 0x55e3598aae00 2024-04-26T18:03:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc2677fe640 1 -- 172.21.15.31:0/1216221082 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7fc27c005020 con 0x55e3598aae00 2024-04-26T18:03:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc291438740 1 -- 172.21.15.31:0/1216221082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2780316a0 con 0x55e3598aae00 2024-04-26T18:03:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc2677fe640 1 -- 172.21.15.31:0/1216221082 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc27c011e50 con 0x55e3598aae00 2024-04-26T18:03:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc291438740 10 client.16035 fetch_fsmap learned FSMap version 33 2024-04-26T18:03:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc291438740 1 -- 172.21.15.31:0/1216221082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e3595ef620 con 0x55e3598aae00 2024-04-26T18:03:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc2677fe640 1 -- 172.21.15.31:0/1216221082 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2780316a0 con 0x55e3598aae00 2024-04-26T18:03:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc2677fe640 1 -- 172.21.15.31:0/1216221082 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7fc27c01d410 con 0x55e3598aae00 2024-04-26T18:03:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc291438740 10 client.16035 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T18:03:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc291438740 10 client.16035 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:03:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc291438740 20 client.16035 populate_metadata read hostname 'smithi031' 2024-04-26T18:03:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc291438740 1 --2- 172.21.15.31:0/1216221082 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e3598ca610 0x55e3598cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc291438740 4 client.16035 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:03:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.666+0000 7fc291438740 1 -- 172.21.15.31:0/1216221082 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e35981f070 con 0x55e3598ca610 2024-04-26T18:03:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.667+0000 7fc287fff640 1 --2- 172.21.15.31:0/1216221082 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e3598ca610 0x55e3598cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.667+0000 7fc287fff640 1 --2- 172.21.15.31:0/1216221082 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e3598ca610 0x55e3598cca20 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c23810dcd4d90cf2 server_cookie=31e87f8be67a38f6 in_seq=0 out_seq=0 2024-04-26T18:03:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.667+0000 7fc2677fe640 10 client.16035 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc2677fe640 1 -- 172.21.15.31:0/1216221082 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e35981f070 con 0x55e3598ca610 2024-04-26T18:03:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc2677fe640 10 client.16035 handle_command_reply: tid=0 2024-04-26T18:03:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc291438740 1 client.16035 shutdown 2024-04-26T18:03:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc291438740 1 -- 172.21.15.31:0/1216221082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598aae00 msgr2=0x55e3598b7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc291438740 1 --2- 172.21.15.31:0/1216221082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598aae00 0x55e3598b7390 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc27c00ce30 tx=0x7fc27c00cf20 comp rx=0 tx=0).stop 2024-04-26T18:03:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc291438740 1 -- 172.21.15.31:0/1216221082 shutdown_connections 2024-04-26T18:03:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc291438740 1 --2- 172.21.15.31:0/1216221082 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e3598ca610 0x55e3598cca20 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc291438740 1 --2- 172.21.15.31:0/1216221082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598ba4f0 0x55e3598b7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc291438740 1 --2- 172.21.15.31:0/1216221082 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ab720 0x55e3598b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc2677fe640 0 client.16035 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc291438740 1 --2- 172.21.15.31:0/1216221082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598aae00 0x55e3598b7390 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc291438740 1 -- 172.21.15.31:0/1216221082 >> 172.21.15.31:0/1216221082 conn(0x55e3598a7410 msgr2=0x55e3598ad280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc291438740 1 -- 172.21.15.31:0/1216221082 shutdown_connections 2024-04-26T18:03:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc291438740 1 -- 172.21.15.31:0/1216221082 wait complete. 2024-04-26T18:03:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.668+0000 7fc291438740 20 client.16035 trim_cache size 0 max 16384 2024-04-26T18:03:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.683+0000 7fc291438740 1 Processor -- start 2024-04-26T18:03:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.683+0000 7fc291438740 1 -- start start 2024-04-26T18:03:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc291438740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598aae00 0x55e3598c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc291438740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598ab720 0x55e3598ad0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc291438740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ba4f0 0x55e3598d2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc291438740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc2881aeb90 con 0x55e3598ab720 2024-04-26T18:03:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc291438740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc2881ac400 con 0x55e3598ba4f0 2024-04-26T18:03:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc291438740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2881aa250 con 0x55e3598aae00 2024-04-26T18:03:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc287fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598ab720 0x55e3598ad0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc287fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598ab720 0x55e3598ad0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44000/0 (socket says 172.21.15.31:44000) 2024-04-26T18:03:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc287fff640 1 -- 172.21.15.31:0/2177100214 learned_addr learned my addr 172.21.15.31:0/2177100214 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc2877fe640 1 --2- 172.21.15.31:0/2177100214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598aae00 0x55e3598c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc287fff640 1 -- 172.21.15.31:0/2177100214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598aae00 msgr2=0x55e3598c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc287fff640 1 --2- 172.21.15.31:0/2177100214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598aae00 0x55e3598c5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc287fff640 1 -- 172.21.15.31:0/2177100214 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ba4f0 msgr2=0x55e3598d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc287fff640 1 --2- 172.21.15.31:0/2177100214 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ba4f0 0x55e3598d2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc287fff640 1 -- 172.21.15.31:0/2177100214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3598ab550 con 0x55e3598ab720 2024-04-26T18:03:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc2877fe640 1 --2- 172.21.15.31:0/2177100214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598aae00 0x55e3598c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:03:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc287fff640 1 --2- 172.21.15.31:0/2177100214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598ab720 0x55e3598ad0b0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc278009930 tx=0x7fc27802a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac96008aa5c75dc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc26dffb640 1 -- 172.21.15.31:0/2177100214 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc278055070 con 0x55e3598ab720 2024-04-26T18:03:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc26dffb640 1 -- 172.21.15.31:0/2177100214 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc278004040 con 0x55e3598ab720 2024-04-26T18:03:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.684+0000 7fc26dffb640 1 -- 172.21.15.31:0/2177100214 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc278035040 con 0x55e3598ab720 2024-04-26T18:03:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.685+0000 7fc291438740 1 -- 172.21.15.31:0/2177100214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598ab720 msgr2=0x55e3598ad0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.685+0000 7fc291438740 1 --2- 172.21.15.31:0/2177100214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598ab720 0x55e3598ad0b0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc278009930 tx=0x7fc27802a420 comp rx=0 tx=0).stop 2024-04-26T18:03:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.685+0000 7fc291438740 1 -- 172.21.15.31:0/2177100214 shutdown_connections 2024-04-26T18:03:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.685+0000 7fc291438740 1 --2- 172.21.15.31:0/2177100214 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ba4f0 0x55e3598d2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.685+0000 7fc291438740 1 --2- 172.21.15.31:0/2177100214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598ab720 0x55e3598ad0b0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.685+0000 7fc291438740 1 --2- 172.21.15.31:0/2177100214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598aae00 0x55e3598c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.685+0000 7fc291438740 1 -- 172.21.15.31:0/2177100214 >> 172.21.15.31:0/2177100214 conn(0x55e3598a7410 msgr2=0x55e3598ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.685+0000 7fc291438740 1 -- 172.21.15.31:0/2177100214 shutdown_connections 2024-04-26T18:03:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.685+0000 7fc291438740 1 -- 172.21.15.31:0/2177100214 wait complete. 2024-04-26T18:03:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.685+0000 7fc291438740 1 -- start start 2024-04-26T18:03:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc291438740 1 Processor -- start 2024-04-26T18:03:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc291438740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598aae00 0x55e3598c2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc291438740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598ab720 0x55e3598b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc291438740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ba4f0 0x55e3598b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc291438740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc2881aa8c0 con 0x55e3598aae00 2024-04-26T18:03:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc291438740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc2881abd30 con 0x55e3598ba4f0 2024-04-26T18:03:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc291438740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2881aaf90 con 0x55e3598ab720 2024-04-26T18:03:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc291438740 10 client.0 _ll_register_callbacks cb 0x7fc27c00c3d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:03:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc2877fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598aae00 0x55e3598c2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc2877fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598aae00 0x55e3598c2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44008/0 (socket says 172.21.15.31:44008) 2024-04-26T18:03:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc2877fe640 1 -- 172.21.15.31:0/2412657276 learned_addr learned my addr 172.21.15.31:0/2412657276 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc287fff640 1 --2- 172.21.15.31:0/2412657276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598ab720 0x55e3598b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc2877fe640 1 -- 172.21.15.31:0/2412657276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598ab720 msgr2=0x55e3598b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc2877fe640 1 --2- 172.21.15.31:0/2412657276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598ab720 0x55e3598b3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc2877fe640 1 -- 172.21.15.31:0/2412657276 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ba4f0 msgr2=0x55e3598b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc2877fe640 1 --2- 172.21.15.31:0/2412657276 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ba4f0 0x55e3598b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.686+0000 7fc2877fe640 1 -- 172.21.15.31:0/2412657276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3598b4bc0 con 0x55e3598aae00 2024-04-26T18:03:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc2877fe640 1 --2- 172.21.15.31:0/2412657276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598aae00 0x55e3598c2970 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0118f0 tx=0x7fc27c00c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a48ce21fa87aa80 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc2677fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:03:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc2677fe640 1 -- 172.21.15.31:0/2412657276 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c02a040 con 0x55e3598aae00 2024-04-26T18:03:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc2677fe640 1 -- 172.21.15.31:0/2412657276 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7fc27c01da00 con 0x55e3598aae00 2024-04-26T18:03:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc291438740 1 -- 172.21.15.31:0/2412657276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc278004040 con 0x55e3598aae00 2024-04-26T18:03:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc2677fe640 1 -- 172.21.15.31:0/2412657276 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc27c00ca40 con 0x55e3598aae00 2024-04-26T18:03:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc291438740 10 client.16041 fetch_fsmap learned FSMap version 33 2024-04-26T18:03:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc291438740 1 -- 172.21.15.31:0/2412657276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e359930bc0 con 0x55e3598aae00 2024-04-26T18:03:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc2677fe640 1 -- 172.21.15.31:0/2412657276 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc278004040 con 0x55e3598aae00 2024-04-26T18:03:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc2677fe640 1 -- 172.21.15.31:0/2412657276 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7fc27c036020 con 0x55e3598aae00 2024-04-26T18:03:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc291438740 10 client.16041 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T18:03:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc291438740 10 client.16041 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:03:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc291438740 20 client.16041 populate_metadata read hostname 'smithi031' 2024-04-26T18:03:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc291438740 1 --2- 172.21.15.31:0/2412657276 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e359931290 0x55e3599336b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc291438740 4 client.16041 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:03:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.687+0000 7fc291438740 1 -- 172.21.15.31:0/2412657276 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e359818ee0 con 0x55e359931290 2024-04-26T18:03:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.688+0000 7fc287fff640 1 --2- 172.21.15.31:0/2412657276 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e359931290 0x55e3599336b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.688+0000 7fc287fff640 1 --2- 172.21.15.31:0/2412657276 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e359931290 0x55e3599336b0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17c8bd0a98c35e95 server_cookie=b747d7f4be499503 in_seq=0 out_seq=0 2024-04-26T18:03:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.688+0000 7fc2677fe640 10 client.16041 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc2677fe640 1 -- 172.21.15.31:0/2412657276 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e359818ee0 con 0x55e359931290 2024-04-26T18:03:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc2677fe640 10 client.16041 handle_command_reply: tid=0 2024-04-26T18:03:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc291438740 1 client.16041 shutdown 2024-04-26T18:03:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc291438740 1 -- 172.21.15.31:0/2412657276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598aae00 msgr2=0x55e3598c2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc291438740 1 --2- 172.21.15.31:0/2412657276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598aae00 0x55e3598c2970 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0118f0 tx=0x7fc27c00c670 comp rx=0 tx=0).stop 2024-04-26T18:03:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc291438740 1 -- 172.21.15.31:0/2412657276 shutdown_connections 2024-04-26T18:03:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc291438740 1 --2- 172.21.15.31:0/2412657276 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55e359931290 0x55e3599336b0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc291438740 1 --2- 172.21.15.31:0/2412657276 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55e3598ba4f0 0x55e3598b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc2677fe640 0 client.16041 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc291438740 1 --2- 172.21.15.31:0/2412657276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3598ab720 0x55e3598b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc291438740 1 --2- 172.21.15.31:0/2412657276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3598aae00 0x55e3598c2970 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc291438740 1 -- 172.21.15.31:0/2412657276 >> 172.21.15.31:0/2412657276 conn(0x55e3598a7410 msgr2=0x55e3598b9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc291438740 1 -- 172.21.15.31:0/2412657276 shutdown_connections 2024-04-26T18:03:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc291438740 1 -- 172.21.15.31:0/2412657276 wait complete. 2024-04-26T18:03:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.701+0000 7fc291438740 20 client.16041 trim_cache size 0 max 16384 2024-04-26T18:03:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:03:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:03:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:03:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:03:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.702+0000 7fc28e719640 1 -- 172.21.15.31:0/3313968018 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc2700a3dc0 msgr2=0x7fc2700a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.702+0000 7fc28e719640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc2700a3dc0 0x7fc2700a6280 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc28009c8c0 tx=0x7fc280005890 comp rx=0 tx=0).stop 2024-04-26T18:03:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.702+0000 7fc28e719640 1 -- 172.21.15.31:0/3313968018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288108a30 msgr2=0x7fc28819d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.702+0000 7fc28e719640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288108a30 0x7fc28819d970 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc28000eaa0 tx=0x7fc28000ef60 comp rx=0 tx=0).stop 2024-04-26T18:03:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.702+0000 7fc28e719640 1 -- 172.21.15.31:0/3313968018 shutdown_connections 2024-04-26T18:03:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.702+0000 7fc28e719640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc2700a3dc0 0x7fc2700a6280 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.702+0000 7fc28e719640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288108a30 0x7fc28819d970 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.702+0000 7fc28e719640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc28806bb00 0x7fc2881995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.702+0000 7fc28e719640 1 --2- 172.21.15.31:0/3313968018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc28806b160 0x7fc2881990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.702+0000 7fc28e719640 1 -- 172.21.15.31:0/3313968018 >> 172.21.15.31:0/3313968018 conn(0x7fc288077630 msgr2=0x7fc288100910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.702+0000 7fc28e719640 1 -- 172.21.15.31:0/3313968018 shutdown_connections 2024-04-26T18:03:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:21.702+0000 7fc28e719640 1 -- 172.21.15.31:0/3313968018 wait complete. 2024-04-26T18:03:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 360.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:03:23.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:22 smithi186 ceph-mon[32797]: pgmap v211: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 MiB/s wr, 1.28k op/s 2024-04-26T18:03:23.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:22 smithi031 ceph-mon[29164]: pgmap v211: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 MiB/s wr, 1.28k op/s 2024-04-26T18:03:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:22 smithi097 ceph-mon[32419]: pgmap v211: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 MiB/s wr, 1.28k op/s 2024-04-26T18:03:25.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:24 smithi186 ceph-mon[32797]: pgmap v212: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 MiB/s wr, 1.25k op/s 2024-04-26T18:03:25.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:24 smithi031 ceph-mon[29164]: pgmap v212: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 MiB/s wr, 1.25k op/s 2024-04-26T18:03:25.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:24 smithi097 ceph-mon[32419]: pgmap v212: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 MiB/s wr, 1.25k op/s 2024-04-26T18:03:27.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:26 smithi186 ceph-mon[32797]: pgmap v213: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 28 MiB/s wr, 1.07k op/s 2024-04-26T18:03:27.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:26 smithi031 ceph-mon[29164]: pgmap v213: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 28 MiB/s wr, 1.07k op/s 2024-04-26T18:03:27.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:26 smithi097 ceph-mon[32419]: pgmap v213: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 28 MiB/s wr, 1.07k op/s 2024-04-26T18:03:28.097 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 365.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:03:29.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:28 smithi186 ceph-mon[32797]: pgmap v214: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 MiB/s wr, 735 op/s 2024-04-26T18:03:29.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:28 smithi031 ceph-mon[29164]: pgmap v214: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 MiB/s wr, 735 op/s 2024-04-26T18:03:29.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:28 smithi097 ceph-mon[32419]: pgmap v214: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 MiB/s wr, 735 op/s 2024-04-26T18:03:31.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:30 smithi186 ceph-mon[32797]: pgmap v215: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 734 KiB/s wr, 89 op/s 2024-04-26T18:03:31.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:30 smithi031 ceph-mon[29164]: pgmap v215: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 734 KiB/s wr, 89 op/s 2024-04-26T18:03:31.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:30 smithi097 ceph-mon[32419]: pgmap v215: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 734 KiB/s wr, 89 op/s 2024-04-26T18:03:33.099 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 370.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:03:33.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:33 smithi186 ceph-mon[32797]: pgmap v216: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 737 KiB/s wr, 92 op/s 2024-04-26T18:03:33.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:32 smithi031 ceph-mon[29164]: pgmap v216: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 737 KiB/s wr, 92 op/s 2024-04-26T18:03:33.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:32 smithi097 ceph-mon[32419]: pgmap v216: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 737 KiB/s wr, 92 op/s 2024-04-26T18:03:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.801+0000 7f2e232a9640 1 -- 172.21.15.31:0/3995471221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c102210 msgr2=0x7f2e1c10e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.801+0000 7f2e232a9640 1 --2- 172.21.15.31:0/3995471221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c102210 0x7f2e1c10e720 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2e14009cd0 tx=0x7f2e1402a8e0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.801+0000 7f2e232a9640 1 -- 172.21.15.31:0/3995471221 shutdown_connections 2024-04-26T18:03:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.801+0000 7f2e232a9640 1 --2- 172.21.15.31:0/3995471221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c102210 0x7f2e1c10e720 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.802+0000 7f2e232a9640 1 --2- 172.21.15.31:0/3995471221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e1c101850 0x7f2e1c101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.802+0000 7f2e232a9640 1 --2- 172.21.15.31:0/3995471221 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e1c107860 0x7f2e1c107c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.802+0000 7f2e232a9640 1 -- 172.21.15.31:0/3995471221 >> 172.21.15.31:0/3995471221 conn(0x7f2e1c077bb0 msgr2=0x7f2e1c077fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.802+0000 7f2e232a9640 1 -- 172.21.15.31:0/3995471221 shutdown_connections 2024-04-26T18:03:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.802+0000 7f2e232a9640 1 -- 172.21.15.31:0/3995471221 wait complete. 2024-04-26T18:03:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.802+0000 7f2e232a9640 1 Processor -- start 2024-04-26T18:03:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.802+0000 7f2e232a9640 1 -- start start 2024-04-26T18:03:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.802+0000 7f2e232a9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e1c101850 0x7f2e1c199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.802+0000 7f2e232a9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c102210 0x7f2e1c19a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.802+0000 7f2e232a9640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e1c107860 0x7f2e1c1a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.802+0000 7f2e232a9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c111b70 con 0x7f2e1c102210 2024-04-26T18:03:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.802+0000 7f2e232a9640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c1119f0 con 0x7f2e1c107860 2024-04-26T18:03:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.802+0000 7f2e232a9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c111cf0 con 0x7f2e1c101850 2024-04-26T18:03:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e20844640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c102210 0x7f2e1c19a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e20844640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c102210 0x7f2e1c19a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37688/0 (socket says 172.21.15.31:37688) 2024-04-26T18:03:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e20844640 1 -- 172.21.15.31:0/2046614763 learned_addr learned my addr 172.21.15.31:0/2046614763 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e21045640 1 --2- 172.21.15.31:0/2046614763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e1c101850 0x7f2e1c199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e20844640 1 -- 172.21.15.31:0/2046614763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e1c101850 msgr2=0x7f2e1c199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e21846640 1 --2- 172.21.15.31:0/2046614763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e1c107860 0x7f2e1c1a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e20844640 1 --2- 172.21.15.31:0/2046614763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e1c101850 0x7f2e1c199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e20844640 1 -- 172.21.15.31:0/2046614763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e1c107860 msgr2=0x7f2e1c1a4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e20844640 1 --2- 172.21.15.31:0/2046614763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e1c107860 0x7f2e1c1a4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e20844640 1 -- 172.21.15.31:0/2046614763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1c1a5470 con 0x7f2e1c102210 2024-04-26T18:03:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e20844640 1 --2- 172.21.15.31:0/2046614763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c102210 0x7f2e1c19a3f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00d950 tx=0x7f2e0c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e127fc640 1 -- 172.21.15.31:0/2046614763 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c017030 con 0x7f2e1c102210 2024-04-26T18:03:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e232a9640 1 -- 172.21.15.31:0/2046614763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e1c1a5640 con 0x7f2e1c102210 2024-04-26T18:03:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e127fc640 1 -- 172.21.15.31:0/2046614763 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e0c004020 con 0x7f2e1c102210 2024-04-26T18:03:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e127fc640 1 -- 172.21.15.31:0/2046614763 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c0047a0 con 0x7f2e1c102210 2024-04-26T18:03:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.803+0000 7f2e232a9640 1 -- 172.21.15.31:0/2046614763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e1c1a5b00 con 0x7f2e1c102210 2024-04-26T18:03:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.806+0000 7f2e127fc640 1 -- 172.21.15.31:0/2046614763 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2e0c009a30 con 0x7f2e1c102210 2024-04-26T18:03:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.806+0000 7f2e25fc4740 1 Processor -- start 2024-04-26T18:03:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.806+0000 7f2e25fc4740 1 -- start start 2024-04-26T18:03:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.806+0000 7f2e25fc4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f67f00 0x55bfe6f682e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.806+0000 7f2e25fc4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f68820 0x55bfe6f72ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.806+0000 7f2e25fc4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f730e0 0x55bfe6f754d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.806+0000 7f2e25fc4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bfe6ece460 con 0x55bfe6f730e0 2024-04-26T18:03:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.806+0000 7f2e25fc4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55bfe6e88590 con 0x55bfe6f67f00 2024-04-26T18:03:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.806+0000 7f2e25fc4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bfe6eca420 con 0x55bfe6f68820 2024-04-26T18:03:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.806+0000 7f2e20844640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f730e0 0x55bfe6f754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.807+0000 7f2e20844640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f730e0 0x55bfe6f754d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37704/0 (socket says 172.21.15.31:37704) 2024-04-26T18:03:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.807+0000 7f2e20844640 1 -- 172.21.15.31:0/405867454 learned_addr learned my addr 172.21.15.31:0/405867454 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.807+0000 7f2e21045640 1 --2- 172.21.15.31:0/405867454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f68820 0x55bfe6f72ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.807+0000 7f2e21846640 1 --2- 172.21.15.31:0/405867454 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f67f00 0x55bfe6f682e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.807+0000 7f2e127fc640 1 --2- 172.21.15.31:0/2046614763 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2e080a3dc0 0x7f2e080a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.807+0000 7f2e20844640 1 -- 172.21.15.31:0/405867454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f68820 msgr2=0x55bfe6f72ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.807+0000 7f2e20844640 1 --2- 172.21.15.31:0/405867454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f68820 0x55bfe6f72ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.807+0000 7f2e20844640 1 -- 172.21.15.31:0/405867454 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f67f00 msgr2=0x55bfe6f682e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.807+0000 7f2e20844640 1 --2- 172.21.15.31:0/405867454 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f67f00 0x55bfe6f682e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.807+0000 7f2e20844640 1 -- 172.21.15.31:0/405867454 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfe6e950d0 con 0x55bfe6f730e0 2024-04-26T18:03:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.807+0000 7f2e21846640 1 --2- 172.21.15.31:0/2046614763 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2e080a3dc0 0x7f2e080a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.807+0000 7f2e127fc640 1 -- 172.21.15.31:0/2046614763 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f2e0c0d0260 con 0x7f2e1c102210 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.807+0000 7f2e20844640 1 --2- 172.21.15.31:0/405867454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f730e0 0x55bfe6f754d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c0cd940 tx=0x7f2e0c066820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1681117c0d41da94 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.807+0000 7f2e077fe640 1 -- 172.21.15.31:0/405867454 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c09bff0 con 0x55bfe6f730e0 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.808+0000 7f2e21846640 1 --2- 172.21.15.31:0/2046614763 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2e080a3dc0 0x7f2e080a6280 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2e14004af0 tx=0x7f2e140095c0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.808+0000 7f2e077fe640 1 -- 172.21.15.31:0/405867454 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e0c066d30 con 0x55bfe6f730e0 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.808+0000 7f2e077fe640 1 -- 172.21.15.31:0/405867454 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c09bb80 con 0x55bfe6f730e0 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.808+0000 7f2e25fc4740 1 -- 172.21.15.31:0/405867454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f730e0 msgr2=0x55bfe6f754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.808+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/405867454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f730e0 0x55bfe6f754d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c0cd940 tx=0x7f2e0c066820 comp rx=0 tx=0).stop 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.808+0000 7f2e25fc4740 1 -- 172.21.15.31:0/405867454 shutdown_connections 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.808+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/405867454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f730e0 0x55bfe6f754d0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.808+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/405867454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f68820 0x55bfe6f72ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.808+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/405867454 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f67f00 0x55bfe6f682e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.808+0000 7f2e25fc4740 1 -- 172.21.15.31:0/405867454 >> 172.21.15.31:0/405867454 conn(0x55bfe6f64410 msgr2=0x55bfe6f647f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.808+0000 7f2e25fc4740 1 -- 172.21.15.31:0/405867454 shutdown_connections 2024-04-26T18:03:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.808+0000 7f2e25fc4740 1 -- 172.21.15.31:0/405867454 wait complete. 2024-04-26T18:03:34.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.809+0000 7f2e25fc4740 1 -- start start 2024-04-26T18:03:34.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.809+0000 7f2e25fc4740 1 Processor -- start 2024-04-26T18:03:34.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.809+0000 7f2e25fc4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f67f00 0x55bfe6f70050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.809+0000 7f2e25fc4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f68820 0x55bfe6f70590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.809+0000 7f2e25fc4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f730e0 0x55bfe6f70ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.809+0000 7f2e25fc4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bfe6f75c50 con 0x55bfe6f68820 2024-04-26T18:03:34.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.809+0000 7f2e25fc4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55bfe6e55b60 con 0x55bfe6f730e0 2024-04-26T18:03:34.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.809+0000 7f2e25fc4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bfe6f75dd0 con 0x55bfe6f67f00 2024-04-26T18:03:34.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.809+0000 7f2e25fc4740 10 client.0 _ll_register_callbacks cb 0x55bfe6e7a2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:03:34.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.809+0000 7f2e20844640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f68820 0x55bfe6f70590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.809+0000 7f2e20844640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f68820 0x55bfe6f70590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37714/0 (socket says 172.21.15.31:37714) 2024-04-26T18:03:34.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.809+0000 7f2e20844640 1 -- 172.21.15.31:0/1749159568 learned_addr learned my addr 172.21.15.31:0/1749159568 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:34.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.809+0000 7f2e21045640 1 --2- 172.21.15.31:0/1749159568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f67f00 0x55bfe6f70050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.810+0000 7f2e21846640 1 --2- 172.21.15.31:0/1749159568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f730e0 0x55bfe6f70ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.810+0000 7f2e20844640 1 -- 172.21.15.31:0/1749159568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f67f00 msgr2=0x55bfe6f70050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:34.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.810+0000 7f2e20844640 1 --2- 172.21.15.31:0/1749159568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f67f00 0x55bfe6f70050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.810+0000 7f2e20844640 1 -- 172.21.15.31:0/1749159568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f730e0 msgr2=0x55bfe6f70ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:34.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.810+0000 7f2e20844640 1 --2- 172.21.15.31:0/1749159568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f730e0 0x55bfe6f70ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.810+0000 7f2e20844640 1 -- 172.21.15.31:0/1749159568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfe6be0210 con 0x55bfe6f68820 2024-04-26T18:03:34.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.810+0000 7f2e20844640 1 --2- 172.21.15.31:0/1749159568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f68820 0x55bfe6f70590 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c0a4960 tx=0x7f2e0c0cd9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9921e9489e633ad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:34.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.810+0000 7f2e04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:03:34.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.810+0000 7f2e04ff9640 1 -- 172.21.15.31:0/1749159568 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c09bd60 con 0x55bfe6f68820 2024-04-26T18:03:34.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.810+0000 7f2e04ff9640 1 -- 172.21.15.31:0/1749159568 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f2e0c009d20 con 0x55bfe6f68820 2024-04-26T18:03:34.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.810+0000 7f2e25fc4740 1 -- 172.21.15.31:0/1749159568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e0c066d30 con 0x55bfe6f68820 2024-04-26T18:03:34.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.810+0000 7f2e04ff9640 1 -- 172.21.15.31:0/1749159568 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2e0c09bb80 con 0x55bfe6f68820 2024-04-26T18:03:34.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.810+0000 7f2e25fc4740 10 client.16053 fetch_fsmap learned FSMap version 33 2024-04-26T18:03:34.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.810+0000 7f2e25fc4740 1 -- 172.21.15.31:0/1749159568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55bfe6cac620 con 0x55bfe6f68820 2024-04-26T18:03:34.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.811+0000 7f2e04ff9640 1 -- 172.21.15.31:0/1749159568 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e0c066d30 con 0x55bfe6f68820 2024-04-26T18:03:34.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.811+0000 7f2e04ff9640 1 -- 172.21.15.31:0/1749159568 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f2e0c0cdc50 con 0x55bfe6f68820 2024-04-26T18:03:34.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.811+0000 7f2e25fc4740 10 client.16053 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T18:03:34.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.811+0000 7f2e25fc4740 10 client.16053 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:03:34.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.811+0000 7f2e25fc4740 20 client.16053 populate_metadata read hostname 'smithi031' 2024-04-26T18:03:34.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.811+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/1749159568 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bfe6f82fd0 0x55bfe6f853c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.811+0000 7f2e25fc4740 4 client.16053 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:03:34.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.811+0000 7f2e25fc4740 1 -- 172.21.15.31:0/1749159568 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfe6edc070 con 0x55bfe6f82fd0 2024-04-26T18:03:34.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.811+0000 7f2e21045640 1 --2- 172.21.15.31:0/1749159568 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bfe6f82fd0 0x55bfe6f853c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.812+0000 7f2e04ff9640 10 client.16053 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:34.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.812+0000 7f2e21045640 1 --2- 172.21.15.31:0/1749159568 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bfe6f82fd0 0x55bfe6f853c0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ecee58e27475c7b server_cookie=2e7d40418816ca6b in_seq=0 out_seq=0 2024-04-26T18:03:34.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.812+0000 7f2e04ff9640 1 -- 172.21.15.31:0/1749159568 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bfe6edc070 con 0x55bfe6f82fd0 2024-04-26T18:03:34.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.812+0000 7f2e04ff9640 10 client.16053 handle_command_reply: tid=0 2024-04-26T18:03:34.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.812+0000 7f2e25fc4740 1 client.16053 shutdown 2024-04-26T18:03:34.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.812+0000 7f2e25fc4740 1 -- 172.21.15.31:0/1749159568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f68820 msgr2=0x55bfe6f70590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:34.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.812+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/1749159568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f68820 0x55bfe6f70590 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c0a4960 tx=0x7f2e0c0cd9c0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.813+0000 7f2e25fc4740 1 -- 172.21.15.31:0/1749159568 shutdown_connections 2024-04-26T18:03:34.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.813+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/1749159568 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bfe6f82fd0 0x55bfe6f853c0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.813+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/1749159568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f730e0 0x55bfe6f70ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.813+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/1749159568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f68820 0x55bfe6f70590 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.813+0000 7f2e04ff9640 0 client.16053 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:34.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.813+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/1749159568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f67f00 0x55bfe6f70050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.813+0000 7f2e25fc4740 1 -- 172.21.15.31:0/1749159568 >> 172.21.15.31:0/1749159568 conn(0x55bfe6f64410 msgr2=0x55bfe6f6a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:34.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.813+0000 7f2e25fc4740 1 -- 172.21.15.31:0/1749159568 shutdown_connections 2024-04-26T18:03:34.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.813+0000 7f2e25fc4740 1 -- 172.21.15.31:0/1749159568 wait complete. 2024-04-26T18:03:34.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.813+0000 7f2e25fc4740 20 client.16053 trim_cache size 0 max 16384 2024-04-26T18:03:34.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.828+0000 7f2e25fc4740 1 Processor -- start 2024-04-26T18:03:34.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.828+0000 7f2e25fc4740 1 -- start start 2024-04-26T18:03:34.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.828+0000 7f2e25fc4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f67f00 0x55bfe6f7eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.828+0000 7f2e25fc4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f68820 0x55bfe6f6a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.828+0000 7f2e25fc4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f730e0 0x55bfe6f8aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.828+0000 7f2e25fc4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c1b0d80 con 0x55bfe6f67f00 2024-04-26T18:03:34.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.828+0000 7f2e25fc4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c1ae5f0 con 0x55bfe6f730e0 2024-04-26T18:03:34.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.828+0000 7f2e25fc4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c1ac460 con 0x55bfe6f68820 2024-04-26T18:03:34.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.828+0000 7f2e21045640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f67f00 0x55bfe6f7eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.828+0000 7f2e21045640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f67f00 0x55bfe6f7eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37728/0 (socket says 172.21.15.31:37728) 2024-04-26T18:03:34.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.828+0000 7f2e21045640 1 -- 172.21.15.31:0/2515000828 learned_addr learned my addr 172.21.15.31:0/2515000828 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:34.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.828+0000 7f2e21846640 1 --2- 172.21.15.31:0/2515000828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f730e0 0x55bfe6f8aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.828+0000 7f2e20844640 1 --2- 172.21.15.31:0/2515000828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f68820 0x55bfe6f6a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.829+0000 7f2e21045640 1 -- 172.21.15.31:0/2515000828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f68820 msgr2=0x55bfe6f6a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.829+0000 7f2e21045640 1 --2- 172.21.15.31:0/2515000828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f68820 0x55bfe6f6a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.829+0000 7f2e21045640 1 -- 172.21.15.31:0/2515000828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f730e0 msgr2=0x55bfe6f8aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.829+0000 7f2e21045640 1 --2- 172.21.15.31:0/2515000828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f730e0 0x55bfe6f8aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.829+0000 7f2e21045640 1 -- 172.21.15.31:0/2515000828 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfe6f5e9c0 con 0x55bfe6f67f00 2024-04-26T18:03:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.829+0000 7f2e21045640 1 --2- 172.21.15.31:0/2515000828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f67f00 0x55bfe6f7eb10 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f2e18000c00 tx=0x7f2e18005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c1731c73d39420 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.829+0000 7f2e06ffd640 1 -- 172.21.15.31:0/2515000828 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18003de0 con 0x55bfe6f67f00 2024-04-26T18:03:34.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.829+0000 7f2e06ffd640 1 -- 172.21.15.31:0/2515000828 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e180205a0 con 0x55bfe6f67f00 2024-04-26T18:03:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.829+0000 7f2e06ffd640 1 -- 172.21.15.31:0/2515000828 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18010ae0 con 0x55bfe6f67f00 2024-04-26T18:03:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.830+0000 7f2e25fc4740 1 -- 172.21.15.31:0/2515000828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f67f00 msgr2=0x55bfe6f7eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.830+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/2515000828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f67f00 0x55bfe6f7eb10 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f2e18000c00 tx=0x7f2e18005f90 comp rx=0 tx=0).stop 2024-04-26T18:03:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.830+0000 7f2e25fc4740 1 -- 172.21.15.31:0/2515000828 shutdown_connections 2024-04-26T18:03:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.830+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/2515000828 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f730e0 0x55bfe6f8aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.830+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/2515000828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f68820 0x55bfe6f6a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.830+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/2515000828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f67f00 0x55bfe6f7eb10 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.830+0000 7f2e25fc4740 1 -- 172.21.15.31:0/2515000828 >> 172.21.15.31:0/2515000828 conn(0x55bfe6f64410 msgr2=0x55bfe6f708a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.830+0000 7f2e25fc4740 1 -- 172.21.15.31:0/2515000828 shutdown_connections 2024-04-26T18:03:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.830+0000 7f2e25fc4740 1 -- 172.21.15.31:0/2515000828 wait complete. 2024-04-26T18:03:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.830+0000 7f2e25fc4740 1 -- start start 2024-04-26T18:03:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.830+0000 7f2e25fc4740 1 Processor -- start 2024-04-26T18:03:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.830+0000 7f2e25fc4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f67f00 0x55bfe6f7b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e25fc4740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f68820 0x55bfe6f6c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e25fc4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f730e0 0x55bfe6f6cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e25fc4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c1acab0 con 0x55bfe6f67f00 2024-04-26T18:03:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e25fc4740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c1adf20 con 0x55bfe6f68820 2024-04-26T18:03:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e25fc4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c1ad180 con 0x55bfe6f730e0 2024-04-26T18:03:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e25fc4740 10 client.0 _ll_register_callbacks cb 0x7f2e14031f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:03:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e21045640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f67f00 0x55bfe6f7b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e21045640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f67f00 0x55bfe6f7b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37738/0 (socket says 172.21.15.31:37738) 2024-04-26T18:03:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e21045640 1 -- 172.21.15.31:0/3207282959 learned_addr learned my addr 172.21.15.31:0/3207282959 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e20844640 1 --2- 172.21.15.31:0/3207282959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f68820 0x55bfe6f6c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e21846640 1 --2- 172.21.15.31:0/3207282959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f730e0 0x55bfe6f6cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e21045640 1 -- 172.21.15.31:0/3207282959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f730e0 msgr2=0x55bfe6f6cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e21045640 1 --2- 172.21.15.31:0/3207282959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f730e0 0x55bfe6f6cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e21045640 1 -- 172.21.15.31:0/3207282959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f68820 msgr2=0x55bfe6f6c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e21045640 1 --2- 172.21.15.31:0/3207282959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f68820 0x55bfe6f6c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e21045640 1 -- 172.21.15.31:0/3207282959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfe6f6d660 con 0x55bfe6f67f00 2024-04-26T18:03:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e21045640 1 --2- 172.21.15.31:0/3207282959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f67f00 0x55bfe6f7b310 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f2e180208b0 tx=0x7f2e180208e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232f24c14b700a2d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:03:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e04ff9640 1 -- 172.21.15.31:0/3207282959 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1802e020 con 0x55bfe6f67f00 2024-04-26T18:03:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e04ff9640 1 -- 172.21.15.31:0/3207282959 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f2e180202f0 con 0x55bfe6f67f00 2024-04-26T18:03:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e25fc4740 1 -- 172.21.15.31:0/3207282959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e180205a0 con 0x55bfe6f67f00 2024-04-26T18:03:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.831+0000 7f2e04ff9640 1 -- 172.21.15.31:0/3207282959 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2e18039050 con 0x55bfe6f67f00 2024-04-26T18:03:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.832+0000 7f2e25fc4740 10 client.16059 fetch_fsmap learned FSMap version 33 2024-04-26T18:03:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.832+0000 7f2e25fc4740 1 -- 172.21.15.31:0/3207282959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55bfe6fe92d0 con 0x55bfe6f67f00 2024-04-26T18:03:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.832+0000 7f2e04ff9640 1 -- 172.21.15.31:0/3207282959 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e180205a0 con 0x55bfe6f67f00 2024-04-26T18:03:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.832+0000 7f2e04ff9640 1 -- 172.21.15.31:0/3207282959 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f2e180051c0 con 0x55bfe6f67f00 2024-04-26T18:03:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.832+0000 7f2e25fc4740 10 client.16059 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T18:03:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.832+0000 7f2e25fc4740 10 client.16059 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:03:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.832+0000 7f2e25fc4740 20 client.16059 populate_metadata read hostname 'smithi031' 2024-04-26T18:03:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.832+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/3207282959 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bfe6fe9a10 0x55bfe6febe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.832+0000 7f2e25fc4740 4 client.16059 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:03:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.832+0000 7f2e25fc4740 1 -- 172.21.15.31:0/3207282959 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bfe6ed5ee0 con 0x55bfe6fe9a10 2024-04-26T18:03:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.832+0000 7f2e20844640 1 --2- 172.21.15.31:0/3207282959 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bfe6fe9a10 0x55bfe6febe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.832+0000 7f2e20844640 1 --2- 172.21.15.31:0/3207282959 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bfe6fe9a10 0x55bfe6febe30 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95f260fa5fb29d43 server_cookie=12024c68e1c4257a in_seq=0 out_seq=0 2024-04-26T18:03:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.833+0000 7f2e04ff9640 10 client.16059 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.841+0000 7f2e04ff9640 1 -- 172.21.15.31:0/3207282959 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bfe6ed5ee0 con 0x55bfe6fe9a10 2024-04-26T18:03:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.841+0000 7f2e04ff9640 10 client.16059 handle_command_reply: tid=0 2024-04-26T18:03:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.841+0000 7f2e25fc4740 1 client.16059 shutdown 2024-04-26T18:03:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e25fc4740 1 -- 172.21.15.31:0/3207282959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f67f00 msgr2=0x55bfe6f7b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/3207282959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f67f00 0x55bfe6f7b310 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f2e180208b0 tx=0x7f2e180208e0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e25fc4740 1 -- 172.21.15.31:0/3207282959 shutdown_connections 2024-04-26T18:03:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/3207282959 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55bfe6fe9a10 0x55bfe6febe30 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/3207282959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe6f730e0 0x55bfe6f6cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/3207282959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55bfe6f68820 0x55bfe6f6c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e04ff9640 0 client.16059 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e25fc4740 1 --2- 172.21.15.31:0/3207282959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfe6f67f00 0x55bfe6f7b310 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e25fc4740 1 -- 172.21.15.31:0/3207282959 >> 172.21.15.31:0/3207282959 conn(0x55bfe6f64410 msgr2=0x55bfe6f6fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e25fc4740 1 -- 172.21.15.31:0/3207282959 shutdown_connections 2024-04-26T18:03:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e25fc4740 1 -- 172.21.15.31:0/3207282959 wait complete. 2024-04-26T18:03:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e25fc4740 20 client.16059 trim_cache size 0 max 16384 2024-04-26T18:03:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:03:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:03:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:03:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:03:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e232a9640 1 -- 172.21.15.31:0/2046614763 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2e080a3dc0 msgr2=0x7f2e080a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e232a9640 1 --2- 172.21.15.31:0/2046614763 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2e080a3dc0 0x7f2e080a6280 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2e14004af0 tx=0x7f2e140095c0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e232a9640 1 -- 172.21.15.31:0/2046614763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c102210 msgr2=0x7f2e1c19a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e232a9640 1 --2- 172.21.15.31:0/2046614763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c102210 0x7f2e1c19a3f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00d950 tx=0x7f2e0c00de10 comp rx=0 tx=0).stop 2024-04-26T18:03:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e232a9640 1 -- 172.21.15.31:0/2046614763 shutdown_connections 2024-04-26T18:03:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e232a9640 1 --2- 172.21.15.31:0/2046614763 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2e1c107860 0x7f2e1c1a4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e232a9640 1 --2- 172.21.15.31:0/2046614763 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2e080a3dc0 0x7f2e080a6280 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e232a9640 1 --2- 172.21.15.31:0/2046614763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e1c102210 0x7f2e1c19a3f0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e232a9640 1 --2- 172.21.15.31:0/2046614763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e1c101850 0x7f2e1c199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:34.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.842+0000 7f2e232a9640 1 -- 172.21.15.31:0/2046614763 >> 172.21.15.31:0/2046614763 conn(0x7f2e1c077bb0 msgr2=0x7f2e1c0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:34.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.843+0000 7f2e232a9640 1 -- 172.21.15.31:0/2046614763 shutdown_connections 2024-04-26T18:03:34.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:34.843+0000 7f2e232a9640 1 -- 172.21.15.31:0/2046614763 wait complete. 2024-04-26T18:03:35.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:35 smithi186 ceph-mon[32797]: pgmap v217: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-26T18:03:35.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:35 smithi031 ceph-mon[29164]: pgmap v217: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-26T18:03:35.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:35 smithi097 ceph-mon[32419]: pgmap v217: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-26T18:03:37.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:37 smithi186 ceph-mon[32797]: pgmap v218: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-26T18:03:37.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T18:03:37.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:37 smithi031 ceph-mon[29164]: pgmap v218: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-26T18:03:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:37 smithi097 ceph-mon[32419]: pgmap v218: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-26T18:03:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:03:38.099 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 375.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:03:39.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:39 smithi031 ceph-mon[29164]: pgmap v219: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-26T18:03:39.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:39 smithi186 ceph-mon[32797]: pgmap v219: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-26T18:03:39.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.348+0000 7f8bfac45700 1 -- 172.21.15.31:0/3328242329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4103470 msgr2=0x7f8bf41038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:39.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.348+0000 7f8bfac45700 1 --2- 172.21.15.31:0/3328242329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4103470 0x7f8bf41038f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009990 tx=0x7f8be8009ca0 comp rx=0 tx=0).stop 2024-04-26T18:03:39.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.349+0000 7f8bfac45700 1 -- 172.21.15.31:0/3328242329 shutdown_connections 2024-04-26T18:03:39.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.349+0000 7f8bfac45700 1 --2- 172.21.15.31:0/3328242329 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf41094c0 0x7f8bf41098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:39.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.349+0000 7f8bfac45700 1 --2- 172.21.15.31:0/3328242329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bf4103e30 0x7f8bf4110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:39.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.349+0000 7f8bfac45700 1 --2- 172.21.15.31:0/3328242329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4103470 0x7f8bf41038f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:39.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.349+0000 7f8bfac45700 1 -- 172.21.15.31:0/3328242329 >> 172.21.15.31:0/3328242329 conn(0x7f8bf407a580 msgr2=0x7f8bf407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:39.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.349+0000 7f8bfac45700 1 -- 172.21.15.31:0/3328242329 shutdown_connections 2024-04-26T18:03:39.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.349+0000 7f8bfac45700 1 -- 172.21.15.31:0/3328242329 wait complete. 2024-04-26T18:03:39.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.349+0000 7f8bfac45700 1 Processor -- start 2024-04-26T18:03:39.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.349+0000 7f8bfac45700 1 -- start start 2024-04-26T18:03:39.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bfac45700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf4103470 0x7f8bf419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:39.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bfac45700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4103e30 0x7f8bf419fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:39.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bfac45700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bf41094c0 0x7f8bf4199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:39.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bfac45700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf41135f0 con 0x7f8bf4103e30 2024-04-26T18:03:39.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bfac45700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf4113470 con 0x7f8bf4103470 2024-04-26T18:03:39.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bfac45700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf4113770 con 0x7f8bf41094c0 2024-04-26T18:03:39.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bf3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4103e30 0x7f8bf419fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:39.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bf3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4103e30 0x7f8bf419fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37748/0 (socket says 172.21.15.31:37748) 2024-04-26T18:03:39.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bf3fff700 1 -- 172.21.15.31:0/552993428 learned_addr learned my addr 172.21.15.31:0/552993428 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:39.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bf91e2700 1 --2- 172.21.15.31:0/552993428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bf41094c0 0x7f8bf4199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:39.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bf89e1700 1 --2- 172.21.15.31:0/552993428 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf4103470 0x7f8bf419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:39.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bf3fff700 1 -- 172.21.15.31:0/552993428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bf41094c0 msgr2=0x7f8bf4199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:39.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bf3fff700 1 --2- 172.21.15.31:0/552993428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bf41094c0 0x7f8bf4199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:39.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bf3fff700 1 -- 172.21.15.31:0/552993428 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf4103470 msgr2=0x7f8bf419f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:39.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bf3fff700 1 --2- 172.21.15.31:0/552993428 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf4103470 0x7f8bf419f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:39.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bf3fff700 1 -- 172.21.15.31:0/552993428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf419a1b0 con 0x7f8bf4103e30 2024-04-26T18:03:39.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bf3fff700 1 --2- 172.21.15.31:0/552993428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4103e30 0x7f8bf419fdd0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8be8000c00 tx=0x7f8be80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:39.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.350+0000 7f8bf1ffb700 1 -- 172.21.15.31:0/552993428 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be8027070 con 0x7f8bf4103e30 2024-04-26T18:03:39.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.351+0000 7f8bfac45700 1 -- 172.21.15.31:0/552993428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf419a440 con 0x7f8bf4103e30 2024-04-26T18:03:39.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.351+0000 7f8bf1ffb700 1 -- 172.21.15.31:0/552993428 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8be8003f20 con 0x7f8bf4103e30 2024-04-26T18:03:39.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.351+0000 7f8bf1ffb700 1 -- 172.21.15.31:0/552993428 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be8022ca0 con 0x7f8bf4103e30 2024-04-26T18:03:39.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.351+0000 7f8bfac45700 1 -- 172.21.15.31:0/552993428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf41a6760 con 0x7f8bf4103e30 2024-04-26T18:03:39.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.352+0000 7f8bfac45700 1 -- 172.21.15.31:0/552993428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bf410b930 con 0x7f8bf4103e30 2024-04-26T18:03:39.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.353+0000 7f8bf1ffb700 1 -- 172.21.15.31:0/552993428 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8be802a070 con 0x7f8bf4103e30 2024-04-26T18:03:39.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.353+0000 7f8bf1ffb700 1 --2- 172.21.15.31:0/552993428 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f8bdc0a3db0 0x7f8bdc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:39.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.353+0000 7f8bf89e1700 1 --2- 172.21.15.31:0/552993428 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f8bdc0a3db0 0x7f8bdc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:39.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.354+0000 7f8bf1ffb700 1 -- 172.21.15.31:0/552993428 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f8be80d1a00 con 0x7f8bf4103e30 2024-04-26T18:03:39.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.354+0000 7f8bf89e1700 1 --2- 172.21.15.31:0/552993428 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f8bdc0a3db0 0x7f8bdc0a6270 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8bf419ab90 tx=0x7f8be400b410 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:39.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.357+0000 7f8bf1ffb700 1 -- 172.21.15.31:0/552993428 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8be8031070 con 0x7f8bf4103e30 2024-04-26T18:03:39.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:39 smithi097 ceph-mon[32419]: pgmap v219: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-26T18:03:39.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.621+0000 7f8bfac45700 1 -- 172.21.15.31:0/552993428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bf419af80 con 0x7f8bf4103e30 2024-04-26T18:03:39.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.621+0000 7f8bf1ffb700 1 -- 172.21.15.31:0/552993428 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11875 (secure 0 0 0) 0x7f8be8013070 con 0x7f8bf4103e30 2024-04-26T18:03:39.622 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:03:39.622 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T18:03:05.764896+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/3765855684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":3765855684},{"type":"v1","addr":"172.21.15.97:6839","nonce":3765855684}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24473,24446,24457]},"id":1}]} 2024-04-26T18:03:39.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.623+0000 7f8bfac45700 1 -- 172.21.15.31:0/552993428 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f8bdc0a3db0 msgr2=0x7f8bdc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:39.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.623+0000 7f8bfac45700 1 --2- 172.21.15.31:0/552993428 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f8bdc0a3db0 0x7f8bdc0a6270 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8bf419ab90 tx=0x7f8be400b410 comp rx=0 tx=0).stop 2024-04-26T18:03:39.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.623+0000 7f8bfac45700 1 -- 172.21.15.31:0/552993428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4103e30 msgr2=0x7f8bf419fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:39.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.623+0000 7f8bfac45700 1 --2- 172.21.15.31:0/552993428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4103e30 0x7f8bf419fdd0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8be8000c00 tx=0x7f8be80047b0 comp rx=0 tx=0).stop 2024-04-26T18:03:39.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.623+0000 7f8bfac45700 1 -- 172.21.15.31:0/552993428 shutdown_connections 2024-04-26T18:03:39.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.623+0000 7f8bfac45700 1 --2- 172.21.15.31:0/552993428 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f8bdc0a3db0 0x7f8bdc0a6270 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:39.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.623+0000 7f8bfac45700 1 --2- 172.21.15.31:0/552993428 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8bf4103470 0x7f8bf419f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:39.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.623+0000 7f8bfac45700 1 --2- 172.21.15.31:0/552993428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bf41094c0 0x7f8bf4199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:39.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.623+0000 7f8bfac45700 1 --2- 172.21.15.31:0/552993428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4103e30 0x7f8bf419fdd0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:39.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.624+0000 7f8bfac45700 1 -- 172.21.15.31:0/552993428 >> 172.21.15.31:0/552993428 conn(0x7f8bf407a580 msgr2=0x7f8bf41022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:39.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.624+0000 7f8bfac45700 1 -- 172.21.15.31:0/552993428 shutdown_connections 2024-04-26T18:03:39.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:39.624+0000 7f8bfac45700 1 -- 172.21.15.31:0/552993428 wait complete. 2024-04-26T18:03:39.625 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 33 2024-04-26T18:03:40.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:40 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/552993428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:03:40.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:40 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/552993428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:03:40.282 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24473 quiesce db --set-id 13a7907 --release --await' 2024-04-26T18:03:40.282 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24473 quiesce db --set-id 13a7907 --release --await 2024-04-26T18:03:40.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:40 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/552993428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:03:40.574 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:03:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:41 smithi186 ceph-mon[32797]: pgmap v220: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.7 KiB/s wr, 7 op/s 2024-04-26T18:03:41.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:41 smithi031 ceph-mon[29164]: pgmap v220: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.7 KiB/s wr, 7 op/s 2024-04-26T18:03:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:41 smithi097 ceph-mon[32419]: pgmap v220: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.7 KiB/s wr, 7 op/s 2024-04-26T18:03:42.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:42 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:03:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.313+0000 7fec0be9e700 1 -- 172.21.15.31:0/3483366658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0410fd70 msgr2=0x7fec04112160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.313+0000 7fec0be9e700 1 --2- 172.21.15.31:0/3483366658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0410fd70 0x7fec04112160 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7febfc00b0a0 tx=0x7febfc00b3b0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.313+0000 7fec0be9e700 1 -- 172.21.15.31:0/3483366658 shutdown_connections 2024-04-26T18:03:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.313+0000 7fec0be9e700 1 --2- 172.21.15.31:0/3483366658 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec04104370 0x7fec0406baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.313+0000 7fec0be9e700 1 --2- 172.21.15.31:0/3483366658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0410fd70 0x7fec04112160 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.313+0000 7fec0be9e700 1 --2- 172.21.15.31:0/3483366658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec0406c030 0x7fec0406c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.313+0000 7fec0be9e700 1 -- 172.21.15.31:0/3483366658 >> 172.21.15.31:0/3483366658 conn(0x7fec040792d0 msgr2=0x7fec040796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.313+0000 7fec0be9e700 1 -- 172.21.15.31:0/3483366658 shutdown_connections 2024-04-26T18:03:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.313+0000 7fec0be9e700 1 -- 172.21.15.31:0/3483366658 wait complete. 2024-04-26T18:03:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.314+0000 7fec0be9e700 1 Processor -- start 2024-04-26T18:03:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.314+0000 7fec0be9e700 1 -- start start 2024-04-26T18:03:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.314+0000 7fec0be9e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0406c030 0x7fec041a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.314+0000 7fec0be9e700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec04104370 0x7fec041a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.314+0000 7fec0be9e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec0410fd70 0x7fec0419ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.314+0000 7fec0be9e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec041157e0 con 0x7fec0406c030 2024-04-26T18:03:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.314+0000 7fec0be9e700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fec04115660 con 0x7fec04104370 2024-04-26T18:03:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.314+0000 7fec0be9e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec04115960 con 0x7fec0410fd70 2024-04-26T18:03:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.314+0000 7fec09c3a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0406c030 0x7fec041a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.314+0000 7fec09439700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec04104370 0x7fec041a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.314+0000 7fec09c3a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0406c030 0x7fec041a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37768/0 (socket says 172.21.15.31:37768) 2024-04-26T18:03:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.314+0000 7fec09c3a700 1 -- 172.21.15.31:0/3976798198 learned_addr learned my addr 172.21.15.31:0/3976798198 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.314+0000 7fec0a43b700 1 --2- 172.21.15.31:0/3976798198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec0410fd70 0x7fec0419ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.315+0000 7fec09439700 1 -- 172.21.15.31:0/3976798198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec0410fd70 msgr2=0x7fec0419ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.315+0000 7fec09439700 1 --2- 172.21.15.31:0/3976798198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec0410fd70 0x7fec0419ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.315+0000 7fec09439700 1 -- 172.21.15.31:0/3976798198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0406c030 msgr2=0x7fec041a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.315+0000 7fec09439700 1 --2- 172.21.15.31:0/3976798198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0406c030 0x7fec041a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.315+0000 7fec09439700 1 -- 172.21.15.31:0/3976798198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0419c1a0 con 0x7fec04104370 2024-04-26T18:03:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.315+0000 7fec09c3a700 1 --2- 172.21.15.31:0/3976798198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0406c030 0x7fec041a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:03:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.315+0000 7fec09439700 1 --2- 172.21.15.31:0/3976798198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec04104370 0x7fec041a1d70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7febf400d950 tx=0x7febf400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.315+0000 7febfaffd700 1 -- 172.21.15.31:0/3976798198 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf4016070 con 0x7fec04104370 2024-04-26T18:03:42.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.315+0000 7febfaffd700 1 -- 172.21.15.31:0/3976798198 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febf400fd40 con 0x7fec04104370 2024-04-26T18:03:42.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.315+0000 7fec0be9e700 1 -- 172.21.15.31:0/3976798198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec0419c490 con 0x7fec04104370 2024-04-26T18:03:42.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.315+0000 7febfaffd700 1 -- 172.21.15.31:0/3976798198 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf400e3b0 con 0x7fec04104370 2024-04-26T18:03:42.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.315+0000 7fec0be9e700 1 -- 172.21.15.31:0/3976798198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec041a8700 con 0x7fec04104370 2024-04-26T18:03:42.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.317+0000 7febfaffd700 1 -- 172.21.15.31:0/3976798198 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7febf4019070 con 0x7fec04104370 2024-04-26T18:03:42.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.318+0000 7febfaffd700 1 --2- 172.21.15.31:0/3976798198 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7febf00a3db0 0x7febf00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.319+0000 7fec09c3a700 1 --2- 172.21.15.31:0/3976798198 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7febf00a3db0 0x7febf00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.319+0000 7febfaffd700 1 -- 172.21.15.31:0/3976798198 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7febf40d0220 con 0x7fec04104370 2024-04-26T18:03:42.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.319+0000 7fec09c3a700 1 --2- 172.21.15.31:0/3976798198 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7febf00a3db0 0x7febf00a6270 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fec0406bda0 tx=0x7fec0000b500 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:42.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.320+0000 7fec16730b80 1 Processor -- start 2024-04-26T18:03:42.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.320+0000 7fec16730b80 1 -- start start 2024-04-26T18:03:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.320+0000 7fec16730b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd9513580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.320+0000 7fec16730b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd9513ac0 0x55ecd951e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.320+0000 7fec16730b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd951e970 0x55ecd9520d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.320+0000 7fec16730b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ecd9466840 con 0x55ecd951e970 2024-04-26T18:03:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.320+0000 7fec16730b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ecd94056e0 con 0x55ecd95131a0 2024-04-26T18:03:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.320+0000 7fec16730b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ecd9462800 con 0x55ecd9513ac0 2024-04-26T18:03:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.320+0000 7fec09c3a700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd9513580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.320+0000 7fec0a43b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd951e970 0x55ecd9520d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.320+0000 7fec09c3a700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd9513580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:45258/0 (socket says 172.21.15.31:45258) 2024-04-26T18:03:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.320+0000 7fec0a43b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd951e970 0x55ecd9520d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37782/0 (socket says 172.21.15.31:37782) 2024-04-26T18:03:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.320+0000 7fec09c3a700 1 -- 172.21.15.31:0/364545465 learned_addr learned my addr 172.21.15.31:0/364545465 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.321+0000 7fec09439700 1 --2- 172.21.15.31:0/364545465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd9513ac0 0x55ecd951e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.321+0000 7fec09c3a700 1 -- 172.21.15.31:0/364545465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd9513ac0 msgr2=0x55ecd951e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.321+0000 7fec09c3a700 1 --2- 172.21.15.31:0/364545465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd9513ac0 0x55ecd951e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.321+0000 7fec09c3a700 1 -- 172.21.15.31:0/364545465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd951e970 msgr2=0x55ecd9520d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.321+0000 7fec09c3a700 1 --2- 172.21.15.31:0/364545465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd951e970 0x55ecd9520d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.321+0000 7fec09c3a700 1 -- 172.21.15.31:0/364545465 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecd93b39d0 con 0x55ecd95131a0 2024-04-26T18:03:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.321+0000 7fec09c3a700 1 --2- 172.21.15.31:0/364545465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd9513580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fec00011790 tx=0x7fec00013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c3caadb34f81491 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.321+0000 7febeffff700 1 -- 172.21.15.31:0/364545465 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec00004220 con 0x55ecd95131a0 2024-04-26T18:03:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.321+0000 7febeffff700 1 -- 172.21.15.31:0/364545465 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec000043c0 con 0x55ecd95131a0 2024-04-26T18:03:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.321+0000 7febeffff700 1 -- 172.21.15.31:0/364545465 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0001e820 con 0x55ecd95131a0 2024-04-26T18:03:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.322+0000 7fec16730b80 1 -- 172.21.15.31:0/364545465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 msgr2=0x55ecd9513580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.322+0000 7fec16730b80 1 --2- 172.21.15.31:0/364545465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd9513580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fec00011790 tx=0x7fec00013000 comp rx=0 tx=0).stop 2024-04-26T18:03:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.322+0000 7fec16730b80 1 -- 172.21.15.31:0/364545465 shutdown_connections 2024-04-26T18:03:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.322+0000 7fec16730b80 1 --2- 172.21.15.31:0/364545465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd9513580 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.322+0000 7fec16730b80 1 --2- 172.21.15.31:0/364545465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd951e970 0x55ecd9520d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.322+0000 7fec16730b80 1 --2- 172.21.15.31:0/364545465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd9513ac0 0x55ecd951e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.322+0000 7fec16730b80 1 -- 172.21.15.31:0/364545465 >> 172.21.15.31:0/364545465 conn(0x55ecd9440e40 msgr2=0x55ecd950f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.322+0000 7fec16730b80 1 -- 172.21.15.31:0/364545465 shutdown_connections 2024-04-26T18:03:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.323+0000 7fec16730b80 1 -- 172.21.15.31:0/364545465 wait complete. 2024-04-26T18:03:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.323+0000 7fec16730b80 1 -- start start 2024-04-26T18:03:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.323+0000 7fec16730b80 1 Processor -- start 2024-04-26T18:03:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec16730b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd951e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec16730b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd9513ac0 0x55ecd951b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec16730b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd951e970 0x55ecd951b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec16730b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ecd9521540 con 0x55ecd951e970 2024-04-26T18:03:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec16730b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55ecd93cca50 con 0x55ecd95131a0 2024-04-26T18:03:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec16730b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ecd95216c0 con 0x55ecd9513ac0 2024-04-26T18:03:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec16730b80 10 client.0 _ll_register_callbacks cb 0x55ecd93fa680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:03:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec09439700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd9513ac0 0x55ecd951b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec09c3a700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd951e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec09439700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd9513ac0 0x55ecd951b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:46908/0 (socket says 172.21.15.31:46908) 2024-04-26T18:03:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec09439700 1 -- 172.21.15.31:0/5221126 learned_addr learned my addr 172.21.15.31:0/5221126 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec09c3a700 1 -- 172.21.15.31:0/5221126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd9513ac0 msgr2=0x55ecd951b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec09c3a700 1 --2- 172.21.15.31:0/5221126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd9513ac0 0x55ecd951b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec09c3a700 1 -- 172.21.15.31:0/5221126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd951e970 msgr2=0x55ecd951b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec09c3a700 1 --2- 172.21.15.31:0/5221126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd951e970 0x55ecd951b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec09c3a700 1 -- 172.21.15.31:0/5221126 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecd951c000 con 0x55ecd95131a0 2024-04-26T18:03:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.324+0000 7fec09439700 1 --2- 172.21.15.31:0/5221126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd9513ac0 0x55ecd951b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:03:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.325+0000 7fec09c3a700 1 --2- 172.21.15.31:0/5221126 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd951e100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fec00005950 tx=0x7fec000042b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d03b9c55323e15a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.325+0000 7febed7fa700 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T18:03:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.325+0000 7febed7fa700 1 -- 172.21.15.31:0/5221126 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec00036060 con 0x55ecd95131a0 2024-04-26T18:03:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.325+0000 7febed7fa700 1 -- 172.21.15.31:0/5221126 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7fec00003b90 con 0x55ecd95131a0 2024-04-26T18:03:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.325+0000 7fec16730b80 1 -- 172.21.15.31:0/5221126 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec000043c0 con 0x55ecd95131a0 2024-04-26T18:03:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.325+0000 7febed7fa700 1 -- 172.21.15.31:0/5221126 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fec0001e420 con 0x55ecd95131a0 2024-04-26T18:03:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.325+0000 7fec16730b80 10 client.25757 fetch_fsmap learned FSMap version 33 2024-04-26T18:03:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.325+0000 7fec16730b80 1 -- 172.21.15.31:0/5221126 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ecd952c000 con 0x55ecd95131a0 2024-04-26T18:03:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.325+0000 7febed7fa700 1 -- 172.21.15.31:0/5221126 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec000043c0 con 0x55ecd95131a0 2024-04-26T18:03:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.326+0000 7febed7fa700 1 -- 172.21.15.31:0/5221126 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7fec00026660 con 0x55ecd95131a0 2024-04-26T18:03:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.326+0000 7fec16730b80 10 client.25757 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T18:03:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.326+0000 7fec16730b80 10 client.25757 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T18:03:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.326+0000 7fec16730b80 20 client.25757 populate_metadata read hostname 'smithi031' 2024-04-26T18:03:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.326+0000 7fec16730b80 1 --2- 172.21.15.31:0/5221126 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ecd952e5e0 0x55ecd95309d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.326+0000 7fec16730b80 4 client.25757 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:03:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.326+0000 7fec16730b80 1 -- 172.21.15.31:0/5221126 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecd9474220 con 0x55ecd952e5e0 2024-04-26T18:03:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.326+0000 7fec09439700 1 --2- 172.21.15.31:0/5221126 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ecd952e5e0 0x55ecd95309d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.326+0000 7febed7fa700 10 client.25757 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.326+0000 7fec09439700 1 --2- 172.21.15.31:0/5221126 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ecd952e5e0 0x55ecd95309d0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0a020abbf04dc35 server_cookie=3993ef5c58404499 in_seq=0 out_seq=0 2024-04-26T18:03:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.327+0000 7febed7fa700 1 -- 172.21.15.31:0/5221126 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ecd9474220 con 0x55ecd952e5e0 2024-04-26T18:03:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.327+0000 7febed7fa700 10 client.25757 handle_command_reply: tid=0 2024-04-26T18:03:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.327+0000 7fec16730b80 1 client.25757 shutdown 2024-04-26T18:03:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.327+0000 7fec16730b80 1 -- 172.21.15.31:0/5221126 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 msgr2=0x55ecd951e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.327+0000 7fec16730b80 1 --2- 172.21.15.31:0/5221126 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd951e100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fec00005950 tx=0x7fec000042b0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.327+0000 7fec16730b80 1 -- 172.21.15.31:0/5221126 shutdown_connections 2024-04-26T18:03:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.327+0000 7fec16730b80 1 --2- 172.21.15.31:0/5221126 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ecd952e5e0 0x55ecd95309d0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.327+0000 7fec16730b80 1 --2- 172.21.15.31:0/5221126 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd951e100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.327+0000 7fec16730b80 1 --2- 172.21.15.31:0/5221126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd951e970 0x55ecd951b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.327+0000 7fec16730b80 1 --2- 172.21.15.31:0/5221126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd9513ac0 0x55ecd951b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.327+0000 7febed7fa700 0 client.25757 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.327+0000 7fec16730b80 1 -- 172.21.15.31:0/5221126 >> 172.21.15.31:0/5221126 conn(0x55ecd9440e40 msgr2=0x55ecd950f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:42.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.327+0000 7fec16730b80 1 -- 172.21.15.31:0/5221126 shutdown_connections 2024-04-26T18:03:42.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.327+0000 7fec16730b80 1 -- 172.21.15.31:0/5221126 wait complete. 2024-04-26T18:03:42.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.328+0000 7fec16730b80 20 client.25757 trim_cache size 0 max 16384 2024-04-26T18:03:42.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.348+0000 7fec16730b80 1 Processor -- start 2024-04-26T18:03:42.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.348+0000 7fec16730b80 1 -- start start 2024-04-26T18:03:42.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.348+0000 7fec16730b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd950f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.348+0000 7fec16730b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd9513ac0 0x55ecd9516b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.348+0000 7fec16730b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd951e970 0x55ecd951d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.348+0000 7fec16730b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec041b27a0 con 0x55ecd9513ac0 2024-04-26T18:03:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.348+0000 7fec16730b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fec041affd0 con 0x55ecd95131a0 2024-04-26T18:03:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.348+0000 7fec16730b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec041ade40 con 0x55ecd951e970 2024-04-26T18:03:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.348+0000 7fec09439700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd9513ac0 0x55ecd9516b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.348+0000 7fec09c3a700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd950f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.348+0000 7fec09439700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd9513ac0 0x55ecd9516b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37806/0 (socket says 172.21.15.31:37806) 2024-04-26T18:03:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.348+0000 7fec09439700 1 -- 172.21.15.31:0/2324093183 learned_addr learned my addr 172.21.15.31:0/2324093183 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.348+0000 7fec0a43b700 1 --2- 172.21.15.31:0/2324093183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd951e970 0x55ecd951d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.349+0000 7fec09c3a700 1 -- 172.21.15.31:0/2324093183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd951e970 msgr2=0x55ecd951d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.349+0000 7fec09c3a700 1 --2- 172.21.15.31:0/2324093183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd951e970 0x55ecd951d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.349+0000 7fec09c3a700 1 -- 172.21.15.31:0/2324093183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd9513ac0 msgr2=0x55ecd9516b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.349+0000 7fec09c3a700 1 --2- 172.21.15.31:0/2324093183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd9513ac0 0x55ecd9516b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.349+0000 7fec09c3a700 1 -- 172.21.15.31:0/2324093183 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecd9506f40 con 0x55ecd95131a0 2024-04-26T18:03:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.349+0000 7fec09c3a700 1 --2- 172.21.15.31:0/2324093183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd950f840 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fec00011790 tx=0x7fec0000bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92a9adc9b8fcfc60 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:42.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.349+0000 7febef7fe700 1 -- 172.21.15.31:0/2324093183 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec00036040 con 0x55ecd95131a0 2024-04-26T18:03:42.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.349+0000 7febef7fe700 1 -- 172.21.15.31:0/2324093183 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec00026e50 con 0x55ecd95131a0 2024-04-26T18:03:42.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.349+0000 7febef7fe700 1 -- 172.21.15.31:0/2324093183 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0001e820 con 0x55ecd95131a0 2024-04-26T18:03:42.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.350+0000 7fec16730b80 1 -- 172.21.15.31:0/2324093183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 msgr2=0x55ecd950f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.350+0000 7fec16730b80 1 --2- 172.21.15.31:0/2324093183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd950f840 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fec00011790 tx=0x7fec0000bf60 comp rx=0 tx=0).stop 2024-04-26T18:03:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.350+0000 7fec16730b80 1 -- 172.21.15.31:0/2324093183 shutdown_connections 2024-04-26T18:03:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.350+0000 7fec16730b80 1 --2- 172.21.15.31:0/2324093183 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd95131a0 0x55ecd950f840 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.350+0000 7fec16730b80 1 --2- 172.21.15.31:0/2324093183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd951e970 0x55ecd951d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.350+0000 7fec16730b80 1 --2- 172.21.15.31:0/2324093183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd9513ac0 0x55ecd9516b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.350+0000 7fec16730b80 1 -- 172.21.15.31:0/2324093183 >> 172.21.15.31:0/2324093183 conn(0x55ecd9440e40 msgr2=0x55ecd951cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.350+0000 7fec16730b80 1 -- 172.21.15.31:0/2324093183 shutdown_connections 2024-04-26T18:03:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.350+0000 7fec16730b80 1 -- 172.21.15.31:0/2324093183 wait complete. 2024-04-26T18:03:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.350+0000 7fec16730b80 1 -- start start 2024-04-26T18:03:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.350+0000 7fec16730b80 1 Processor -- start 2024-04-26T18:03:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec16730b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd95131a0 0x55ecd95225c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec16730b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd9513ac0 0x55ecd9522b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec16730b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd951e970 0x55ecd9523040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec16730b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec041ae490 con 0x55ecd95131a0 2024-04-26T18:03:42.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec16730b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fec041af900 con 0x55ecd9513ac0 2024-04-26T18:03:42.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec16730b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec041aeb60 con 0x55ecd951e970 2024-04-26T18:03:42.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec16730b80 10 client.0 _ll_register_callbacks cb 0x55ecd93fa680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:03:42.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec09439700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd9513ac0 0x55ecd9522b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec09c3a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd95131a0 0x55ecd95225c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec09439700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd9513ac0 0x55ecd9522b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:45294/0 (socket says 172.21.15.31:45294) 2024-04-26T18:03:42.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec09c3a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd95131a0 0x55ecd95225c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37820/0 (socket says 172.21.15.31:37820) 2024-04-26T18:03:42.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec09439700 1 -- 172.21.15.31:0/1859708783 learned_addr learned my addr 172.21.15.31:0/1859708783 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:42.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec0a43b700 1 --2- 172.21.15.31:0/1859708783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd951e970 0x55ecd9523040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec09439700 1 -- 172.21.15.31:0/1859708783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd951e970 msgr2=0x55ecd9523040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:42.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec09439700 1 --2- 172.21.15.31:0/1859708783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd951e970 0x55ecd9523040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec09439700 1 -- 172.21.15.31:0/1859708783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd95131a0 msgr2=0x55ecd95225c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:42.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec09439700 1 --2- 172.21.15.31:0/1859708783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd95131a0 0x55ecd95225c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.351+0000 7fec09439700 1 -- 172.21.15.31:0/1859708783 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecd9593bc0 con 0x55ecd9513ac0 2024-04-26T18:03:42.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7fec09439700 1 --2- 172.21.15.31:0/1859708783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd9513ac0 0x55ecd9522b00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7febf40a48c0 tx=0x7febf4010460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ba4004cc9919664 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:42.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7febed7fa700 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-04-26T18:03:42.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7febed7fa700 1 -- 172.21.15.31:0/1859708783 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf40b9070 con 0x55ecd9513ac0 2024-04-26T18:03:42.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7febed7fa700 1 -- 172.21.15.31:0/1859708783 <== mon.2 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7febf40a0120 con 0x55ecd9513ac0 2024-04-26T18:03:42.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7fec16730b80 1 -- 172.21.15.31:0/1859708783 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec00026e50 con 0x55ecd9513ac0 2024-04-26T18:03:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7febed7fa700 1 -- 172.21.15.31:0/1859708783 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7febf407f050 con 0x55ecd9513ac0 2024-04-26T18:03:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7fec16730b80 10 client.25763 fetch_fsmap learned FSMap version 33 2024-04-26T18:03:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7fec16730b80 1 -- 172.21.15.31:0/1859708783 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ecd9594f40 con 0x55ecd9513ac0 2024-04-26T18:03:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7febed7fa700 1 -- 172.21.15.31:0/1859708783 <== mon.2 v2:172.21.15.97:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec00026e50 con 0x55ecd9513ac0 2024-04-26T18:03:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7febed7fa700 1 -- 172.21.15.31:0/1859708783 <== mon.2 v2:172.21.15.97:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7febf409c3e0 con 0x55ecd9513ac0 2024-04-26T18:03:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7fec16730b80 10 client.25763 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T18:03:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7fec16730b80 10 client.25763 resolve_mds: validated gid 24473 aka daemon mds.b 2024-04-26T18:03:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7fec16730b80 20 client.25763 populate_metadata read hostname 'smithi031' 2024-04-26T18:03:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7fec16730b80 1 --2- 172.21.15.31:0/1859708783 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ecd95955f0 0x55ecd9597a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7fec16730b80 4 client.25763 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "13a7907", "release": true, "await": true}] 2024-04-26T18:03:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.352+0000 7fec16730b80 1 -- 172.21.15.31:0/1859708783 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "quiesce db", "set_id": "13a7907", "release": true, "await": true}) -- 0x55ecd946e2c0 con 0x55ecd95955f0 2024-04-26T18:03:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.353+0000 7fec09c3a700 1 --2- 172.21.15.31:0/1859708783 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ecd95955f0 0x55ecd9597a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.353+0000 7fec09c3a700 1 --2- 172.21.15.31:0/1859708783 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ecd95955f0 0x55ecd9597a50 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dcab657f7e9764d server_cookie=ca90cb0225c014d9 in_seq=0 out_seq=0 2024-04-26T18:03:42.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.353+0000 7febed7fa700 10 client.25763 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:42.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.374+0000 7febed7fa700 1 -- 172.21.15.31:0/1859708783 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55ecd946e2c0 con 0x55ecd95955f0 2024-04-26T18:03:42.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.374+0000 7febed7fa700 10 client.25763 handle_command_reply: tid=0 2024-04-26T18:03:42.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.374+0000 7fec16730b80 1 client.25763 shutdown 2024-04-26T18:03:42.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.374+0000 7fec16730b80 1 -- 172.21.15.31:0/1859708783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd9513ac0 msgr2=0x55ecd9522b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:42.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.374+0000 7fec16730b80 1 --2- 172.21.15.31:0/1859708783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd9513ac0 0x55ecd9522b00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7febf40a48c0 tx=0x7febf4010460 comp rx=0 tx=0).stop 2024-04-26T18:03:42.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.374+0000 7fec16730b80 1 -- 172.21.15.31:0/1859708783 shutdown_connections 2024-04-26T18:03:42.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.374+0000 7fec16730b80 1 --2- 172.21.15.31:0/1859708783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecd951e970 0x55ecd9523040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec16730b80 1 --2- 172.21.15.31:0/1859708783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecd95131a0 0x55ecd95225c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec16730b80 1 --2- 172.21.15.31:0/1859708783 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55ecd95955f0 0x55ecd9597a50 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec16730b80 1 --2- 172.21.15.31:0/1859708783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55ecd9513ac0 0x55ecd9522b00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7febed7fa700 0 client.25763 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:42.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec16730b80 1 -- 172.21.15.31:0/1859708783 >> 172.21.15.31:0/1859708783 conn(0x55ecd9440e40 msgr2=0x55ecd951ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:42.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec16730b80 1 -- 172.21.15.31:0/1859708783 shutdown_connections 2024-04-26T18:03:42.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec16730b80 1 -- 172.21.15.31:0/1859708783 wait complete. 2024-04-26T18:03:42.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec16730b80 20 client.25763 trim_cache size 0 max 16384 2024-04-26T18:03:42.377 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T18:03:42.377 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 33, 2024-04-26T18:03:42.377 INFO:teuthology.orchestra.run.smithi031.stdout: "leader": 24473, 2024-04-26T18:03:42.377 INFO:teuthology.orchestra.run.smithi031.stdout: "set_version": 20, 2024-04-26T18:03:42.377 INFO:teuthology.orchestra.run.smithi031.stdout: "sets": { 2024-04-26T18:03:42.377 INFO:teuthology.orchestra.run.smithi031.stdout: "13a7907": { 2024-04-26T18:03:42.377 INFO:teuthology.orchestra.run.smithi031.stdout: "version": 20, 2024-04-26T18:03:42.378 INFO:teuthology.orchestra.run.smithi031.stdout: "age_ref": 0.0, 2024-04-26T18:03:42.378 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T18:03:42.378 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RELEASED", 2024-04-26T18:03:42.378 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T18:03:42.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:03:42.378 INFO:teuthology.orchestra.run.smithi031.stdout: "timeout": 60.0, 2024-04-26T18:03:42.378 INFO:teuthology.orchestra.run.smithi031.stdout: "expiration": 91.7, 2024-04-26T18:03:42.378 INFO:teuthology.orchestra.run.smithi031.stdout: "members": { 2024-04-26T18:03:42.378 INFO:teuthology.orchestra.run.smithi031.stdout: "file:/": { 2024-04-26T18:03:42.378 INFO:teuthology.orchestra.run.smithi031.stdout: "excluded": false, 2024-04-26T18:03:42.378 INFO:teuthology.orchestra.run.smithi031.stdout: "state": { 2024-04-26T18:03:42.378 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RELEASED", 2024-04-26T18:03:42.379 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0 2024-04-26T18:03:42.379 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:03:42.379 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:03:42.379 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:03:42.379 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:03:42.379 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:03:42.379 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T18:03:42.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec0be9e700 1 -- 172.21.15.31:0/3976798198 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7febf00a3db0 msgr2=0x7febf00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:42.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec0be9e700 1 --2- 172.21.15.31:0/3976798198 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7febf00a3db0 0x7febf00a6270 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fec0406bda0 tx=0x7fec0000b500 comp rx=0 tx=0).stop 2024-04-26T18:03:42.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec0be9e700 1 -- 172.21.15.31:0/3976798198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec04104370 msgr2=0x7fec041a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:42.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec0be9e700 1 --2- 172.21.15.31:0/3976798198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec04104370 0x7fec041a1d70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7febf400d950 tx=0x7febf400dc60 comp rx=0 tx=0).stop 2024-04-26T18:03:42.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec0be9e700 1 -- 172.21.15.31:0/3976798198 shutdown_connections 2024-04-26T18:03:42.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec0be9e700 1 --2- 172.21.15.31:0/3976798198 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7febf00a3db0 0x7febf00a6270 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec0be9e700 1 --2- 172.21.15.31:0/3976798198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec0410fd70 0x7fec0419ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec0be9e700 1 --2- 172.21.15.31:0/3976798198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec0406c030 0x7fec041a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec0be9e700 1 --2- 172.21.15.31:0/3976798198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec04104370 0x7fec041a1d70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:42.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.375+0000 7fec0be9e700 1 -- 172.21.15.31:0/3976798198 >> 172.21.15.31:0/3976798198 conn(0x7fec040792d0 msgr2=0x7fec041016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:42.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.376+0000 7fec0be9e700 1 -- 172.21.15.31:0/3976798198 shutdown_connections 2024-04-26T18:03:42.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:03:42.376+0000 7fec0be9e700 1 -- 172.21.15.31:0/3976798198 wait complete. 2024-04-26T18:03:42.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:42 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:03:42.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:42 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:03:43.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 380.0 s, 362.4 tps, lat 2127.171 ms stddev 8654.288 2024-04-26T18:03:43.110 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '13a7907', seconds elapsed: 44.6 2024-04-26T18:03:43.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:43 smithi031 ceph-mon[29164]: pgmap v221: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-26T18:03:43.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:43 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:03:43.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:43 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:03:43.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:43 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:03:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:43 smithi097 ceph-mon[32419]: pgmap v221: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-26T18:03:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:43 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:03:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:43 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:03:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:43 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:03:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:43 smithi186 ceph-mon[32797]: pgmap v221: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-26T18:03:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:43 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:03:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:43 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:03:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:43 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:03:45.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:45 smithi031 ceph-mon[29164]: pgmap v222: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-04-26T18:03:45.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:45 smithi097 ceph-mon[32419]: pgmap v222: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-04-26T18:03:45.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:45 smithi186 ceph-mon[32797]: pgmap v222: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-04-26T18:03:47.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:47 smithi031 ceph-mon[29164]: pgmap v223: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 438 KiB/s wr, 63 op/s 2024-04-26T18:03:47.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:47 smithi097 ceph-mon[32419]: pgmap v223: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 438 KiB/s wr, 63 op/s 2024-04-26T18:03:47.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:47 smithi186 ceph-mon[32797]: pgmap v223: 97 pgs: 97 active+clean; 3.8 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 438 KiB/s wr, 63 op/s 2024-04-26T18:03:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.977+0000 7f4472c59640 1 -- 172.21.15.31:0/348965778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f446c100e30 msgr2=0x7f446c1012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.977+0000 7f4472c59640 1 --2- 172.21.15.31:0/348965778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f446c100e30 0x7f446c1012b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4454009930 tx=0x7f445402a460 comp rx=0 tx=0).stop 2024-04-26T18:03:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.980+0000 7f4472c59640 1 -- 172.21.15.31:0/348965778 shutdown_connections 2024-04-26T18:03:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.980+0000 7f4472c59640 1 --2- 172.21.15.31:0/348965778 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f446c1017f0 0x7f446c10f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.980+0000 7f4472c59640 1 --2- 172.21.15.31:0/348965778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f446c100e30 0x7f446c1012b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.980+0000 7f4472c59640 1 --2- 172.21.15.31:0/348965778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f446c107910 0x7f446c107cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.980+0000 7f4472c59640 1 -- 172.21.15.31:0/348965778 >> 172.21.15.31:0/348965778 conn(0x7f446c077dc0 msgr2=0x7f446c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.983+0000 7f4472c59640 1 -- 172.21.15.31:0/348965778 shutdown_connections 2024-04-26T18:03:47.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.987+0000 7f4472c59640 1 -- 172.21.15.31:0/348965778 wait complete. 2024-04-26T18:03:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.989+0000 7f4472c59640 1 Processor -- start 2024-04-26T18:03:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.989+0000 7f4472c59640 1 -- start start 2024-04-26T18:03:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.989+0000 7f4472c59640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f446c100e30 0x7f446c19fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.989+0000 7f4472c59640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f446c1017f0 0x7f446c1a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.989+0000 7f4472c59640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f446c107910 0x7f446c10e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.990+0000 7f4472c59640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f446c112dc0 con 0x7f446c107910 2024-04-26T18:03:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.990+0000 7f4472c59640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f446c112c40 con 0x7f446c1017f0 2024-04-26T18:03:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.990+0000 7f4472c59640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f446c112f40 con 0x7f446c100e30 2024-04-26T18:03:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.990+0000 7f44709f5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f446c100e30 0x7f446c19fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.992+0000 7f44711f6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f446c107910 0x7f446c10e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.992+0000 7f44709f5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f446c100e30 0x7f446c19fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:41646/0 (socket says 172.21.15.31:41646) 2024-04-26T18:03:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.992+0000 7f44709f5640 1 -- 172.21.15.31:0/1164485737 learned_addr learned my addr 172.21.15.31:0/1164485737 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:47.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.992+0000 7f446bfff640 1 --2- 172.21.15.31:0/1164485737 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f446c1017f0 0x7f446c1a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.994+0000 7f44709f5640 1 -- 172.21.15.31:0/1164485737 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f446c1017f0 msgr2=0x7f446c1a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.994+0000 7f44709f5640 1 --2- 172.21.15.31:0/1164485737 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f446c1017f0 0x7f446c1a02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.994+0000 7f44709f5640 1 -- 172.21.15.31:0/1164485737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f446c107910 msgr2=0x7f446c10e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.994+0000 7f44709f5640 1 --2- 172.21.15.31:0/1164485737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f446c107910 0x7f446c10e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.994+0000 7f44709f5640 1 -- 172.21.15.31:0/1164485737 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f446c10ef90 con 0x7f446c100e30 2024-04-26T18:03:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.994+0000 7f44709f5640 1 --2- 172.21.15.31:0/1164485737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f446c100e30 0x7f446c19fd80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f445c00ba50 tx=0x7f445c00bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.997+0000 7f4469ffb640 1 -- 172.21.15.31:0/1164485737 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445c019070 con 0x7f446c100e30 2024-04-26T18:03:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.997+0000 7f4469ffb640 1 -- 172.21.15.31:0/1164485737 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f445c00cbe0 con 0x7f446c100e30 2024-04-26T18:03:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:47.997+0000 7f4469ffb640 1 -- 172.21.15.31:0/1164485737 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445c007630 con 0x7f446c100e30 2024-04-26T18:03:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.000+0000 7f4472c59640 1 -- 172.21.15.31:0/1164485737 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f446c10f280 con 0x7f446c100e30 2024-04-26T18:03:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.000+0000 7f4472c59640 1 -- 172.21.15.31:0/1164485737 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f446c19b8b0 con 0x7f446c100e30 2024-04-26T18:03:48.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.004+0000 7f4475973740 1 Processor -- start 2024-04-26T18:03:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.006+0000 7f4469ffb640 1 -- 172.21.15.31:0/1164485737 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f445c014440 con 0x7f446c100e30 2024-04-26T18:03:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.008+0000 7f4469ffb640 1 --2- 172.21.15.31:0/1164485737 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f44580a3ff0 0x7f44580a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.008+0000 7f4469ffb640 1 -- 172.21.15.31:0/1164485737 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f445c0d04b0 con 0x7f446c100e30 2024-04-26T18:03:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.009+0000 7f4475973740 1 -- start start 2024-04-26T18:03:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.009+0000 7f4475973740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e448f00 0x56518e4492e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.009+0000 7f4475973740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e449820 0x56518e453ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.009+0000 7f44711f6640 1 --2- 172.21.15.31:0/1164485737 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f44580a3ff0 0x7f44580a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.009+0000 7f4475973740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e4540e0 0x56518e4564d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.009+0000 7f4475973740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56518e3af460 con 0x56518e449820 2024-04-26T18:03:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.009+0000 7f4475973740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56518e369590 con 0x56518e448f00 2024-04-26T18:03:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.009+0000 7f4475973740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56518e3ab420 con 0x56518e4540e0 2024-04-26T18:03:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.012+0000 7f44711f6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e4540e0 0x56518e4564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:48.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.012+0000 7f44711f6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e4540e0 0x56518e4564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:41648/0 (socket says 172.21.15.31:41648) 2024-04-26T18:03:48.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.012+0000 7f44711f6640 1 -- 172.21.15.31:0/4149236589 learned_addr learned my addr 172.21.15.31:0/4149236589 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:48.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.013+0000 7f44711f6640 1 -- 172.21.15.31:0/4149236589 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e448f00 msgr2=0x56518e4492e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T18:03:48.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.013+0000 7f44711f6640 1 --2- 172.21.15.31:0/4149236589 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e448f00 0x56518e4492e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.013+0000 7f44711f6640 1 -- 172.21.15.31:0/4149236589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e449820 msgr2=0x56518e453ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:48.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.013+0000 7f44711f6640 1 --2- 172.21.15.31:0/4149236589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e449820 0x56518e453ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.013+0000 7f44711f6640 1 -- 172.21.15.31:0/4149236589 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56518e3760d0 con 0x56518e4540e0 2024-04-26T18:03:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.014+0000 7f44711f6640 1 --2- 172.21.15.31:0/4149236589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e4540e0 0x56518e4564d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4464015860 tx=0x7f4464015d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cbf10f094ea2ef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:48.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.015+0000 7f4462ffd640 1 -- 172.21.15.31:0/4149236589 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4464027070 con 0x56518e4540e0 2024-04-26T18:03:48.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.015+0000 7f4462ffd640 1 -- 172.21.15.31:0/4149236589 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4464018c00 con 0x56518e4540e0 2024-04-26T18:03:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.016+0000 7f4475973740 1 -- 172.21.15.31:0/4149236589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e4540e0 msgr2=0x56518e4564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.016+0000 7f4475973740 1 --2- 172.21.15.31:0/4149236589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e4540e0 0x56518e4564d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4464015860 tx=0x7f4464015d20 comp rx=0 tx=0).stop 2024-04-26T18:03:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.016+0000 7f44711f6640 1 --2- 172.21.15.31:0/1164485737 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f44580a3ff0 0x7f44580a64b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4464017750 tx=0x7f446400f000 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.018+0000 7f4475973740 1 -- 172.21.15.31:0/4149236589 shutdown_connections 2024-04-26T18:03:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.018+0000 7f4475973740 1 --2- 172.21.15.31:0/4149236589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e4540e0 0x56518e4564d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.018+0000 7f4475973740 1 --2- 172.21.15.31:0/4149236589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e449820 0x56518e453ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.018+0000 7f4475973740 1 --2- 172.21.15.31:0/4149236589 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e448f00 0x56518e4492e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.018+0000 7f4475973740 1 -- 172.21.15.31:0/4149236589 >> 172.21.15.31:0/4149236589 conn(0x56518e445410 msgr2=0x56518e4457f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.024+0000 7f4475973740 1 -- 172.21.15.31:0/4149236589 shutdown_connections 2024-04-26T18:03:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.027+0000 7f4475973740 1 -- 172.21.15.31:0/4149236589 wait complete. 2024-04-26T18:03:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.027+0000 7f4475973740 1 -- start start 2024-04-26T18:03:48.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.027+0000 7f4475973740 1 Processor -- start 2024-04-26T18:03:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.029+0000 7f4475973740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e448f00 0x56518e4510c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.029+0000 7f4475973740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e449820 0x56518e451600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.029+0000 7f4475973740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e4540e0 0x56518e451b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.029+0000 7f4475973740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56518e456cb0 con 0x56518e449820 2024-04-26T18:03:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.029+0000 7f4475973740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56518e336b60 con 0x56518e4540e0 2024-04-26T18:03:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.029+0000 7f4475973740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56518e456e30 con 0x56518e448f00 2024-04-26T18:03:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.029+0000 7f4475973740 10 client.0 _ll_register_callbacks cb 0x56518e35b2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:03:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.032+0000 7f44711f6640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e4540e0 0x56518e451b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.032+0000 7f44711f6640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e4540e0 0x56518e451b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:52086/0 (socket says 172.21.15.31:52086) 2024-04-26T18:03:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.032+0000 7f44711f6640 1 -- 172.21.15.31:0/1443848549 learned_addr learned my addr 172.21.15.31:0/1443848549 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.033+0000 7f44709f5640 1 --2- 172.21.15.31:0/1443848549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e448f00 0x56518e4510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.034+0000 7f446bfff640 1 --2- 172.21.15.31:0/1443848549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e449820 0x56518e451600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.035+0000 7f446bfff640 1 -- 172.21.15.31:0/1443848549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e448f00 msgr2=0x56518e4510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.035+0000 7f446bfff640 1 --2- 172.21.15.31:0/1443848549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e448f00 0x56518e4510c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.035+0000 7f446bfff640 1 -- 172.21.15.31:0/1443848549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e4540e0 msgr2=0x56518e451b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.035+0000 7f446bfff640 1 --2- 172.21.15.31:0/1443848549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e4540e0 0x56518e451b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.035+0000 7f446bfff640 1 -- 172.21.15.31:0/1443848549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56518e0c1210 con 0x56518e449820 2024-04-26T18:03:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.035+0000 7f446bfff640 1 --2- 172.21.15.31:0/1443848549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e449820 0x56518e451600 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f4454009850 tx=0x7f44540043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d481abe1cb4979 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.036+0000 7f444bfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:03:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.036+0000 7f444bfff640 1 -- 172.21.15.31:0/1443848549 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445403f070 con 0x56518e449820 2024-04-26T18:03:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.036+0000 7f444bfff640 1 -- 172.21.15.31:0/1443848549 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f4454005450 con 0x56518e449820 2024-04-26T18:03:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.037+0000 7f4475973740 1 -- 172.21.15.31:0/1443848549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4464018c00 con 0x56518e449820 2024-04-26T18:03:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.039+0000 7f444bfff640 1 -- 172.21.15.31:0/1443848549 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4454049050 con 0x56518e449820 2024-04-26T18:03:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.039+0000 7f4475973740 10 client.16086 fetch_fsmap learned FSMap version 33 2024-04-26T18:03:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.039+0000 7f4475973740 1 -- 172.21.15.31:0/1443848549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56518e18d620 con 0x56518e449820 2024-04-26T18:03:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.043+0000 7f444bfff640 1 -- 172.21.15.31:0/1443848549 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4464018c00 con 0x56518e449820 2024-04-26T18:03:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.043+0000 7f444bfff640 1 -- 172.21.15.31:0/1443848549 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f4454003740 con 0x56518e449820 2024-04-26T18:03:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.043+0000 7f4475973740 10 client.16086 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T18:03:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.043+0000 7f4475973740 10 client.16086 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:03:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.043+0000 7f4475973740 20 client.16086 populate_metadata read hostname 'smithi031' 2024-04-26T18:03:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.043+0000 7f4475973740 1 --2- 172.21.15.31:0/1443848549 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56518e464720 0x56518e466ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.043+0000 7f4475973740 4 client.16086 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:03:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.043+0000 7f4475973740 1 -- 172.21.15.31:0/1443848549 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56518e3bd070 con 0x56518e464720 2024-04-26T18:03:48.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.043+0000 7f44709f5640 1 --2- 172.21.15.31:0/1443848549 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56518e464720 0x56518e466ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.044+0000 7f444bfff640 10 client.16086 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.044+0000 7f44709f5640 1 --2- 172.21.15.31:0/1443848549 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56518e464720 0x56518e466ba0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcdf80d177893ffc server_cookie=183511c0a76d3f46 in_seq=0 out_seq=0 2024-04-26T18:03:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.045+0000 7f444bfff640 1 -- 172.21.15.31:0/1443848549 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56518e3bd070 con 0x56518e464720 2024-04-26T18:03:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.045+0000 7f444bfff640 10 client.16086 handle_command_reply: tid=0 2024-04-26T18:03:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.046+0000 7f4475973740 1 client.16086 shutdown 2024-04-26T18:03:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.046+0000 7f4475973740 1 -- 172.21.15.31:0/1443848549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e449820 msgr2=0x56518e451600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.046+0000 7f4475973740 1 --2- 172.21.15.31:0/1443848549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e449820 0x56518e451600 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f4454009850 tx=0x7f44540043b0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.046+0000 7f4475973740 1 -- 172.21.15.31:0/1443848549 shutdown_connections 2024-04-26T18:03:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.046+0000 7f4475973740 1 --2- 172.21.15.31:0/1443848549 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56518e464720 0x56518e466ba0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.046+0000 7f4475973740 1 --2- 172.21.15.31:0/1443848549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e4540e0 0x56518e451b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.046+0000 7f4475973740 1 --2- 172.21.15.31:0/1443848549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e449820 0x56518e451600 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.046+0000 7f4475973740 1 --2- 172.21.15.31:0/1443848549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e448f00 0x56518e4510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.046+0000 7f444bfff640 0 client.16086 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.046+0000 7f4475973740 1 -- 172.21.15.31:0/1443848549 >> 172.21.15.31:0/1443848549 conn(0x56518e445410 msgr2=0x56518e44b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.046+0000 7f4475973740 1 -- 172.21.15.31:0/1443848549 shutdown_connections 2024-04-26T18:03:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.046+0000 7f4475973740 1 -- 172.21.15.31:0/1443848549 wait complete. 2024-04-26T18:03:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.046+0000 7f4475973740 20 client.16086 trim_cache size 0 max 16384 2024-04-26T18:03:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.070+0000 7f4475973740 1 Processor -- start 2024-04-26T18:03:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.070+0000 7f4475973740 1 -- start start 2024-04-26T18:03:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.070+0000 7f4475973740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e448f00 0x56518e44b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.070+0000 7f4475973740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e449820 0x56518e451960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.070+0000 7f4475973740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e4540e0 0x56518e46c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.071+0000 7f4475973740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f446c1c8450 con 0x56518e448f00 2024-04-26T18:03:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.071+0000 7f4475973740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f446c073d40 con 0x56518e449820 2024-04-26T18:03:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.071+0000 7f4475973740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f446c071bd0 con 0x56518e4540e0 2024-04-26T18:03:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.071+0000 7f446bfff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e449820 0x56518e451960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.071+0000 7f446bfff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e449820 0x56518e451960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:52102/0 (socket says 172.21.15.31:52102) 2024-04-26T18:03:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.071+0000 7f446bfff640 1 -- 172.21.15.31:0/1064498248 learned_addr learned my addr 172.21.15.31:0/1064498248 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.071+0000 7f44711f6640 1 --2- 172.21.15.31:0/1064498248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e4540e0 0x56518e46c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.071+0000 7f44709f5640 1 --2- 172.21.15.31:0/1064498248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e448f00 0x56518e44b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.071+0000 7f44711f6640 1 -- 172.21.15.31:0/1064498248 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e449820 msgr2=0x56518e451960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.071+0000 7f44711f6640 1 --2- 172.21.15.31:0/1064498248 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e449820 0x56518e451960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.071+0000 7f44711f6640 1 -- 172.21.15.31:0/1064498248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e448f00 msgr2=0x56518e44b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.071+0000 7f44711f6640 1 --2- 172.21.15.31:0/1064498248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e448f00 0x56518e44b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.071+0000 7f44711f6640 1 -- 172.21.15.31:0/1064498248 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56518e4564d0 con 0x56518e4540e0 2024-04-26T18:03:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.072+0000 7f44709f5640 1 --2- 172.21.15.31:0/1064498248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e448f00 0x56518e44b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:03:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.072+0000 7f44711f6640 1 --2- 172.21.15.31:0/1064498248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e4540e0 0x56518e46c4b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4464015830 tx=0x7f4464009870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79466d6f185d33ba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:48.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.073+0000 7f44627fc640 1 -- 172.21.15.31:0/1064498248 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4464032070 con 0x56518e4540e0 2024-04-26T18:03:48.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.073+0000 7f44627fc640 1 -- 172.21.15.31:0/1064498248 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4464022a90 con 0x56518e4540e0 2024-04-26T18:03:48.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.073+0000 7f44627fc640 1 -- 172.21.15.31:0/1064498248 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4464027040 con 0x56518e4540e0 2024-04-26T18:03:48.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.074+0000 7f4475973740 1 -- 172.21.15.31:0/1064498248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e4540e0 msgr2=0x56518e46c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.074+0000 7f4475973740 1 --2- 172.21.15.31:0/1064498248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e4540e0 0x56518e46c4b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4464015830 tx=0x7f4464009870 comp rx=0 tx=0).stop 2024-04-26T18:03:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.074+0000 7f4475973740 1 -- 172.21.15.31:0/1064498248 shutdown_connections 2024-04-26T18:03:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.074+0000 7f4475973740 1 --2- 172.21.15.31:0/1064498248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e4540e0 0x56518e46c4b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.074+0000 7f4475973740 1 --2- 172.21.15.31:0/1064498248 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e449820 0x56518e451960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.074+0000 7f4475973740 1 --2- 172.21.15.31:0/1064498248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e448f00 0x56518e44b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.074+0000 7f4475973740 1 -- 172.21.15.31:0/1064498248 >> 172.21.15.31:0/1064498248 conn(0x56518e445410 msgr2=0x56518e453180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.074+0000 7f4475973740 1 -- 172.21.15.31:0/1064498248 shutdown_connections 2024-04-26T18:03:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.074+0000 7f4475973740 1 -- 172.21.15.31:0/1064498248 wait complete. 2024-04-26T18:03:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.074+0000 7f4475973740 1 -- start start 2024-04-26T18:03:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.074+0000 7f4475973740 1 Processor -- start 2024-04-26T18:03:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f4475973740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e448f00 0x56518e45c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f4475973740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e449820 0x56518e44d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f4475973740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e4540e0 0x56518e44deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f44709f5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e448f00 0x56518e45c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f44709f5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e448f00 0x56518e45c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:41682/0 (socket says 172.21.15.31:41682) 2024-04-26T18:03:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f44709f5640 1 -- 172.21.15.31:0/2887141101 learned_addr learned my addr 172.21.15.31:0/2887141101 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:03:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f4475973740 1 -- 172.21.15.31:0/2887141101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f446c072220 con 0x56518e4540e0 2024-04-26T18:03:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f4475973740 1 -- 172.21.15.31:0/2887141101 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f446c073670 con 0x56518e449820 2024-04-26T18:03:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f4475973740 1 -- 172.21.15.31:0/2887141101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f446c0728d0 con 0x56518e448f00 2024-04-26T18:03:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f4475973740 10 client.0 _ll_register_callbacks cb 0x7f4454009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:03:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f44711f6640 1 --2- 172.21.15.31:0/2887141101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e4540e0 0x56518e44deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f44709f5640 1 -- 172.21.15.31:0/2887141101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e449820 msgr2=0x56518e44d970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T18:03:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f44709f5640 1 --2- 172.21.15.31:0/2887141101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e449820 0x56518e44d970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f44709f5640 1 -- 172.21.15.31:0/2887141101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e4540e0 msgr2=0x56518e44deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:03:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f44709f5640 1 --2- 172.21.15.31:0/2887141101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e4540e0 0x56518e44deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.075+0000 7f44709f5640 1 -- 172.21.15.31:0/2887141101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56518e44e570 con 0x56518e448f00 2024-04-26T18:03:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.076+0000 7f44711f6640 1 --2- 172.21.15.31:0/2887141101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e4540e0 0x56518e44deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:03:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.076+0000 7f44709f5640 1 --2- 172.21.15.31:0/2887141101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e448f00 0x56518e45c400 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f445c09c4b0 tx=0x7f445c00c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaae77419d78c04b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:03:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.076+0000 7f444bfff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T18:03:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.076+0000 7f444bfff640 1 -- 172.21.15.31:0/2887141101 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445c0a5190 con 0x56518e448f00 2024-04-26T18:03:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.076+0000 7f444bfff640 1 -- 172.21.15.31:0/2887141101 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f445c098b90 con 0x56518e448f00 2024-04-26T18:03:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.076+0000 7f4475973740 1 -- 172.21.15.31:0/2887141101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4464022a90 con 0x56518e448f00 2024-04-26T18:03:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.076+0000 7f444bfff640 1 -- 172.21.15.31:0/2887141101 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f445c0bb2e0 con 0x56518e448f00 2024-04-26T18:03:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.076+0000 7f4475973740 10 client.25690 fetch_fsmap learned FSMap version 33 2024-04-26T18:03:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.076+0000 7f4475973740 1 -- 172.21.15.31:0/2887141101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56518e4cac60 con 0x56518e448f00 2024-04-26T18:03:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.076+0000 7f444bfff640 1 -- 172.21.15.31:0/2887141101 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4464022a90 con 0x56518e448f00 2024-04-26T18:03:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.077+0000 7f444bfff640 1 -- 172.21.15.31:0/2887141101 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f445c0a4d00 con 0x56518e448f00 2024-04-26T18:03:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.077+0000 7f4475973740 10 client.25690 fetch_fsmap finished waiting for FSMap version 33 2024-04-26T18:03:48.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.077+0000 7f4475973740 10 client.25690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:03:48.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.077+0000 7f4475973740 20 client.25690 populate_metadata read hostname 'smithi031' 2024-04-26T18:03:48.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.077+0000 7f4475973740 1 --2- 172.21.15.31:0/2887141101 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56518e4cb390 0x56518e4cd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:03:48.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.077+0000 7f4475973740 4 client.25690 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:03:48.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.077+0000 7f4475973740 1 -- 172.21.15.31:0/2887141101 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56518e3b6ee0 con 0x56518e4cb390 2024-04-26T18:03:48.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.077+0000 7f446bfff640 1 --2- 172.21.15.31:0/2887141101 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56518e4cb390 0x56518e4cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:03:48.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.077+0000 7f446bfff640 1 --2- 172.21.15.31:0/2887141101 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56518e4cb390 0x56518e4cd780 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fca394453c2c41df server_cookie=8718e648cb26ad in_seq=0 out_seq=0 2024-04-26T18:03:48.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.077+0000 7f444bfff640 10 client.25690 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 385.0 s, 3254.1 tps, lat 30.933 ms stddev 32.384 2024-04-26T18:03:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.096+0000 7f444bfff640 1 -- 172.21.15.31:0/2887141101 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56518e3b6ee0 con 0x56518e4cb390 2024-04-26T18:03:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.096+0000 7f444bfff640 10 client.25690 handle_command_reply: tid=0 2024-04-26T18:03:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.096+0000 7f4475973740 1 client.25690 shutdown 2024-04-26T18:03:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.096+0000 7f4475973740 1 -- 172.21.15.31:0/2887141101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e448f00 msgr2=0x56518e45c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.096+0000 7f4475973740 1 --2- 172.21.15.31:0/2887141101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e448f00 0x56518e45c400 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f445c09c4b0 tx=0x7f445c00c650 comp rx=0 tx=0).stop 2024-04-26T18:03:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.096+0000 7f4475973740 1 -- 172.21.15.31:0/2887141101 shutdown_connections 2024-04-26T18:03:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.096+0000 7f4475973740 1 --2- 172.21.15.31:0/2887141101 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x56518e4cb390 0x56518e4cd780 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.096+0000 7f4475973740 1 --2- 172.21.15.31:0/2887141101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56518e4540e0 0x56518e44deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.096+0000 7f4475973740 1 --2- 172.21.15.31:0/2887141101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56518e449820 0x56518e44d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.096+0000 7f444bfff640 0 client.25690 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:03:48.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.096+0000 7f4475973740 1 --2- 172.21.15.31:0/2887141101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56518e448f00 0x56518e45c400 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.096+0000 7f4475973740 1 -- 172.21.15.31:0/2887141101 >> 172.21.15.31:0/2887141101 conn(0x56518e445410 msgr2=0x56518e453180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:48.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.096+0000 7f4475973740 1 -- 172.21.15.31:0/2887141101 shutdown_connections 2024-04-26T18:03:48.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.096+0000 7f4475973740 1 -- 172.21.15.31:0/2887141101 wait complete. 2024-04-26T18:03:48.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.097+0000 7f4475973740 20 client.25690 trim_cache size 0 max 16384 2024-04-26T18:03:48.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:03:48.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:03:48.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:03:48.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:03:48.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.097+0000 7f4472c59640 1 -- 172.21.15.31:0/1164485737 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f44580a3ff0 msgr2=0x7f44580a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:48.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.097+0000 7f4472c59640 1 --2- 172.21.15.31:0/1164485737 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f44580a3ff0 0x7f44580a64b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4464017750 tx=0x7f446400f000 comp rx=0 tx=0).stop 2024-04-26T18:03:48.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.097+0000 7f4472c59640 1 -- 172.21.15.31:0/1164485737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f446c100e30 msgr2=0x7f446c19fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:03:48.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.097+0000 7f4472c59640 1 --2- 172.21.15.31:0/1164485737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f446c100e30 0x7f446c19fd80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f445c00ba50 tx=0x7f445c00bf10 comp rx=0 tx=0).stop 2024-04-26T18:03:48.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.097+0000 7f4472c59640 1 -- 172.21.15.31:0/1164485737 shutdown_connections 2024-04-26T18:03:48.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.097+0000 7f4472c59640 1 --2- 172.21.15.31:0/1164485737 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f44580a3ff0 0x7f44580a64b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.097+0000 7f4472c59640 1 --2- 172.21.15.31:0/1164485737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f446c107910 0x7f446c10e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.097+0000 7f4472c59640 1 --2- 172.21.15.31:0/1164485737 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f446c1017f0 0x7f446c1a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.097+0000 7f4472c59640 1 --2- 172.21.15.31:0/1164485737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f446c100e30 0x7f446c19fd80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:03:48.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.097+0000 7f4472c59640 1 -- 172.21.15.31:0/1164485737 >> 172.21.15.31:0/1164485737 conn(0x7f446c077dc0 msgr2=0x7f446c105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:03:48.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.097+0000 7f4472c59640 1 -- 172.21.15.31:0/1164485737 shutdown_connections 2024-04-26T18:03:48.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:03:48.097+0000 7f4472c59640 1 -- 172.21.15.31:0/1164485737 wait complete. 2024-04-26T18:03:49.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:49 smithi031 ceph-mon[29164]: pgmap v224: 97 pgs: 97 active+clean; 3.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 2.2 MiB/s wr, 221 op/s 2024-04-26T18:03:49.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:49 smithi097 ceph-mon[32419]: pgmap v224: 97 pgs: 97 active+clean; 3.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 2.2 MiB/s wr, 221 op/s 2024-04-26T18:03:49.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:49 smithi186 ceph-mon[32797]: pgmap v224: 97 pgs: 97 active+clean; 3.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 2.2 MiB/s wr, 221 op/s 2024-04-26T18:03:51.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:51 smithi031 ceph-mon[29164]: pgmap v225: 97 pgs: 97 active+clean; 3.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-04-26T18:03:51.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:51 smithi097 ceph-mon[32419]: pgmap v225: 97 pgs: 97 active+clean; 3.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-04-26T18:03:51.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:51 smithi186 ceph-mon[32797]: pgmap v225: 97 pgs: 97 active+clean; 3.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-04-26T18:03:52.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:52 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:03:52.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:52 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:03:52.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:52 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:03:53.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 390.0 s, 2453.5 tps, lat 40.512 ms stddev 41.855 2024-04-26T18:03:53.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:53 smithi031 ceph-mon[29164]: pgmap v226: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 5.3 MiB/s wr, 517 op/s 2024-04-26T18:03:53.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:53 smithi097 ceph-mon[32419]: pgmap v226: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 5.3 MiB/s wr, 517 op/s 2024-04-26T18:03:53.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:53 smithi186 ceph-mon[32797]: pgmap v226: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 5.3 MiB/s wr, 517 op/s 2024-04-26T18:03:55.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:55 smithi031 ceph-mon[29164]: pgmap v227: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 5.3 MiB/s wr, 513 op/s 2024-04-26T18:03:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:55 smithi097 ceph-mon[32419]: pgmap v227: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 5.3 MiB/s wr, 513 op/s 2024-04-26T18:03:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:55 smithi186 ceph-mon[32797]: pgmap v227: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 5.3 MiB/s wr, 513 op/s 2024-04-26T18:03:57.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:57 smithi031 ceph-mon[29164]: pgmap v228: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 5.9 MiB/s wr, 589 op/s 2024-04-26T18:03:57.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:57 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:03:57.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:57 smithi097 ceph-mon[32419]: pgmap v228: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 5.9 MiB/s wr, 589 op/s 2024-04-26T18:03:57.386 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:57 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:03:57.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:57 smithi186 ceph-mon[32797]: pgmap v228: 97 pgs: 97 active+clean; 3.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 5.9 MiB/s wr, 589 op/s 2024-04-26T18:03:57.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:57 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:03:58.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 395.0 s, 3038.4 tps, lat 33.147 ms stddev 34.731 2024-04-26T18:03:59.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:03:59 smithi097 ceph-mon[32419]: pgmap v229: 97 pgs: 97 active+clean; 3.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 7.7 MiB/s wr, 740 op/s 2024-04-26T18:03:59.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:03:59 smithi186 ceph-mon[32797]: pgmap v229: 97 pgs: 97 active+clean; 3.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 7.7 MiB/s wr, 740 op/s 2024-04-26T18:03:59.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:03:59 smithi031 ceph-mon[29164]: pgmap v229: 97 pgs: 97 active+clean; 3.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 7.7 MiB/s wr, 740 op/s 2024-04-26T18:04:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.201+0000 7f043aeea640 1 -- 172.21.15.31:0/985155676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0434100e30 msgr2=0x7f04341012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.201+0000 7f043aeea640 1 --2- 172.21.15.31:0/985155676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0434100e30 0x7f04341012b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f0420009930 tx=0x7f042002a420 comp rx=0 tx=0).stop 2024-04-26T18:04:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.201+0000 7f043aeea640 1 -- 172.21.15.31:0/985155676 shutdown_connections 2024-04-26T18:04:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.201+0000 7f043aeea640 1 --2- 172.21.15.31:0/985155676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04341017f0 0x7f043410f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.201+0000 7f043aeea640 1 --2- 172.21.15.31:0/985155676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0434100e30 0x7f04341012b0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.201+0000 7f043aeea640 1 --2- 172.21.15.31:0/985155676 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0434107910 0x7f0434107cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.201+0000 7f043aeea640 1 -- 172.21.15.31:0/985155676 >> 172.21.15.31:0/985155676 conn(0x7f0434077dc0 msgr2=0x7f04340ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:04:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.201+0000 7f043aeea640 1 -- 172.21.15.31:0/985155676 shutdown_connections 2024-04-26T18:04:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.201+0000 7f043aeea640 1 -- 172.21.15.31:0/985155676 wait complete. 2024-04-26T18:04:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.201+0000 7f043aeea640 1 Processor -- start 2024-04-26T18:04:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.201+0000 7f043aeea640 1 -- start start 2024-04-26T18:04:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f043aeea640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0434100e30 0x7f0434102010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f043aeea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04341017f0 0x7f0434102550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f043aeea640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0434107910 0x7f0434102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f043aeea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0434112cd0 con 0x7f04341017f0 2024-04-26T18:04:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f043aeea640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f0434112b50 con 0x7f0434100e30 2024-04-26T18:04:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f043aeea640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0434112e50 con 0x7f0434107910 2024-04-26T18:04:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0433fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04341017f0 0x7f0434102550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0433fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04341017f0 0x7f0434102550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60066/0 (socket says 172.21.15.31:60066) 2024-04-26T18:04:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0433fff640 1 -- 172.21.15.31:0/3670972540 learned_addr learned my addr 172.21.15.31:0/3670972540 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:04:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0438c86640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0434100e30 0x7f0434102010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0439487640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0434107910 0x7f0434102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0433fff640 1 -- 172.21.15.31:0/3670972540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0434107910 msgr2=0x7f0434102a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0433fff640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0434107910 0x7f0434102a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0433fff640 1 -- 172.21.15.31:0/3670972540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0434100e30 msgr2=0x7f0434102010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0433fff640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0434100e30 0x7f0434102010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0433fff640 1 -- 172.21.15.31:0/3670972540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04341a7610 con 0x7f04341017f0 2024-04-26T18:04:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0439487640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0434107910 0x7f0434102a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:04:01.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0433fff640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04341017f0 0x7f0434102550 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f0420005940 tx=0x7f04200040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:01.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0431ffb640 1 -- 172.21.15.31:0/3670972540 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042003f030 con 0x7f04341017f0 2024-04-26T18:04:01.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0431ffb640 1 -- 172.21.15.31:0/3670972540 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04200044c0 con 0x7f04341017f0 2024-04-26T18:04:01.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f043aeea640 1 -- 172.21.15.31:0/3670972540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04341a78a0 con 0x7f04341017f0 2024-04-26T18:04:01.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.202+0000 7f0431ffb640 1 -- 172.21.15.31:0/3670972540 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0420035070 con 0x7f04341017f0 2024-04-26T18:04:01.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.203+0000 7f043aeea640 1 -- 172.21.15.31:0/3670972540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04341a7de0 con 0x7f04341017f0 2024-04-26T18:04:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.205+0000 7f0431ffb640 1 -- 172.21.15.31:0/3670972540 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0420042030 con 0x7f04341017f0 2024-04-26T18:04:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.205+0000 7f043dc00740 1 Processor -- start 2024-04-26T18:04:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.205+0000 7f043dc00740 1 -- start start 2024-04-26T18:04:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.205+0000 7f0431ffb640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f04240a3e40 0x7f04240a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.205+0000 7f043dc00740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b35f00 0x55aed5b362e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f043dc00740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b36820 0x55aed5b40ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f043dc00740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b410e0 0x55aed5b434d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f043dc00740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55aed5a9c460 con 0x55aed5b410e0 2024-04-26T18:04:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f043dc00740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55aed5a56590 con 0x55aed5b36820 2024-04-26T18:04:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f043dc00740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55aed5a98420 con 0x55aed5b35f00 2024-04-26T18:04:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f0431ffb640 1 -- 172.21.15.31:0/3670972540 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f04200f0b70 con 0x7f04341017f0 2024-04-26T18:04:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f0439487640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f04240a3e40 0x7f04240a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f0438c86640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b35f00 0x55aed5b362e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f0439487640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b410e0 0x55aed5b434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f0438c86640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b35f00 0x55aed5b362e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:46354/0 (socket says 172.21.15.31:46354) 2024-04-26T18:04:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f0438c86640 1 -- 172.21.15.31:0/2832551203 learned_addr learned my addr 172.21.15.31:0/2832551203 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:04:01.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f0433fff640 1 --2- 172.21.15.31:0/2832551203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b36820 0x55aed5b40ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f0438c86640 1 -- 172.21.15.31:0/2832551203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b36820 msgr2=0x55aed5b40ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:04:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f0438c86640 1 --2- 172.21.15.31:0/2832551203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b36820 0x55aed5b40ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f0438c86640 1 -- 172.21.15.31:0/2832551203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b410e0 msgr2=0x55aed5b434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:04:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f0438c86640 1 --2- 172.21.15.31:0/2832551203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b410e0 0x55aed5b434d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f0438c86640 1 -- 172.21.15.31:0/2832551203 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aed5a630d0 con 0x55aed5b35f00 2024-04-26T18:04:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f0439487640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f04240a3e40 0x7f04240a6300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f042c005c40 tx=0x7f042c009e50 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f0438c86640 1 --2- 172.21.15.31:0/2832551203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b35f00 0x55aed5b362e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0428009c70 tx=0x7f0428011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=638c23bafcccb369 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.206+0000 7f041effd640 1 -- 172.21.15.31:0/2832551203 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042801e070 con 0x55aed5b35f00 2024-04-26T18:04:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.207+0000 7f041effd640 1 -- 172.21.15.31:0/2832551203 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0428009e20 con 0x55aed5b35f00 2024-04-26T18:04:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.207+0000 7f041effd640 1 -- 172.21.15.31:0/2832551203 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0428019690 con 0x55aed5b35f00 2024-04-26T18:04:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.207+0000 7f043dc00740 1 -- 172.21.15.31:0/2832551203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b35f00 msgr2=0x55aed5b362e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.207+0000 7f043dc00740 1 --2- 172.21.15.31:0/2832551203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b35f00 0x55aed5b362e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0428009c70 tx=0x7f0428011810 comp rx=0 tx=0).stop 2024-04-26T18:04:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.207+0000 7f043dc00740 1 -- 172.21.15.31:0/2832551203 shutdown_connections 2024-04-26T18:04:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.207+0000 7f043dc00740 1 --2- 172.21.15.31:0/2832551203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b410e0 0x55aed5b434d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.207+0000 7f043dc00740 1 --2- 172.21.15.31:0/2832551203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b36820 0x55aed5b40ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.207+0000 7f043dc00740 1 --2- 172.21.15.31:0/2832551203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b35f00 0x55aed5b362e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.207+0000 7f043dc00740 1 -- 172.21.15.31:0/2832551203 >> 172.21.15.31:0/2832551203 conn(0x55aed5b32410 msgr2=0x55aed5b327f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:04:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.207+0000 7f043dc00740 1 -- 172.21.15.31:0/2832551203 shutdown_connections 2024-04-26T18:04:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.207+0000 7f043dc00740 1 -- 172.21.15.31:0/2832551203 wait complete. 2024-04-26T18:04:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.208+0000 7f043dc00740 1 -- start start 2024-04-26T18:04:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.208+0000 7f043dc00740 1 Processor -- start 2024-04-26T18:04:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.208+0000 7f043dc00740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b35f00 0x55aed5b3e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.208+0000 7f043dc00740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b36820 0x55aed5b3e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.208+0000 7f043dc00740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b410e0 0x55aed5b3eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.208+0000 7f043dc00740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55aed5b43d10 con 0x55aed5b36820 2024-04-26T18:04:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.208+0000 7f043dc00740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x55aed5a23b60 con 0x55aed5b35f00 2024-04-26T18:04:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.208+0000 7f043dc00740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55aed5b43e90 con 0x55aed5b410e0 2024-04-26T18:04:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.208+0000 7f043dc00740 10 client.0 _ll_register_callbacks cb 0x55aed5a482b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:04:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.208+0000 7f0433fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b36820 0x55aed5b3e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.208+0000 7f0438c86640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b35f00 0x55aed5b3e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.208+0000 7f0433fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b36820 0x55aed5b3e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60086/0 (socket says 172.21.15.31:60086) 2024-04-26T18:04:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.208+0000 7f0433fff640 1 -- 172.21.15.31:0/1841724249 learned_addr learned my addr 172.21.15.31:0/1841724249 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:04:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f0439487640 1 --2- 172.21.15.31:0/1841724249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b410e0 0x55aed5b3eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f0433fff640 1 -- 172.21.15.31:0/1841724249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b410e0 msgr2=0x55aed5b3eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:04:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f0433fff640 1 --2- 172.21.15.31:0/1841724249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b410e0 0x55aed5b3eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f0433fff640 1 -- 172.21.15.31:0/1841724249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b35f00 msgr2=0x55aed5b3e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:04:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f0433fff640 1 --2- 172.21.15.31:0/1841724249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b35f00 0x55aed5b3e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f0433fff640 1 -- 172.21.15.31:0/1841724249 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aed57ae210 con 0x55aed5b36820 2024-04-26T18:04:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f0438c86640 1 --2- 172.21.15.31:0/1841724249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b35f00 0x55aed5b3e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:04:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f0433fff640 1 --2- 172.21.15.31:0/1841724249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b36820 0x55aed5b3e5a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f0420000c00 tx=0x7f04200bc550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bcad095515f3710 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f0413fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:04:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f0413fff640 1 -- 172.21.15.31:0/1841724249 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04200d6060 con 0x55aed5b36820 2024-04-26T18:04:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f0413fff640 1 -- 172.21.15.31:0/1841724249 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f04200ee560 con 0x55aed5b36820 2024-04-26T18:04:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f043dc00740 1 -- 172.21.15.31:0/1841724249 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0428009e20 con 0x55aed5b36820 2024-04-26T18:04:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f0413fff640 1 -- 172.21.15.31:0/1841724249 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f04200bde60 con 0x55aed5b36820 2024-04-26T18:04:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f043dc00740 10 client.16104 fetch_fsmap learned FSMap version 35 2024-04-26T18:04:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.209+0000 7f043dc00740 1 -- 172.21.15.31:0/1841724249 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55aed587a620 con 0x55aed5b36820 2024-04-26T18:04:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.210+0000 7f0413fff640 1 -- 172.21.15.31:0/1841724249 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0428009e20 con 0x55aed5b36820 2024-04-26T18:04:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.210+0000 7f0413fff640 1 -- 172.21.15.31:0/1841724249 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7f04200c5310 con 0x55aed5b36820 2024-04-26T18:04:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.210+0000 7f043dc00740 10 client.16104 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T18:04:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.210+0000 7f043dc00740 10 client.16104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:04:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.210+0000 7f043dc00740 20 client.16104 populate_metadata read hostname 'smithi031' 2024-04-26T18:04:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.210+0000 7f043dc00740 1 --2- 172.21.15.31:0/1841724249 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55aed5b511a0 0x55aed5b53630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.210+0000 7f043dc00740 4 client.16104 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:04:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.210+0000 7f043dc00740 1 -- 172.21.15.31:0/1841724249 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aed5aaa070 con 0x55aed5b511a0 2024-04-26T18:04:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.210+0000 7f0438c86640 1 --2- 172.21.15.31:0/1841724249 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55aed5b511a0 0x55aed5b53630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.211+0000 7f0438c86640 1 --2- 172.21.15.31:0/1841724249 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55aed5b511a0 0x55aed5b53630 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9764e80165e63bc5 server_cookie=746a8de82d6c7cb8 in_seq=0 out_seq=0 2024-04-26T18:04:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.211+0000 7f0413fff640 10 client.16104 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:04:01.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.211+0000 7f0413fff640 1 -- 172.21.15.31:0/1841724249 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aed5aaa070 con 0x55aed5b511a0 2024-04-26T18:04:01.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.211+0000 7f0413fff640 10 client.16104 handle_command_reply: tid=0 2024-04-26T18:04:01.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.211+0000 7f043dc00740 1 client.16104 shutdown 2024-04-26T18:04:01.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.211+0000 7f043dc00740 1 -- 172.21.15.31:0/1841724249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b36820 msgr2=0x55aed5b3e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:01.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.212+0000 7f043dc00740 1 --2- 172.21.15.31:0/1841724249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b36820 0x55aed5b3e5a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f0420000c00 tx=0x7f04200bc550 comp rx=0 tx=0).stop 2024-04-26T18:04:01.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.212+0000 7f043dc00740 1 -- 172.21.15.31:0/1841724249 shutdown_connections 2024-04-26T18:04:01.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.212+0000 7f043dc00740 1 --2- 172.21.15.31:0/1841724249 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55aed5b511a0 0x55aed5b53630 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.212+0000 7f043dc00740 1 --2- 172.21.15.31:0/1841724249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b410e0 0x55aed5b3eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.212+0000 7f0413fff640 0 client.16104 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:04:01.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.212+0000 7f043dc00740 1 --2- 172.21.15.31:0/1841724249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b36820 0x55aed5b3e5a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.212+0000 7f043dc00740 1 --2- 172.21.15.31:0/1841724249 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b35f00 0x55aed5b3e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.212+0000 7f043dc00740 1 -- 172.21.15.31:0/1841724249 >> 172.21.15.31:0/1841724249 conn(0x55aed5b32410 msgr2=0x55aed5b38140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:04:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.212+0000 7f043dc00740 1 -- 172.21.15.31:0/1841724249 shutdown_connections 2024-04-26T18:04:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.212+0000 7f043dc00740 1 -- 172.21.15.31:0/1841724249 wait complete. 2024-04-26T18:04:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.212+0000 7f043dc00740 20 client.16104 trim_cache size 0 max 16384 2024-04-26T18:04:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.231+0000 7f043dc00740 1 Processor -- start 2024-04-26T18:04:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.231+0000 7f043dc00740 1 -- start start 2024-04-26T18:04:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.231+0000 7f043dc00740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b35f00 0x55aed5b4cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.231+0000 7f043dc00740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b36820 0x55aed5b58f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.231+0000 7f043dc00740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b410e0 0x55aed5b59480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.231+0000 7f043dc00740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f04341b3020 con 0x55aed5b35f00 2024-04-26T18:04:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.231+0000 7f043dc00740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f04341b0890 con 0x55aed5b36820 2024-04-26T18:04:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.231+0000 7f043dc00740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04341ae700 con 0x55aed5b410e0 2024-04-26T18:04:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.231+0000 7f0438c86640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b35f00 0x55aed5b4cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.231+0000 7f0438c86640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b35f00 0x55aed5b4cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60102/0 (socket says 172.21.15.31:60102) 2024-04-26T18:04:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.231+0000 7f0438c86640 1 -- 172.21.15.31:0/629716597 learned_addr learned my addr 172.21.15.31:0/629716597 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:04:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.232+0000 7f0439487640 1 --2- 172.21.15.31:0/629716597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b410e0 0x55aed5b59480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.232+0000 7f0433fff640 1 --2- 172.21.15.31:0/629716597 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b36820 0x55aed5b58f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.232+0000 7f0438c86640 1 -- 172.21.15.31:0/629716597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b410e0 msgr2=0x55aed5b59480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:04:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.232+0000 7f0438c86640 1 --2- 172.21.15.31:0/629716597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b410e0 0x55aed5b59480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.232+0000 7f0438c86640 1 -- 172.21.15.31:0/629716597 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b36820 msgr2=0x55aed5b58f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:04:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.232+0000 7f0438c86640 1 --2- 172.21.15.31:0/629716597 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b36820 0x55aed5b58f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.232+0000 7f0438c86640 1 -- 172.21.15.31:0/629716597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aed5b36650 con 0x55aed5b35f00 2024-04-26T18:04:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.232+0000 7f0438c86640 1 --2- 172.21.15.31:0/629716597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b35f00 0x55aed5b4cbd0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f0428009bc0 tx=0x7f04280097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a386142de117d293 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.232+0000 7f041e7fc640 1 -- 172.21.15.31:0/629716597 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042801e040 con 0x55aed5b35f00 2024-04-26T18:04:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.232+0000 7f041e7fc640 1 -- 172.21.15.31:0/629716597 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0428004680 con 0x55aed5b35f00 2024-04-26T18:04:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.232+0000 7f041e7fc640 1 -- 172.21.15.31:0/629716597 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042802b330 con 0x55aed5b35f00 2024-04-26T18:04:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.233+0000 7f043dc00740 1 -- 172.21.15.31:0/629716597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b35f00 msgr2=0x55aed5b4cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.233+0000 7f043dc00740 1 --2- 172.21.15.31:0/629716597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b35f00 0x55aed5b4cbd0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f0428009bc0 tx=0x7f04280097a0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.233+0000 7f043dc00740 1 -- 172.21.15.31:0/629716597 shutdown_connections 2024-04-26T18:04:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.233+0000 7f043dc00740 1 --2- 172.21.15.31:0/629716597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b410e0 0x55aed5b59480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.233+0000 7f043dc00740 1 --2- 172.21.15.31:0/629716597 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b36820 0x55aed5b58f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.233+0000 7f043dc00740 1 --2- 172.21.15.31:0/629716597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b35f00 0x55aed5b4cbd0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.233+0000 7f043dc00740 1 -- 172.21.15.31:0/629716597 >> 172.21.15.31:0/629716597 conn(0x55aed5b32410 msgr2=0x55aed5b327f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:04:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.233+0000 7f043dc00740 1 -- 172.21.15.31:0/629716597 shutdown_connections 2024-04-26T18:04:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.233+0000 7f043dc00740 1 -- 172.21.15.31:0/629716597 wait complete. 2024-04-26T18:04:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.233+0000 7f043dc00740 1 -- start start 2024-04-26T18:04:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.233+0000 7f043dc00740 1 Processor -- start 2024-04-26T18:04:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f043dc00740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b35f00 0x55aed5b42020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f043dc00740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b36820 0x55aed5b42560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f043dc00740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b410e0 0x55aed5b42aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f043dc00740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f04341aed50 con 0x55aed5b35f00 2024-04-26T18:04:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f043dc00740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f04341b01c0 con 0x55aed5b410e0 2024-04-26T18:04:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f043dc00740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04341af420 con 0x55aed5b36820 2024-04-26T18:04:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f043dc00740 10 client.0 _ll_register_callbacks cb 0x7f0420009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:04:01.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f0438c86640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b35f00 0x55aed5b42020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f0438c86640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b35f00 0x55aed5b42020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60104/0 (socket says 172.21.15.31:60104) 2024-04-26T18:04:01.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f0438c86640 1 -- 172.21.15.31:0/740614943 learned_addr learned my addr 172.21.15.31:0/740614943 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:04:01.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f0439487640 1 --2- 172.21.15.31:0/740614943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b410e0 0x55aed5b42aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f0433fff640 1 --2- 172.21.15.31:0/740614943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b36820 0x55aed5b42560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f0438c86640 1 -- 172.21.15.31:0/740614943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b36820 msgr2=0x55aed5b42560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:04:01.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f0438c86640 1 --2- 172.21.15.31:0/740614943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b36820 0x55aed5b42560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f0438c86640 1 -- 172.21.15.31:0/740614943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b410e0 msgr2=0x55aed5b42aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:04:01.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f0438c86640 1 --2- 172.21.15.31:0/740614943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b410e0 0x55aed5b42aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f0438c86640 1 -- 172.21.15.31:0/740614943 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aed5b473e0 con 0x55aed5b35f00 2024-04-26T18:04:01.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f0439487640 1 --2- 172.21.15.31:0/740614943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b410e0 0x55aed5b42aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:04:01.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f0438c86640 1 --2- 172.21.15.31:0/740614943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b35f00 0x55aed5b42020 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0428012fd0 tx=0x7f0428011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d607f6cc719b68a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:01.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.234+0000 7f0413fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:04:01.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f0413fff640 1 -- 172.21.15.31:0/740614943 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042803a060 con 0x55aed5b35f00 2024-04-26T18:04:01.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f0413fff640 1 -- 172.21.15.31:0/740614943 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f04280052f0 con 0x55aed5b35f00 2024-04-26T18:04:01.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f043dc00740 1 -- 172.21.15.31:0/740614943 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0428004680 con 0x55aed5b35f00 2024-04-26T18:04:01.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f0413fff640 1 -- 172.21.15.31:0/740614943 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f042801e040 con 0x55aed5b35f00 2024-04-26T18:04:01.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f043dc00740 10 client.16110 fetch_fsmap learned FSMap version 35 2024-04-26T18:04:01.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f043dc00740 1 -- 172.21.15.31:0/740614943 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55aed5bb7b10 con 0x55aed5b35f00 2024-04-26T18:04:01.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f0413fff640 1 -- 172.21.15.31:0/740614943 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0428004680 con 0x55aed5b35f00 2024-04-26T18:04:01.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f0413fff640 1 -- 172.21.15.31:0/740614943 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7f042802b3d0 con 0x55aed5b35f00 2024-04-26T18:04:01.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f043dc00740 10 client.16110 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T18:04:01.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f043dc00740 10 client.16110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:04:01.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f043dc00740 20 client.16110 populate_metadata read hostname 'smithi031' 2024-04-26T18:04:01.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f043dc00740 1 --2- 172.21.15.31:0/740614943 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55aed5bb8030 0x55aed5bba420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:01.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f043dc00740 4 client.16110 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T18:04:01.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f043dc00740 1 -- 172.21.15.31:0/740614943 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aed5aa3ee0 con 0x55aed5bb8030 2024-04-26T18:04:01.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.235+0000 7f0433fff640 1 --2- 172.21.15.31:0/740614943 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55aed5bb8030 0x55aed5bba420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:01.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.236+0000 7f0433fff640 1 --2- 172.21.15.31:0/740614943 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55aed5bb8030 0x55aed5bba420 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d3061f3321e57ab server_cookie=404451c009962a05 in_seq=0 out_seq=0 2024-04-26T18:04:01.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.236+0000 7f0413fff640 10 client.16110 ms_handle_connect on v2:172.21.15.97:6838/3765855684 2024-04-26T18:04:01.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.251+0000 7f0413fff640 1 -- 172.21.15.31:0/740614943 <== mds.0 v2:172.21.15.97:6838/3765855684 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aed5aa3ee0 con 0x55aed5bb8030 2024-04-26T18:04:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.251+0000 7f0413fff640 10 client.16110 handle_command_reply: tid=0 2024-04-26T18:04:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.251+0000 7f043dc00740 1 client.16110 shutdown 2024-04-26T18:04:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.251+0000 7f043dc00740 1 -- 172.21.15.31:0/740614943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b35f00 msgr2=0x55aed5b42020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.251+0000 7f043dc00740 1 --2- 172.21.15.31:0/740614943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b35f00 0x55aed5b42020 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0428012fd0 tx=0x7f0428011e20 comp rx=0 tx=0).stop 2024-04-26T18:04:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.251+0000 7f043dc00740 1 -- 172.21.15.31:0/740614943 shutdown_connections 2024-04-26T18:04:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.251+0000 7f043dc00740 1 --2- 172.21.15.31:0/740614943 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x55aed5bb8030 0x55aed5bba420 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.251+0000 7f0413fff640 0 client.16110 ms_handle_reset on v2:172.21.15.97:6838/3765855684 2024-04-26T18:04:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.251+0000 7f043dc00740 1 --2- 172.21.15.31:0/740614943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aed5b410e0 0x55aed5b42aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.251+0000 7f043dc00740 1 --2- 172.21.15.31:0/740614943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aed5b36820 0x55aed5b42560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.251+0000 7f043dc00740 1 --2- 172.21.15.31:0/740614943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aed5b35f00 0x55aed5b42020 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.252+0000 7f043dc00740 1 -- 172.21.15.31:0/740614943 >> 172.21.15.31:0/740614943 conn(0x55aed5b32410 msgr2=0x55aed5b3e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:04:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.252+0000 7f043dc00740 1 -- 172.21.15.31:0/740614943 shutdown_connections 2024-04-26T18:04:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.252+0000 7f043dc00740 1 -- 172.21.15.31:0/740614943 wait complete. 2024-04-26T18:04:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.252+0000 7f043dc00740 20 client.16110 trim_cache size 0 max 16384 2024-04-26T18:04:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-26T18:04:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-26T18:04:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-26T18:04:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-26T18:04:01.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.252+0000 7f043aeea640 1 -- 172.21.15.31:0/3670972540 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f04240a3e40 msgr2=0x7f04240a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:01.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.252+0000 7f043aeea640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f04240a3e40 0x7f04240a6300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f042c005c40 tx=0x7f042c009e50 comp rx=0 tx=0).stop 2024-04-26T18:04:01.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.252+0000 7f043aeea640 1 -- 172.21.15.31:0/3670972540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04341017f0 msgr2=0x7f0434102550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:01.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.252+0000 7f043aeea640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04341017f0 0x7f0434102550 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f0420005940 tx=0x7f04200040c0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.253+0000 7f043aeea640 1 -- 172.21.15.31:0/3670972540 shutdown_connections 2024-04-26T18:04:01.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.253+0000 7f043aeea640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0434107910 0x7f0434102a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.253+0000 7f043aeea640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f04240a3e40 0x7f04240a6300 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.253+0000 7f043aeea640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04341017f0 0x7f0434102550 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.253+0000 7f043aeea640 1 --2- 172.21.15.31:0/3670972540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0434100e30 0x7f0434102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:01.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.253+0000 7f043aeea640 1 -- 172.21.15.31:0/3670972540 >> 172.21.15.31:0/3670972540 conn(0x7f0434077dc0 msgr2=0x7f04340ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:04:01.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.253+0000 7f043aeea640 1 -- 172.21.15.31:0/3670972540 shutdown_connections 2024-04-26T18:04:01.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:01.253+0000 7f043aeea640 1 -- 172.21.15.31:0/3670972540 wait complete. 2024-04-26T18:04:01.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:01 smithi097 ceph-mon[32419]: pgmap v230: 97 pgs: 97 active+clean; 3.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 5.9 MiB/s wr, 579 op/s 2024-04-26T18:04:01.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:01 smithi186 ceph-mon[32797]: pgmap v230: 97 pgs: 97 active+clean; 3.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 5.9 MiB/s wr, 579 op/s 2024-04-26T18:04:01.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:01 smithi031 ceph-mon[29164]: pgmap v230: 97 pgs: 97 active+clean; 3.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 5.9 MiB/s wr, 579 op/s 2024-04-26T18:04:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 400.0 s, 1565.4 tps, lat 62.333 ms stddev 124.760 2024-04-26T18:04:03.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:03 smithi097 ceph-mon[32419]: pgmap v231: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 8.7 MiB/s wr, 876 op/s 2024-04-26T18:04:03.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:03 smithi186 ceph-mon[32797]: pgmap v231: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 8.7 MiB/s wr, 876 op/s 2024-04-26T18:04:03.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:03 smithi031 ceph-mon[29164]: pgmap v231: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 8.7 MiB/s wr, 876 op/s 2024-04-26T18:04:05.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-mon[32419]: pgmap v232: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.6 MiB/s wr, 579 op/s 2024-04-26T18:04:05.384 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f83a36ed700 time 2024-04-26T18:04:05.173318+0000 2024-04-26T18:04:05.384 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T18:04:05.384 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:05.384 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f83add163dd] 2024-04-26T18:04:05.384 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f83add165a3] 2024-04-26T18:04:05.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x558751b24822] 2024-04-26T18:04:05.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558751a94598] 2024-04-26T18:04:05.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558751a96c17] 2024-04-26T18:04:05.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55875198fd0b] 2024-04-26T18:04:05.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5587519902ac] 2024-04-26T18:04:05.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558751978fad] 2024-04-26T18:04:05.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f83adfaae20] 2024-04-26T18:04:05.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 10: (DispatchQueue::entry()+0x51f) [0x7f83adfa7fdf] 2024-04-26T18:04:05.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83ae07afc1] 2024-04-26T18:04:05.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f83aca941ca] 2024-04-26T18:04:05.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 13: clone() 2024-04-26T18:04:05.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: *** Caught signal (Aborted) ** 2024-04-26T18:04:05.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: in thread 7f83a36ed700 thread_name:ms_dispatch 2024-04-26T18:04:05.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T18:04:05.174+0000 7f83a36ed700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f83a36ed700 time 2024-04-26T18:04:05.173318+0000 2024-04-26T18:04:05.386 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T18:04:05.386 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T18:04:05.387 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:05.387 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f83add163dd] 2024-04-26T18:04:05.387 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f83add165a3] 2024-04-26T18:04:05.387 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x558751b24822] 2024-04-26T18:04:05.387 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558751a94598] 2024-04-26T18:04:05.387 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558751a96c17] 2024-04-26T18:04:05.387 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55875198fd0b] 2024-04-26T18:04:05.387 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5587519902ac] 2024-04-26T18:04:05.387 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558751978fad] 2024-04-26T18:04:05.387 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f83adfaae20] 2024-04-26T18:04:05.387 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 10: (DispatchQueue::entry()+0x51f) [0x7f83adfa7fdf] 2024-04-26T18:04:05.387 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83ae07afc1] 2024-04-26T18:04:05.387 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f83aca941ca] 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 13: clone() 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f83aca9ed20] 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2: gsignal() 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 3: abort() 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f83add16437] 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f83add165a3] 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x558751b24822] 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558751a94598] 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558751a96c17] 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55875198fd0b] 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5587519902ac] 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558751978fad] 2024-04-26T18:04:05.388 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f83adfaae20] 2024-04-26T18:04:05.389 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 13: (DispatchQueue::entry()+0x51f) [0x7f83adfa7fdf] 2024-04-26T18:04:05.389 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83ae07afc1] 2024-04-26T18:04:05.389 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f83aca941ca] 2024-04-26T18:04:05.389 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 16: clone() 2024-04-26T18:04:05.389 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T18:04:05.175+0000 7f83a36ed700 -1 *** Caught signal (Aborted) ** 2024-04-26T18:04:05.389 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: in thread 7f83a36ed700 thread_name:ms_dispatch 2024-04-26T18:04:05.389 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T18:04:05.389 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:05.389 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f83aca9ed20] 2024-04-26T18:04:05.389 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2: gsignal() 2024-04-26T18:04:05.389 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 3: abort() 2024-04-26T18:04:05.389 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f83add16437] 2024-04-26T18:04:05.389 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f83add165a3] 2024-04-26T18:04:05.389 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x558751b24822] 2024-04-26T18:04:05.390 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558751a94598] 2024-04-26T18:04:05.390 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558751a96c17] 2024-04-26T18:04:05.390 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55875198fd0b] 2024-04-26T18:04:05.390 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5587519902ac] 2024-04-26T18:04:05.390 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558751978fad] 2024-04-26T18:04:05.390 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f83adfaae20] 2024-04-26T18:04:05.390 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 13: (DispatchQueue::entry()+0x51f) [0x7f83adfa7fdf] 2024-04-26T18:04:05.390 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83ae07afc1] 2024-04-26T18:04:05.390 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f83aca941ca] 2024-04-26T18:04:05.390 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 16: clone() 2024-04-26T18:04:05.390 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T18:04:05.390 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T18:04:05.390 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: -1> 2024-04-26T18:04:05.174+0000 7f83a36ed700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f83a36ed700 time 2024-04-26T18:04:05.173318+0000 2024-04-26T18:04:05.390 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T18:04:05.391 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T18:04:05.391 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:05.391 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f83add163dd] 2024-04-26T18:04:05.391 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f83add165a3] 2024-04-26T18:04:05.391 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x558751b24822] 2024-04-26T18:04:05.391 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558751a94598] 2024-04-26T18:04:05.391 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558751a96c17] 2024-04-26T18:04:05.391 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55875198fd0b] 2024-04-26T18:04:05.391 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5587519902ac] 2024-04-26T18:04:05.391 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558751978fad] 2024-04-26T18:04:05.391 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f83adfaae20] 2024-04-26T18:04:05.391 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 10: (DispatchQueue::entry()+0x51f) [0x7f83adfa7fdf] 2024-04-26T18:04:05.391 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83ae07afc1] 2024-04-26T18:04:05.391 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f83aca941ca] 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 13: clone() 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 0> 2024-04-26T18:04:05.175+0000 7f83a36ed700 -1 *** Caught signal (Aborted) ** 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: in thread 7f83a36ed700 thread_name:ms_dispatch 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f83aca9ed20] 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2: gsignal() 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 3: abort() 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f83add16437] 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f83add165a3] 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x558751b24822] 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558751a94598] 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558751a96c17] 2024-04-26T18:04:05.392 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55875198fd0b] 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5587519902ac] 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558751978fad] 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f83adfaae20] 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 13: (DispatchQueue::entry()+0x51f) [0x7f83adfa7fdf] 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83ae07afc1] 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f83aca941ca] 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 16: clone() 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: -1> 2024-04-26T18:04:05.174+0000 7f83a36ed700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f83a36ed700 time 2024-04-26T18:04:05.173318+0000 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f83add163dd] 2024-04-26T18:04:05.393 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f83add165a3] 2024-04-26T18:04:05.394 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x558751b24822] 2024-04-26T18:04:05.394 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558751a94598] 2024-04-26T18:04:05.394 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558751a96c17] 2024-04-26T18:04:05.394 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55875198fd0b] 2024-04-26T18:04:05.394 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5587519902ac] 2024-04-26T18:04:05.394 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558751978fad] 2024-04-26T18:04:05.394 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f83adfaae20] 2024-04-26T18:04:05.395 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 10: (DispatchQueue::entry()+0x51f) [0x7f83adfa7fdf] 2024-04-26T18:04:05.395 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83ae07afc1] 2024-04-26T18:04:05.395 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f83aca941ca] 2024-04-26T18:04:05.395 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 13: clone() 2024-04-26T18:04:05.395 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T18:04:05.395 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 0> 2024-04-26T18:04:05.175+0000 7f83a36ed700 -1 *** Caught signal (Aborted) ** 2024-04-26T18:04:05.395 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: in thread 7f83a36ed700 thread_name:ms_dispatch 2024-04-26T18:04:05.395 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T18:04:05.395 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:05.395 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f83aca9ed20] 2024-04-26T18:04:05.395 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2: gsignal() 2024-04-26T18:04:05.395 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 3: abort() 2024-04-26T18:04:05.395 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f83add16437] 2024-04-26T18:04:05.395 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f83add165a3] 2024-04-26T18:04:05.396 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x558751b24822] 2024-04-26T18:04:05.396 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x558751a94598] 2024-04-26T18:04:05.396 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558751a96c17] 2024-04-26T18:04:05.396 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55875198fd0b] 2024-04-26T18:04:05.396 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5587519902ac] 2024-04-26T18:04:05.396 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x558751978fad] 2024-04-26T18:04:05.396 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7f83adfaae20] 2024-04-26T18:04:05.396 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 13: (DispatchQueue::entry()+0x51f) [0x7f83adfa7fdf] 2024-04-26T18:04:05.396 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f83ae07afc1] 2024-04-26T18:04:05.396 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f83aca941ca] 2024-04-26T18:04:05.396 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 16: clone() 2024-04-26T18:04:05.396 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T18:04:05.396 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[63069]: 2024-04-26T18:04:05.396 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:05 smithi097 podman[67736]: 2024-04-26 18:04:05.250693857 +0000 UTC m=+0.021176710 container died 6ab9bc8bb9b0b41aea137e49f67971e38325d4627e2ac6b38d3b0fe138fff597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T18:04:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:05 smithi186 ceph-mon[32797]: pgmap v232: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.6 MiB/s wr, 579 op/s 2024-04-26T18:04:05.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:05 smithi031 ceph-mon[29164]: pgmap v232: 97 pgs: 97 active+clean; 3.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.6 MiB/s wr, 579 op/s 2024-04-26T18:04:06.566 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:06 smithi097 podman[67736]: 2024-04-26 18:04:06.27533163 +0000 UTC m=+1.045814474 container remove 6ab9bc8bb9b0b41aea137e49f67971e38325d4627e2ac6b38d3b0fe138fff597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-26T18:04:06.566 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:06 smithi097 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-26T18:04:06.884 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:06 smithi097 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-26T18:04:06.884 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:06 smithi097 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.b.service: Consumed 10.727s CPU time. 2024-04-26T18:04:07.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:07 smithi097 ceph-mon[32419]: pgmap v233: 97 pgs: 97 active+clean; 4.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 17 MiB/s wr, 1.33k op/s 2024-04-26T18:04:07.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:07 smithi186 ceph-mon[32797]: pgmap v233: 97 pgs: 97 active+clean; 4.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 17 MiB/s wr, 1.33k op/s 2024-04-26T18:04:07.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:07 smithi031 ceph-mon[29164]: pgmap v233: 97 pgs: 97 active+clean; 4.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 17 MiB/s wr, 1.33k op/s 2024-04-26T18:04:08.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 405.0 s, 2485.4 tps, lat 41.057 ms stddev 45.675 2024-04-26T18:04:08.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:08 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:04:08.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:08 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:04:08.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:08 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:04:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:09 smithi097 ceph-mon[32419]: pgmap v234: 97 pgs: 97 active+clean; 4.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 26 MiB/s wr, 1.93k op/s 2024-04-26T18:04:09.460 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:09 smithi031 ceph-mon[29164]: pgmap v234: 97 pgs: 97 active+clean; 4.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 26 MiB/s wr, 1.93k op/s 2024-04-26T18:04:09.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:09 smithi186 ceph-mon[32797]: pgmap v234: 97 pgs: 97 active+clean; 4.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 26 MiB/s wr, 1.93k op/s 2024-04-26T18:04:11.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:11 smithi097 ceph-mon[32419]: pgmap v235: 97 pgs: 97 active+clean; 4.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 24 MiB/s wr, 1.73k op/s 2024-04-26T18:04:11.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:11 smithi186 ceph-mon[32797]: pgmap v235: 97 pgs: 97 active+clean; 4.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 24 MiB/s wr, 1.73k op/s 2024-04-26T18:04:11.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:11 smithi031 ceph-mon[29164]: pgmap v235: 97 pgs: 97 active+clean; 4.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 24 MiB/s wr, 1.73k op/s 2024-04-26T18:04:13.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 410.0 s, 2734.6 tps, lat 36.608 ms stddev 42.013 2024-04-26T18:04:13.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:13 smithi097 ceph-mon[32419]: pgmap v236: 97 pgs: 97 active+clean; 4.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 27 MiB/s wr, 2.01k op/s 2024-04-26T18:04:13.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:13 smithi186 ceph-mon[32797]: pgmap v236: 97 pgs: 97 active+clean; 4.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 27 MiB/s wr, 2.01k op/s 2024-04-26T18:04:13.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:13 smithi031 ceph-mon[29164]: pgmap v236: 97 pgs: 97 active+clean; 4.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 27 MiB/s wr, 2.01k op/s 2024-04-26T18:04:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.376+0000 7f75e38f1640 1 -- 172.21.15.31:0/2740929014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc11c110 msgr2=0x7f75dc11e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.376+0000 7f75e38f1640 1 --2- 172.21.15.31:0/2740929014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc11c110 0x7f75dc11e500 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f75d8009930 tx=0x7f75d802a4d0 comp rx=0 tx=0).stop 2024-04-26T18:04:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.376+0000 7f75e38f1640 1 -- 172.21.15.31:0/2740929014 shutdown_connections 2024-04-26T18:04:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.376+0000 7f75e38f1640 1 --2- 172.21.15.31:0/2740929014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc11c110 0x7f75dc11e500 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.377+0000 7f75e38f1640 1 --2- 172.21.15.31:0/2740929014 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75dc111500 0x7f75dc109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.377+0000 7f75e38f1640 1 --2- 172.21.15.31:0/2740929014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75dc111120 0x7f75dc109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.377+0000 7f75e38f1640 1 -- 172.21.15.31:0/2740929014 >> 172.21.15.31:0/2740929014 conn(0x7f75dc06da00 msgr2=0x7f75dc06de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:04:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.377+0000 7f75e38f1640 1 -- 172.21.15.31:0/2740929014 shutdown_connections 2024-04-26T18:04:14.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.377+0000 7f75e38f1640 1 -- 172.21.15.31:0/2740929014 wait complete. 2024-04-26T18:04:14.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.378+0000 7f75e38f1640 1 Processor -- start 2024-04-26T18:04:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.378+0000 7f75e38f1640 1 -- start start 2024-04-26T18:04:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.378+0000 7f75e38f1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75dc111120 0x7f75dc1a7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.378+0000 7f75e38f1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc11c110 0x7f75dc1a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.378+0000 7f75e38f1640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75dc1ac7c0 0x7f75dc1acc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.378+0000 7f75e38f1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc1219e0 con 0x7f75dc11c110 2024-04-26T18:04:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.378+0000 7f75e38f1640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc121860 con 0x7f75dc1ac7c0 2024-04-26T18:04:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.378+0000 7f75e38f1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc121b60 con 0x7f75dc111120 2024-04-26T18:04:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.378+0000 7f75e0e8c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc11c110 0x7f75dc1a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.378+0000 7f75e0e8c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc11c110 0x7f75dc1a8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46348/0 (socket says 172.21.15.31:46348) 2024-04-26T18:04:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.378+0000 7f75e0e8c640 1 -- 172.21.15.31:0/865661195 learned_addr learned my addr 172.21.15.31:0/865661195 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:04:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.379+0000 7f75e1e8e640 1 --2- 172.21.15.31:0/865661195 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75dc1ac7c0 0x7f75dc1acc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:14.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.379+0000 7f75e0e8c640 1 -- 172.21.15.31:0/865661195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75dc111120 msgr2=0x7f75dc1a7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.379+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/865661195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75dc111120 0x7f75dc1a7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.379+0000 7f75e0e8c640 1 -- 172.21.15.31:0/865661195 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75dc1ac7c0 msgr2=0x7f75dc1acc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.379+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/865661195 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f75dc1ac7c0 0x7f75dc1acc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.379+0000 7f75e0e8c640 1 -- 172.21.15.31:0/865661195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75dc1ad2b0 con 0x7f75dc11c110 2024-04-26T18:04:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.379+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/865661195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75dc11c110 0x7f75dc1a8440 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f75d400eb90 tx=0x7f75d400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.379+0000 7f75d27fc640 1 -- 172.21.15.31:0/865661195 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d4010630 con 0x7f75dc11c110 2024-04-26T18:04:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.379+0000 7f75d27fc640 1 -- 172.21.15.31:0/865661195 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75d400ee90 con 0x7f75dc11c110 2024-04-26T18:04:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.379+0000 7f75e38f1640 1 -- 172.21.15.31:0/865661195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75dc1ad5a0 con 0x7f75dc11c110 2024-04-26T18:04:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.380+0000 7f75d27fc640 1 -- 172.21.15.31:0/865661195 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d4018ac0 con 0x7f75dc11c110 2024-04-26T18:04:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.380+0000 7f75e38f1640 1 -- 172.21.15.31:0/865661195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75dc1b4de0 con 0x7f75dc11c110 2024-04-26T18:04:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.384+0000 7f75d27fc640 1 -- 172.21.15.31:0/865661195 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f75d4018c60 con 0x7f75dc11c110 2024-04-26T18:04:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.385+0000 7f75e6607740 1 Processor -- start 2024-04-26T18:04:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.385+0000 7f75e6607740 1 -- start start 2024-04-26T18:04:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.385+0000 7f75d27fc640 1 --2- 172.21.15.31:0/865661195 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75c80a3e40 0x7f75c80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.385+0000 7f75e6607740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556108f4ff00 0x556108f502e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.385+0000 7f75e6607740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556108f50820 0x556108f5aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.385+0000 7f75e6607740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556108f5b0e0 0x556108f5d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.385+0000 7f75e6607740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556108eb6460 con 0x556108f50820 2024-04-26T18:04:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.385+0000 7f75e6607740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x556108e70590 con 0x556108f4ff00 2024-04-26T18:04:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.385+0000 7f75e6607740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556108eb2420 con 0x556108f5b0e0 2024-04-26T18:04:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75e0e8c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556108f50820 0x556108f5aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75e168d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556108f4ff00 0x556108f502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75e0e8c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556108f50820 0x556108f5aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46352/0 (socket says 172.21.15.31:46352) 2024-04-26T18:04:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75e0e8c640 1 -- 172.21.15.31:0/2035774486 learned_addr learned my addr 172.21.15.31:0/2035774486 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:04:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75d27fc640 1 -- 172.21.15.31:0/865661195 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f75d40d0e20 con 0x7f75dc11c110 2024-04-26T18:04:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75e1e8e640 1 --2- 172.21.15.31:0/2035774486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556108f5b0e0 0x556108f5d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75e1e8e640 1 --2- 172.21.15.31:0/865661195 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75c80a3e40 0x7f75c80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75e0e8c640 1 -- 172.21.15.31:0/2035774486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556108f5b0e0 msgr2=0x556108f5d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:04:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/2035774486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556108f5b0e0 0x556108f5d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75e0e8c640 1 -- 172.21.15.31:0/2035774486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556108f4ff00 msgr2=0x556108f502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:04:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/2035774486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556108f4ff00 0x556108f502e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75e0e8c640 1 -- 172.21.15.31:0/2035774486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556108e7d0d0 con 0x556108f50820 2024-04-26T18:04:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75e168d640 1 --2- 172.21.15.31:0/2035774486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556108f4ff00 0x556108f502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:04:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/2035774486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556108f50820 0x556108f5aba0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f75d40a54c0 tx=0x7f75d4009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28b518566ffa56c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.386+0000 7f75e1e8e640 1 --2- 172.21.15.31:0/865661195 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f75c80a3e40 0x7f75c80a6300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f75d80096c0 tx=0x7f75d8005fd0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.387+0000 7f75c77fe640 1 -- 172.21.15.31:0/2035774486 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d409cd80 con 0x556108f50820 2024-04-26T18:04:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.387+0000 7f75c77fe640 1 -- 172.21.15.31:0/2035774486 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75d40a1130 con 0x556108f50820 2024-04-26T18:04:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.387+0000 7f75c77fe640 1 -- 172.21.15.31:0/2035774486 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d4099580 con 0x556108f50820 2024-04-26T18:04:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.388+0000 7f75e6607740 1 -- 172.21.15.31:0/2035774486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556108f50820 msgr2=0x556108f5aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.388+0000 7f75e6607740 1 --2- 172.21.15.31:0/2035774486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556108f50820 0x556108f5aba0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f75d40a54c0 tx=0x7f75d4009200 comp rx=0 tx=0).stop 2024-04-26T18:04:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.388+0000 7f75e6607740 1 -- 172.21.15.31:0/2035774486 shutdown_connections 2024-04-26T18:04:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.388+0000 7f75e6607740 1 --2- 172.21.15.31:0/2035774486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556108f5b0e0 0x556108f5d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.388+0000 7f75e6607740 1 --2- 172.21.15.31:0/2035774486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556108f50820 0x556108f5aba0 secure :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f75d40a54c0 tx=0x7f75d4009200 comp rx=0 tx=0).stop 2024-04-26T18:04:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.388+0000 7f75e6607740 1 --2- 172.21.15.31:0/2035774486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556108f4ff00 0x556108f502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.388+0000 7f75e6607740 1 -- 172.21.15.31:0/2035774486 >> 172.21.15.31:0/2035774486 conn(0x556108f4c410 msgr2=0x556108f4c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:04:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.389+0000 7f75e6607740 1 -- 172.21.15.31:0/2035774486 shutdown_connections 2024-04-26T18:04:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.389+0000 7f75e6607740 1 -- 172.21.15.31:0/2035774486 wait complete. 2024-04-26T18:04:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.389+0000 7f75e6607740 1 -- start start 2024-04-26T18:04:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.390+0000 7f75e6607740 1 Processor -- start 2024-04-26T18:04:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.390+0000 7f75e6607740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556108f4ff00 0x556108f58330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.390+0000 7f75e6607740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556108f5b0e0 0x556108f58870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.390+0000 7f75e6607740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556108f58e00 0x556108f670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.390+0000 7f75e168d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556108f4ff00 0x556108f58330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.390+0000 7f75e168d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556108f4ff00 0x556108f58330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:37972/0 (socket says 172.21.15.31:37972) 2024-04-26T18:04:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.390+0000 7f75e168d640 1 -- 172.21.15.31:0/164021214 learned_addr learned my addr 172.21.15.31:0/164021214 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:04:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.390+0000 7f75e6607740 1 -- 172.21.15.31:0/164021214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556108f5dcb0 con 0x556108f58e00 2024-04-26T18:04:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.390+0000 7f75e6607740 1 -- 172.21.15.31:0/164021214 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x556108e3db60 con 0x556108f5b0e0 2024-04-26T18:04:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.390+0000 7f75e6607740 1 -- 172.21.15.31:0/164021214 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556108f5de30 con 0x556108f4ff00 2024-04-26T18:04:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.390+0000 7f75e1e8e640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556108f58e00 0x556108f670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.390+0000 7f75e6607740 10 client.0 _ll_register_callbacks cb 0x556108e622b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:04:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.391+0000 7f75e168d640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556108f5b0e0 msgr2=0x556108f58870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T18:04:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.391+0000 7f75e168d640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x556108f5b0e0 0x556108f58870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.391+0000 7f75e168d640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556108f58e00 msgr2=0x556108f670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:04:14.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.391+0000 7f75e168d640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556108f58e00 0x556108f670a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:14.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.391+0000 7f75e168d640 1 -- 172.21.15.31:0/164021214 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556108bc8210 con 0x556108f4ff00 2024-04-26T18:04:14.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.391+0000 7f75e1e8e640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556108f58e00 0x556108f670a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:04:14.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.391+0000 7f75e168d640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556108f4ff00 0x556108f58330 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0116f0 tx=0x7f75cc011bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83c8613da1b063c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:14.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.391+0000 7f75c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T18:04:14.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.392+0000 7f75c4ff9640 1 -- 172.21.15.31:0/164021214 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc00fdf0 con 0x556108f4ff00 2024-04-26T18:04:14.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.392+0000 7f75c4ff9640 1 -- 172.21.15.31:0/164021214 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f75cc004030 con 0x556108f4ff00 2024-04-26T18:04:14.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.392+0000 7f75e6607740 1 -- 172.21.15.31:0/164021214 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75d40a1130 con 0x556108f4ff00 2024-04-26T18:04:14.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.392+0000 7f75c4ff9640 1 -- 172.21.15.31:0/164021214 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f75cc004270 con 0x556108f4ff00 2024-04-26T18:04:14.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.392+0000 7f75e6607740 10 client.25717 fetch_fsmap learned FSMap version 36 2024-04-26T18:04:14.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.392+0000 7f75e6607740 1 -- 172.21.15.31:0/164021214 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556108c94620 con 0x556108f4ff00 2024-04-26T18:04:14.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.393+0000 7f75c4ff9640 1 -- 172.21.15.31:0/164021214 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75d40a1130 con 0x556108f4ff00 2024-04-26T18:04:14.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.393+0000 7f75c4ff9640 1 -- 172.21.15.31:0/164021214 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f75cc020020 con 0x556108f4ff00 2024-04-26T18:04:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.394+0000 7f75e6607740 10 client.25717 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T18:04:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.394+0000 7f75e6607740 10 client.25717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T18:04:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.394+0000 7f75e6607740 20 client.25717 populate_metadata read hostname 'smithi031' 2024-04-26T18:04:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.394+0000 7f75e6607740 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.394+0000 7f75e6607740 4 client.25717 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:04:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.394+0000 7f75e6607740 1 -- 172.21.15.31:0/164021214 --> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556108ec4070 con 0x556108f6b8d0 2024-04-26T18:04:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.395+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:6838/3765855684 2024-04-26T18:04:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.395+0000 7f75c4ff9640 1 client.25717 ms_handle_refused on v2:172.21.15.97:6838/3765855684 2024-04-26T18:04:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.395+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T18:04:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.596+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:6838/3765855684 2024-04-26T18:04:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.596+0000 7f75c4ff9640 1 client.25717 ms_handle_refused on v2:172.21.15.97:6838/3765855684 2024-04-26T18:04:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.596+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T18:04:14.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.997+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:6838/3765855684 2024-04-26T18:04:14.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.997+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T18:04:14.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:14.997+0000 7f75c4ff9640 1 client.25717 ms_handle_refused on v2:172.21.15.97:6838/3765855684 2024-04-26T18:04:15.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:15 smithi097 ceph-mon[32419]: pgmap v237: 97 pgs: 97 active+clean; 4.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 24 MiB/s wr, 1.71k op/s 2024-04-26T18:04:15.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:15 smithi186 ceph-mon[32797]: pgmap v237: 97 pgs: 97 active+clean; 4.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 24 MiB/s wr, 1.71k op/s 2024-04-26T18:04:15.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:15 smithi031 ceph-mon[29164]: pgmap v237: 97 pgs: 97 active+clean; 4.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 24 MiB/s wr, 1.71k op/s 2024-04-26T18:04:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:15.798+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:6838/3765855684 2024-04-26T18:04:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:15.798+0000 7f75c4ff9640 1 client.25717 ms_handle_refused on v2:172.21.15.97:6838/3765855684 2024-04-26T18:04:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:15.798+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T18:04:16.907 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:16 smithi097 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 1. 2024-04-26T18:04:16.907 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:16 smithi097 systemd[1]: Stopped Ceph mds.b for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T18:04:16.908 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:16 smithi097 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.b.service: Consumed 10.727s CPU time. 2024-04-26T18:04:16.908 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:16 smithi097 systemd[1]: Starting Ceph mds.b for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:04:16.908 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:16 smithi097 podman[67830]: 2024-04-26 18:04:16.770734183 +0000 UTC m=+0.020362250 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T18:04:16.908 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:16 smithi097 podman[67830]: 2024-04-26 18:04:16.908690023 +0000 UTC m=+0.158318061 container create f9ca1792ea8814f8d198be5cf29b068bc73313db17d7c48eeebe786df3a84ab0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-26T18:04:17.179 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:17 smithi097 ceph-mon[32419]: pgmap v238: 97 pgs: 97 active+clean; 4.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 27 MiB/s wr, 1.97k op/s 2024-04-26T18:04:17.180 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:17 smithi097 podman[67830]: 2024-04-26 18:04:17.153946462 +0000 UTC m=+0.403574509 container init f9ca1792ea8814f8d198be5cf29b068bc73313db17d7c48eeebe786df3a84ab0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-26T18:04:17.180 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:17 smithi097 podman[67830]: 2024-04-26 18:04:17.15721142 +0000 UTC m=+0.406839467 container start f9ca1792ea8814f8d198be5cf29b068bc73313db17d7c48eeebe786df3a84ab0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-26T18:04:17.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:17.399+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:17.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:17.399+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:04:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:17.399+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:04:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:17.399+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:04:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:17.399+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T18:04:17.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:17 smithi186 ceph-mon[32797]: pgmap v238: 97 pgs: 97 active+clean; 4.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 27 MiB/s wr, 1.97k op/s 2024-04-26T18:04:17.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:17 smithi031 ceph-mon[29164]: pgmap v238: 97 pgs: 97 active+clean; 4.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 27 MiB/s wr, 1.97k op/s 2024-04-26T18:04:17.571 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:17 smithi097 ceph-mds[67848]: -- 172.21.15.97:0/2843439170 <== mon.2 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5556c722da40 con 0x5556c7200800 2024-04-26T18:04:17.571 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:17 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[67844]: starting mds.b at 2024-04-26T18:04:17.571 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:17 smithi097 bash[67830]: f9ca1792ea8814f8d198be5cf29b068bc73313db17d7c48eeebe786df3a84ab0 2024-04-26T18:04:17.571 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:04:17 smithi097 systemd[1]: Started Ceph mds.b for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T18:04:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 415.0 s, 2869.4 tps, lat 34.596 ms stddev 40.186 2024-04-26T18:04:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:18 smithi097 ceph-mon[32419]: daemon mds.b restarted 2024-04-26T18:04:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:18 smithi097 ceph-mon[32419]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T18:04:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:18 smithi097 ceph-mon[32419]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T18:04:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:18 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:boot 2024-04-26T18:04:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:18 smithi097 ceph-mon[32419]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-26T18:04:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:18 smithi097 ceph-mon[32419]: fsmap cephfs:2/3 {1=i=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T18:04:18.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:18 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T18:04:18.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:18 smithi097 ceph-mon[32419]: fsmap cephfs:3/3 {0=c=up:replay,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:18.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:18 smithi186 ceph-mon[32797]: daemon mds.b restarted 2024-04-26T18:04:18.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:18 smithi186 ceph-mon[32797]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T18:04:18.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:18 smithi186 ceph-mon[32797]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T18:04:18.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:18 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:boot 2024-04-26T18:04:18.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:18 smithi186 ceph-mon[32797]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-26T18:04:18.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:18 smithi186 ceph-mon[32797]: fsmap cephfs:2/3 {1=i=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T18:04:18.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:18 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T18:04:18.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:18 smithi186 ceph-mon[32797]: fsmap cephfs:3/3 {0=c=up:replay,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:18.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:18 smithi031 ceph-mon[29164]: daemon mds.b restarted 2024-04-26T18:04:18.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:18 smithi031 ceph-mon[29164]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T18:04:18.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:18 smithi031 ceph-mon[29164]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T18:04:18.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:18 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:boot 2024-04-26T18:04:18.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:18 smithi031 ceph-mon[29164]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-26T18:04:18.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:18 smithi031 ceph-mon[29164]: fsmap cephfs:2/3 {1=i=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T18:04:18.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:18 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T18:04:18.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:18 smithi031 ceph-mon[29164]: fsmap cephfs:3/3 {0=c=up:replay,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:19 smithi186 ceph-mon[32797]: pgmap v240: 97 pgs: 97 active+clean; 4.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 8.2 MiB/s wr, 804 op/s 2024-04-26T18:04:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:19 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:resolve 2024-04-26T18:04:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:19 smithi186 ceph-mon[32797]: fsmap cephfs:3/3 {0=c=up:resolve,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:19.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:19 smithi031 ceph-mon[29164]: pgmap v240: 97 pgs: 97 active+clean; 4.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 8.2 MiB/s wr, 804 op/s 2024-04-26T18:04:19.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:19 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:resolve 2024-04-26T18:04:19.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:19 smithi031 ceph-mon[29164]: fsmap cephfs:3/3 {0=c=up:resolve,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:19 smithi097 ceph-mon[32419]: pgmap v240: 97 pgs: 97 active+clean; 4.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 8.2 MiB/s wr, 804 op/s 2024-04-26T18:04:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:19 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:resolve 2024-04-26T18:04:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:19 smithi097 ceph-mon[32419]: fsmap cephfs:3/3 {0=c=up:resolve,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:20.603+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:20.603+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:04:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:20.603+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:04:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:20.603+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:04:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:20.603+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-26T18:04:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:21 smithi097 ceph-mon[32419]: pgmap v241: 97 pgs: 97 active+clean; 4.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 8.2 MiB/s wr, 804 op/s 2024-04-26T18:04:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:21 smithi097 ceph-mon[32419]: reconnect by client.15156 192.168.0.1:0/2547335269 after 0.00199996 2024-04-26T18:04:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:21 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:reconnect 2024-04-26T18:04:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:21 smithi097 ceph-mon[32419]: fsmap cephfs:3/3 {0=c=up:reconnect,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:21 smithi097 ceph-mon[32419]: reconnect by client.15330 172.21.15.31:0/1539336114 after 0.00899983 2024-04-26T18:04:21.428 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:21 smithi031 ceph-mon[29164]: pgmap v241: 97 pgs: 97 active+clean; 4.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 8.2 MiB/s wr, 804 op/s 2024-04-26T18:04:21.428 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:21 smithi031 ceph-mon[29164]: reconnect by client.15156 192.168.0.1:0/2547335269 after 0.00199996 2024-04-26T18:04:21.429 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:21 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:reconnect 2024-04-26T18:04:21.429 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:21 smithi031 ceph-mon[29164]: fsmap cephfs:3/3 {0=c=up:reconnect,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:21.429 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:21 smithi031 ceph-mon[29164]: reconnect by client.15330 172.21.15.31:0/1539336114 after 0.00899983 2024-04-26T18:04:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:21 smithi186 ceph-mon[32797]: pgmap v241: 97 pgs: 97 active+clean; 4.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 8.2 MiB/s wr, 804 op/s 2024-04-26T18:04:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:21 smithi186 ceph-mon[32797]: reconnect by client.15156 192.168.0.1:0/2547335269 after 0.00199996 2024-04-26T18:04:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:21 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:reconnect 2024-04-26T18:04:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:21 smithi186 ceph-mon[32797]: fsmap cephfs:3/3 {0=c=up:reconnect,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:21.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:21 smithi186 ceph-mon[32797]: reconnect by client.15330 172.21.15.31:0/1539336114 after 0.00899983 2024-04-26T18:04:22.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:22 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:rejoin 2024-04-26T18:04:22.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:22 smithi097 ceph-mon[32419]: fsmap cephfs:3/3 {0=c=up:rejoin,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:22.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:22 smithi097 ceph-mon[32419]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-26T18:04:22.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:22 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:rejoin 2024-04-26T18:04:22.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:22 smithi186 ceph-mon[32797]: fsmap cephfs:3/3 {0=c=up:rejoin,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:22.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:22 smithi186 ceph-mon[32797]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-26T18:04:22.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:22 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:rejoin 2024-04-26T18:04:22.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:22 smithi031 ceph-mon[29164]: fsmap cephfs:3/3 {0=c=up:rejoin,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:22.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:22 smithi031 ceph-mon[29164]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-26T18:04:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 420.0 s, 1774.0 tps, lat 56.840 ms stddev 213.101 2024-04-26T18:04:23.386 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:23 smithi097 ceph-mon[32419]: pgmap v242: 97 pgs: 97 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.1 MiB/s wr, 807 op/s 2024-04-26T18:04:23.386 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:23 smithi097 ceph-mon[32419]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T18:04:23.386 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:23 smithi097 ceph-mon[32419]: Cluster is now healthy 2024-04-26T18:04:23.386 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:23 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:active 2024-04-26T18:04:23.386 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:23 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=c=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:23.387 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:23 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=c=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:04:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:23 smithi186 ceph-mon[32797]: pgmap v242: 97 pgs: 97 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.1 MiB/s wr, 807 op/s 2024-04-26T18:04:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:23 smithi186 ceph-mon[32797]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T18:04:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:23 smithi186 ceph-mon[32797]: Cluster is now healthy 2024-04-26T18:04:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:23 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:active 2024-04-26T18:04:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:23 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=c=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:23.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:23 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=c=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:04:23.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:23 smithi031 ceph-mon[29164]: pgmap v242: 97 pgs: 97 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.1 MiB/s wr, 807 op/s 2024-04-26T18:04:23.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:23 smithi031 ceph-mon[29164]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T18:04:23.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:23 smithi031 ceph-mon[29164]: Cluster is now healthy 2024-04-26T18:04:23.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:23 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] up:active 2024-04-26T18:04:23.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:23 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=c=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:23.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:23 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=c=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:04:25.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:25 smithi186 ceph-mon[32797]: pgmap v243: 97 pgs: 97 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.1 MiB/s wr, 807 op/s 2024-04-26T18:04:25.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:25 smithi186 ceph-mon[32797]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T18:04:25.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:25 smithi031 ceph-mon[29164]: pgmap v243: 97 pgs: 97 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.1 MiB/s wr, 807 op/s 2024-04-26T18:04:25.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:25 smithi031 ceph-mon[29164]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T18:04:25.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:25 smithi097 ceph-mon[32419]: pgmap v243: 97 pgs: 97 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.1 MiB/s wr, 807 op/s 2024-04-26T18:04:25.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:25 smithi097 ceph-mon[32419]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T18:04:26.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:25.826+0000 7fca17c95700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T18:04:26.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:25.826+0000 7fca17c95700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T18:04:26.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:25.826+0000 7fca17c95700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T18:04:27.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:27.010+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:27.010+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:04:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:27.010+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:04:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:27.010+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:04:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:27.010+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-26T18:04:27.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:27 smithi186 ceph-mon[32797]: pgmap v245: 97 pgs: 97 active+clean; 4.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 17 MiB/s wr, 1.31k op/s 2024-04-26T18:04:27.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:27 smithi031 ceph-mon[29164]: pgmap v245: 97 pgs: 97 active+clean; 4.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 17 MiB/s wr, 1.31k op/s 2024-04-26T18:04:27.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:27 smithi097 ceph-mon[32419]: pgmap v245: 97 pgs: 97 active+clean; 4.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 17 MiB/s wr, 1.31k op/s 2024-04-26T18:04:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 425.0 s, 2254.2 tps, lat 44.341 ms stddev 78.962 2024-04-26T18:04:29.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:29 smithi186 ceph-mon[32797]: pgmap v246: 97 pgs: 97 active+clean; 4.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 27 MiB/s wr, 1.92k op/s 2024-04-26T18:04:29.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:29 smithi186 ceph-mon[32797]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T18:04:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:29 smithi031 ceph-mon[29164]: pgmap v246: 97 pgs: 97 active+clean; 4.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 27 MiB/s wr, 1.92k op/s 2024-04-26T18:04:29.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:29 smithi031 ceph-mon[29164]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T18:04:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:29 smithi097 ceph-mon[32419]: pgmap v246: 97 pgs: 97 active+clean; 4.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 27 MiB/s wr, 1.92k op/s 2024-04-26T18:04:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:29 smithi097 ceph-mon[32419]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T18:04:31.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:31 smithi186 ceph-mon[32797]: pgmap v248: 97 pgs: 97 active+clean; 4.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 29 MiB/s wr, 1.97k op/s 2024-04-26T18:04:31.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:31 smithi031 ceph-mon[29164]: pgmap v248: 97 pgs: 97 active+clean; 4.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 29 MiB/s wr, 1.97k op/s 2024-04-26T18:04:31.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:31 smithi097 ceph-mon[32419]: pgmap v248: 97 pgs: 97 active+clean; 4.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 29 MiB/s wr, 1.97k op/s 2024-04-26T18:04:33.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 430.0 s, 2761.2 tps, lat 36.251 ms stddev 38.400 2024-04-26T18:04:33.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:33 smithi186 ceph-mon[32797]: pgmap v249: 97 pgs: 97 active+clean; 4.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 34 MiB/s wr, 2.41k op/s 2024-04-26T18:04:33.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:33 smithi031 ceph-mon[29164]: pgmap v249: 97 pgs: 97 active+clean; 4.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 34 MiB/s wr, 2.41k op/s 2024-04-26T18:04:33.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:33 smithi097 ceph-mon[32419]: pgmap v249: 97 pgs: 97 active+clean; 4.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 34 MiB/s wr, 2.41k op/s 2024-04-26T18:04:35.018 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:34 smithi186 ceph-mon[32797]: pgmap v250: 97 pgs: 97 active+clean; 4.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 27 MiB/s wr, 1.97k op/s 2024-04-26T18:04:35.386 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:34 smithi097 ceph-mon[32419]: pgmap v250: 97 pgs: 97 active+clean; 4.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 27 MiB/s wr, 1.97k op/s 2024-04-26T18:04:35.387 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:34 smithi031 ceph-mon[29164]: pgmap v250: 97 pgs: 97 active+clean; 4.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 27 MiB/s wr, 1.97k op/s 2024-04-26T18:04:37.270 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:36 smithi186 ceph-mon[32797]: pgmap v251: 97 pgs: 97 active+clean; 4.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 17 MiB/s wr, 1.36k op/s 2024-04-26T18:04:37.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:36 smithi031 ceph-mon[29164]: pgmap v251: 97 pgs: 97 active+clean; 4.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 17 MiB/s wr, 1.36k op/s 2024-04-26T18:04:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:36 smithi097 ceph-mon[32419]: pgmap v251: 97 pgs: 97 active+clean; 4.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 17 MiB/s wr, 1.36k op/s 2024-04-26T18:04:38.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 435.0 s, 2688.4 tps, lat 37.154 ms stddev 41.369 2024-04-26T18:04:39.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:38 smithi186 ceph-mon[32797]: pgmap v252: 97 pgs: 97 active+clean; 4.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.0 MiB/s wr, 797 op/s 2024-04-26T18:04:39.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:38 smithi031 ceph-mon[29164]: pgmap v252: 97 pgs: 97 active+clean; 4.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.0 MiB/s wr, 797 op/s 2024-04-26T18:04:39.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:38 smithi097 ceph-mon[32419]: pgmap v252: 97 pgs: 97 active+clean; 4.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.0 MiB/s wr, 797 op/s 2024-04-26T18:04:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:39.812+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:39.812+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:04:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:39.812+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:04:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:39.812+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:04:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:39.812+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:04:41.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:40 smithi186 ceph-mon[32797]: pgmap v253: 97 pgs: 97 active+clean; 4.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.1 MiB/s wr, 710 op/s 2024-04-26T18:04:41.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:40 smithi031 ceph-mon[29164]: pgmap v253: 97 pgs: 97 active+clean; 4.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.1 MiB/s wr, 710 op/s 2024-04-26T18:04:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:40 smithi097 ceph-mon[32419]: pgmap v253: 97 pgs: 97 active+clean; 4.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.1 MiB/s wr, 710 op/s 2024-04-26T18:04:41.983 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.7 2024-04-26T18:04:41.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T18:04:42.371 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:04:43.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:43 smithi031 ceph-mon[29164]: pgmap v254: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 9.4 MiB/s wr, 944 op/s 2024-04-26T18:04:43.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 440.0 s, 2696.8 tps, lat 36.962 ms stddev 39.116 2024-04-26T18:04:43.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-mon[32797]: pgmap v254: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 9.4 MiB/s wr, 944 op/s 2024-04-26T18:04:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:43 smithi097 ceph-mon[32419]: pgmap v254: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 9.4 MiB/s wr, 944 op/s 2024-04-26T18:04:44.258 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:44 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:04:44.266 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:44 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:04:44.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fca1a49a700 time 2024-04-26T18:04:43.956226+0000 2024-04-26T18:04:44.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T18:04:44.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:44.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fca24ac33dd] 2024-04-26T18:04:44.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fca24ac35a3] 2024-04-26T18:04:44.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557e6d2a8822] 2024-04-26T18:04:44.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557e6d218598] 2024-04-26T18:04:44.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e6d21ac17] 2024-04-26T18:04:44.268 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e6d113d0b] 2024-04-26T18:04:44.268 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e6d1142ac] 2024-04-26T18:04:44.268 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557e6d0fcfad] 2024-04-26T18:04:44.268 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fca24d57e20] 2024-04-26T18:04:44.268 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 10: (DispatchQueue::entry()+0x51f) [0x7fca24d54fdf] 2024-04-26T18:04:44.268 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca24e27fc1] 2024-04-26T18:04:44.268 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fca238411ca] 2024-04-26T18:04:44.268 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 13: clone() 2024-04-26T18:04:44.268 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:43.956+0000 7fca1a49a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fca1a49a700 time 2024-04-26T18:04:43.956226+0000 2024-04-26T18:04:44.268 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T18:04:44.268 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:44.269 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:44.269 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fca24ac33dd] 2024-04-26T18:04:44.269 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fca24ac35a3] 2024-04-26T18:04:44.269 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557e6d2a8822] 2024-04-26T18:04:44.269 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557e6d218598] 2024-04-26T18:04:44.270 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e6d21ac17] 2024-04-26T18:04:44.270 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e6d113d0b] 2024-04-26T18:04:44.270 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e6d1142ac] 2024-04-26T18:04:44.270 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557e6d0fcfad] 2024-04-26T18:04:44.270 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fca24d57e20] 2024-04-26T18:04:44.270 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 10: (DispatchQueue::entry()+0x51f) [0x7fca24d54fdf] 2024-04-26T18:04:44.270 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca24e27fc1] 2024-04-26T18:04:44.270 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fca238411ca] 2024-04-26T18:04:44.270 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 13: clone() 2024-04-26T18:04:44.270 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:44.270 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: *** Caught signal (Aborted) ** 2024-04-26T18:04:44.271 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: in thread 7fca1a49a700 thread_name:ms_dispatch 2024-04-26T18:04:44.271 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:44.271 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fca2384bd20] 2024-04-26T18:04:44.271 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2: gsignal() 2024-04-26T18:04:44.271 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 3: abort() 2024-04-26T18:04:44.271 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fca24ac3437] 2024-04-26T18:04:44.271 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fca24ac35a3] 2024-04-26T18:04:44.271 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557e6d2a8822] 2024-04-26T18:04:44.271 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557e6d218598] 2024-04-26T18:04:44.271 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e6d21ac17] 2024-04-26T18:04:44.271 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e6d113d0b] 2024-04-26T18:04:44.271 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e6d1142ac] 2024-04-26T18:04:44.271 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557e6d0fcfad] 2024-04-26T18:04:44.272 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fca24d57e20] 2024-04-26T18:04:44.272 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 13: (DispatchQueue::entry()+0x51f) [0x7fca24d54fdf] 2024-04-26T18:04:44.272 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca24e27fc1] 2024-04-26T18:04:44.272 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fca238411ca] 2024-04-26T18:04:44.272 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 16: clone() 2024-04-26T18:04:44.272 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:43.958+0000 7fca1a49a700 -1 *** Caught signal (Aborted) ** 2024-04-26T18:04:44.272 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: in thread 7fca1a49a700 thread_name:ms_dispatch 2024-04-26T18:04:44.272 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:44.272 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:44.272 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fca2384bd20] 2024-04-26T18:04:44.272 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2: gsignal() 2024-04-26T18:04:44.272 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 3: abort() 2024-04-26T18:04:44.272 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fca24ac3437] 2024-04-26T18:04:44.272 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fca24ac35a3] 2024-04-26T18:04:44.273 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557e6d2a8822] 2024-04-26T18:04:44.273 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557e6d218598] 2024-04-26T18:04:44.273 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e6d21ac17] 2024-04-26T18:04:44.273 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e6d113d0b] 2024-04-26T18:04:44.273 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e6d1142ac] 2024-04-26T18:04:44.273 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557e6d0fcfad] 2024-04-26T18:04:44.273 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fca24d57e20] 2024-04-26T18:04:44.273 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 13: (DispatchQueue::entry()+0x51f) [0x7fca24d54fdf] 2024-04-26T18:04:44.273 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca24e27fc1] 2024-04-26T18:04:44.273 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fca238411ca] 2024-04-26T18:04:44.273 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 16: clone() 2024-04-26T18:04:44.273 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T18:04:44.273 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:44.274 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: -5913> 2024-04-26T18:04:25.826+0000 7fca17c95700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T18:04:44.274 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: -5911> 2024-04-26T18:04:25.826+0000 7fca17c95700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T18:04:44.274 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: -5909> 2024-04-26T18:04:25.826+0000 7fca17c95700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T18:04:44.274 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: -1> 2024-04-26T18:04:43.956+0000 7fca1a49a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fca1a49a700 time 2024-04-26T18:04:43.956226+0000 2024-04-26T18:04:44.274 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T18:04:44.274 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:44.274 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:44.274 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fca24ac33dd] 2024-04-26T18:04:44.274 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fca24ac35a3] 2024-04-26T18:04:44.274 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557e6d2a8822] 2024-04-26T18:04:44.274 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557e6d218598] 2024-04-26T18:04:44.274 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e6d21ac17] 2024-04-26T18:04:44.275 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e6d113d0b] 2024-04-26T18:04:44.275 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e6d1142ac] 2024-04-26T18:04:44.275 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557e6d0fcfad] 2024-04-26T18:04:44.275 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fca24d57e20] 2024-04-26T18:04:44.275 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 10: (DispatchQueue::entry()+0x51f) [0x7fca24d54fdf] 2024-04-26T18:04:44.275 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca24e27fc1] 2024-04-26T18:04:44.275 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fca238411ca] 2024-04-26T18:04:44.275 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 13: clone() 2024-04-26T18:04:44.275 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:44.275 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 0> 2024-04-26T18:04:43.958+0000 7fca1a49a700 -1 *** Caught signal (Aborted) ** 2024-04-26T18:04:44.275 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: in thread 7fca1a49a700 thread_name:ms_dispatch 2024-04-26T18:04:44.275 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:44.275 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:44.275 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fca2384bd20] 2024-04-26T18:04:44.276 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2: gsignal() 2024-04-26T18:04:44.276 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 3: abort() 2024-04-26T18:04:44.276 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fca24ac3437] 2024-04-26T18:04:44.276 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fca24ac35a3] 2024-04-26T18:04:44.276 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557e6d2a8822] 2024-04-26T18:04:44.276 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557e6d218598] 2024-04-26T18:04:44.276 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e6d21ac17] 2024-04-26T18:04:44.276 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e6d113d0b] 2024-04-26T18:04:44.276 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e6d1142ac] 2024-04-26T18:04:44.276 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557e6d0fcfad] 2024-04-26T18:04:44.276 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fca24d57e20] 2024-04-26T18:04:44.276 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 13: (DispatchQueue::entry()+0x51f) [0x7fca24d54fdf] 2024-04-26T18:04:44.276 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca24e27fc1] 2024-04-26T18:04:44.276 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fca238411ca] 2024-04-26T18:04:44.277 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 16: clone() 2024-04-26T18:04:44.277 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T18:04:44.277 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:44.277 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: -5913> 2024-04-26T18:04:25.826+0000 7fca17c95700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T18:04:44.277 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: -5911> 2024-04-26T18:04:25.826+0000 7fca17c95700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T18:04:44.277 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: -5909> 2024-04-26T18:04:25.826+0000 7fca17c95700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T18:04:44.277 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: -1> 2024-04-26T18:04:43.956+0000 7fca1a49a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fca1a49a700 time 2024-04-26T18:04:43.956226+0000 2024-04-26T18:04:44.277 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3267-gbc94a2a9/rpm/el8/BUILD/ceph-19.0.0-3267-gbc94a2a9/src/mds/MDCache.cc: 9728: FAILED ceph_assert(p != active_requests.end()) 2024-04-26T18:04:44.278 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:44.278 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:44.278 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fca24ac33dd] 2024-04-26T18:04:44.278 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fca24ac35a3] 2024-04-26T18:04:44.278 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 3: (MDCache::request_get(metareqid_t)+0x1c2) [0x557e6d2a8822] 2024-04-26T18:04:44.278 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557e6d218598] 2024-04-26T18:04:44.278 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 5: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e6d21ac17] 2024-04-26T18:04:44.278 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e6d113d0b] 2024-04-26T18:04:44.278 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e6d1142ac] 2024-04-26T18:04:44.278 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557e6d0fcfad] 2024-04-26T18:04:44.278 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fca24d57e20] 2024-04-26T18:04:44.278 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 10: (DispatchQueue::entry()+0x51f) [0x7fca24d54fdf] 2024-04-26T18:04:44.278 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca24e27fc1] 2024-04-26T18:04:44.279 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fca238411ca] 2024-04-26T18:04:44.279 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 13: clone() 2024-04-26T18:04:44.279 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:44.279 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 0> 2024-04-26T18:04:43.958+0000 7fca1a49a700 -1 *** Caught signal (Aborted) ** 2024-04-26T18:04:44.279 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: in thread 7fca1a49a700 thread_name:ms_dispatch 2024-04-26T18:04:44.279 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:44.279 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: ceph version 19.0.0-3267-gbc94a2a9 (bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1) squid (dev) 2024-04-26T18:04:44.279 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fca2384bd20] 2024-04-26T18:04:44.279 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2: gsignal() 2024-04-26T18:04:44.279 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 3: abort() 2024-04-26T18:04:44.279 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fca24ac3437] 2024-04-26T18:04:44.279 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fca24ac35a3] 2024-04-26T18:04:44.279 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 6: (MDCache::request_get(metareqid_t)+0x1c2) [0x557e6d2a8822] 2024-04-26T18:04:44.279 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xa8) [0x557e6d218598] 2024-04-26T18:04:44.280 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557e6d21ac17] 2024-04-26T18:04:44.280 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557e6d113d0b] 2024-04-26T18:04:44.280 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557e6d1142ac] 2024-04-26T18:04:44.280 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bd) [0x557e6d0fcfad] 2024-04-26T18:04:44.280 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x470) [0x7fca24d57e20] 2024-04-26T18:04:44.280 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 13: (DispatchQueue::entry()+0x51f) [0x7fca24d54fdf] 2024-04-26T18:04:44.280 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fca24e27fc1] 2024-04-26T18:04:44.280 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fca238411ca] 2024-04-26T18:04:44.280 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 16: clone() 2024-04-26T18:04:44.280 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-26T18:04:44.280 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[63378]: 2024-04-26T18:04:44.280 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:44 smithi186 podman[68655]: 2024-04-26 18:04:44.044002135 +0000 UTC m=+0.044172625 container died 84410eca4b7856845d58a4b2cfd7e948420e9eb959b2db9630a3d46d35068066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T18:04:44.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:44 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:04:44.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.697+0000 7fe19ca17700 1 -- 172.21.15.31:0/1143371713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe198102ce0 msgr2=0x7fe19810f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:44.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.697+0000 7fe19ca17700 1 --2- 172.21.15.31:0/1143371713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe198102ce0 0x7fe19810f1d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe19000d330 tx=0x7fe19000d640 comp rx=0 tx=0).stop 2024-04-26T18:04:44.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.700+0000 7fe19ca17700 1 -- 172.21.15.31:0/1143371713 shutdown_connections 2024-04-26T18:04:44.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.700+0000 7fe19ca17700 1 --2- 172.21.15.31:0/1143371713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe198108390 0x7fe198108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:44.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.700+0000 7fe19ca17700 1 --2- 172.21.15.31:0/1143371713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe198102ce0 0x7fe19810f1d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:44.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.700+0000 7fe19ca17700 1 --2- 172.21.15.31:0/1143371713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe198102320 0x7fe1981027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:44.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.700+0000 7fe19ca17700 1 -- 172.21.15.31:0/1143371713 >> 172.21.15.31:0/1143371713 conn(0x7fe1980792d0 msgr2=0x7fe1980796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:04:44.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.701+0000 7fe19ca17700 1 -- 172.21.15.31:0/1143371713 shutdown_connections 2024-04-26T18:04:44.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.702+0000 7fe19ca17700 1 -- 172.21.15.31:0/1143371713 wait complete. 2024-04-26T18:04:44.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.703+0000 7fe19ca17700 1 Processor -- start 2024-04-26T18:04:44.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.704+0000 7fe19ca17700 1 -- start start 2024-04-26T18:04:44.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.704+0000 7fe19ca17700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe198102320 0x7fe198074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:44.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.704+0000 7fe19ca17700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe198102ce0 0x7fe19806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:44.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.704+0000 7fe19ca17700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe198108390 0x7fe19806e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:44.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.704+0000 7fe19ca17700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe198119d40 con 0x7fe198102ce0 2024-04-26T18:04:44.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.704+0000 7fe19ca17700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe198119bc0 con 0x7fe198102320 2024-04-26T18:04:44.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.704+0000 7fe19ca17700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe198119ec0 con 0x7fe198108390 2024-04-26T18:04:44.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.705+0000 7fe196d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe198108390 0x7fe19806e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:44.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.705+0000 7fe196d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe198108390 0x7fe19806e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:49908/0 (socket says 172.21.15.31:49908) 2024-04-26T18:04:44.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.705+0000 7fe196d9d700 1 -- 172.21.15.31:0/73011954 learned_addr learned my addr 172.21.15.31:0/73011954 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:04:44.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.706+0000 7fe19659c700 1 --2- 172.21.15.31:0/73011954 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe198102320 0x7fe198074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:44.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.706+0000 7fe195d9b700 1 --2- 172.21.15.31:0/73011954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe198102ce0 0x7fe19806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:44.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.708+0000 7fe195d9b700 1 -- 172.21.15.31:0/73011954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe198108390 msgr2=0x7fe19806e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:44.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.708+0000 7fe195d9b700 1 --2- 172.21.15.31:0/73011954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe198108390 0x7fe19806e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:44.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.708+0000 7fe195d9b700 1 -- 172.21.15.31:0/73011954 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe198102320 msgr2=0x7fe198074c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:44.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.708+0000 7fe195d9b700 1 --2- 172.21.15.31:0/73011954 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe198102320 0x7fe198074c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:44.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.708+0000 7fe195d9b700 1 -- 172.21.15.31:0/73011954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19806e850 con 0x7fe198102ce0 2024-04-26T18:04:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.711+0000 7fe195d9b700 1 --2- 172.21.15.31:0/73011954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe198102ce0 0x7fe19806dcd0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe18c006fd0 tx=0x7fe18c00b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.714+0000 7fe1877fe700 1 -- 172.21.15.31:0/73011954 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c015020 con 0x7fe198102ce0 2024-04-26T18:04:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.714+0000 7fe19ca17700 1 -- 172.21.15.31:0/73011954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe198072770 con 0x7fe198102ce0 2024-04-26T18:04:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.715+0000 7fe19ca17700 1 -- 172.21.15.31:0/73011954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe198072cb0 con 0x7fe198102ce0 2024-04-26T18:04:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.715+0000 7fe1877fe700 1 -- 172.21.15.31:0/73011954 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe18c00be30 con 0x7fe198102ce0 2024-04-26T18:04:44.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.716+0000 7fe1877fe700 1 -- 172.21.15.31:0/73011954 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c010c60 con 0x7fe198102ce0 2024-04-26T18:04:44.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.718+0000 7fe19ca17700 1 -- 172.21.15.31:0/73011954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe178005330 con 0x7fe198102ce0 2024-04-26T18:04:44.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.719+0000 7fe1877fe700 1 -- 172.21.15.31:0/73011954 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe18c018020 con 0x7fe198102ce0 2024-04-26T18:04:44.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.720+0000 7fe1877fe700 1 --2- 172.21.15.31:0/73011954 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe1800a3e60 0x7fe1800a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:44.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.720+0000 7fe1877fe700 1 -- 172.21.15.31:0/73011954 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8570+0+0 (secure 0 0 0) 0x7fe18c0cfbe0 con 0x7fe198102ce0 2024-04-26T18:04:44.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.721+0000 7fe19659c700 1 --2- 172.21.15.31:0/73011954 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe1800a3e60 0x7fe1800a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:44.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.729+0000 7fe1877fe700 1 -- 172.21.15.31:0/73011954 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe18c01f020 con 0x7fe198102ce0 2024-04-26T18:04:44.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:44.729+0000 7fe19659c700 1 --2- 172.21.15.31:0/73011954 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe1800a3e60 0x7fe1800a6320 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe188006fd0 tx=0x7fe188009190 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:45.064 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:45 smithi186 ceph-mon[32797]: pgmap v255: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.5 MiB/s wr, 650 op/s 2024-04-26T18:04:45.065 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:45 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:04:45.065 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:45 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:04:45.065 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:45 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:04:45.065 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:44 smithi186 podman[68655]: 2024-04-26 18:04:44.803948985 +0000 UTC m=+0.804119456 container remove 84410eca4b7856845d58a4b2cfd7e948420e9eb959b2db9630a3d46d35068066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T18:04:45.065 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:44 smithi186 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-26T18:04:45.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:45 smithi031 ceph-mon[29164]: pgmap v255: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.5 MiB/s wr, 650 op/s 2024-04-26T18:04:45.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:45 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:04:45.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:45 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:04:45.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:45 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:04:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:45.229+0000 7fe19ca17700 1 -- 172.21.15.31:0/73011954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1780058f0 con 0x7fe198102ce0 2024-04-26T18:04:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:45.231+0000 7fe1877fe700 1 -- 172.21.15.31:0/73011954 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11875 (secure 0 0 0) 0x7fe18c09c300 con 0x7fe198102ce0 2024-04-26T18:04:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:04:45.231 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T18:04:22.118924+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24445,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":181,"addr":"172.21.15.186:6837/1510454985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1510454985},{"type":"v1","addr":"172.21.15.186:6837","nonce":1510454985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25799":{"gid":25799,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6839/2506986520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2506986520},{"type":"v1","addr":"172.21.15.97:6839","nonce":2506986520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24445,"qdb_cluster":[24445,24446,24457]},"id":1}]} 2024-04-26T18:04:45.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:45.235+0000 7fe19ca17700 1 -- 172.21.15.31:0/73011954 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe1800a3e60 msgr2=0x7fe1800a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:45.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:45.235+0000 7fe19ca17700 1 --2- 172.21.15.31:0/73011954 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe1800a3e60 0x7fe1800a6320 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe188006fd0 tx=0x7fe188009190 comp rx=0 tx=0).stop 2024-04-26T18:04:45.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:45.235+0000 7fe19ca17700 1 -- 172.21.15.31:0/73011954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe198102ce0 msgr2=0x7fe19806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:45.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:45.235+0000 7fe19ca17700 1 --2- 172.21.15.31:0/73011954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe198102ce0 0x7fe19806dcd0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe18c006fd0 tx=0x7fe18c00b730 comp rx=0 tx=0).stop 2024-04-26T18:04:45.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:45.235+0000 7fe19ca17700 1 -- 172.21.15.31:0/73011954 shutdown_connections 2024-04-26T18:04:45.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:45.235+0000 7fe19ca17700 1 --2- 172.21.15.31:0/73011954 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe1800a3e60 0x7fe1800a6320 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:45.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:45.235+0000 7fe19ca17700 1 --2- 172.21.15.31:0/73011954 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe198102320 0x7fe198074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:45.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:45.235+0000 7fe19ca17700 1 --2- 172.21.15.31:0/73011954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe198108390 0x7fe19806e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:45.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:45.235+0000 7fe19ca17700 1 --2- 172.21.15.31:0/73011954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe198102ce0 0x7fe19806dcd0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:45.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:45.235+0000 7fe19ca17700 1 -- 172.21.15.31:0/73011954 >> 172.21.15.31:0/73011954 conn(0x7fe1980792d0 msgr2=0x7fe198100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:04:45.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:45.235+0000 7fe19ca17700 1 -- 172.21.15.31:0/73011954 shutdown_connections 2024-04-26T18:04:45.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:45.236+0000 7fe19ca17700 1 -- 172.21.15.31:0/73011954 wait complete. 2024-04-26T18:04:45.237 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 43 2024-04-26T18:04:45.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:45 smithi097 ceph-mon[32419]: pgmap v255: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.5 MiB/s wr, 650 op/s 2024-04-26T18:04:45.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:45 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:04:45.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:45 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:04:45.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:45 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:04:45.516 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:45 smithi186 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-26T18:04:45.517 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:45 smithi186 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.c.service: Consumed 3.103s CPU time. 2024-04-26T18:04:45.902 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24445 quiesce db / --timeout 60 --expiration 89.7 --await-for 42.0' 2024-04-26T18:04:45.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24445 quiesce db / --timeout 60 --expiration 89.7 --await-for 42.0 2024-04-26T18:04:46.213 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:46 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/73011954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:04:46.305 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:04:46.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:46 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/73011954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:04:46.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:46 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/73011954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:04:47.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:47 smithi031 ceph-mon[29164]: pgmap v256: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.0 MiB/s wr, 869 op/s 2024-04-26T18:04:47.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:47 smithi097 ceph-mon[32419]: pgmap v256: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.0 MiB/s wr, 869 op/s 2024-04-26T18:04:47.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:47 smithi186 ceph-mon[32797]: pgmap v256: 97 pgs: 97 active+clean; 4.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.0 MiB/s wr, 869 op/s 2024-04-26T18:04:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 445.0 s, 2696.6 tps, lat 37.115 ms stddev 45.366 2024-04-26T18:04:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.385+0000 7f7ab303a700 1 -- 172.21.15.31:0/1283074411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7aac105660 msgr2=0x7f7aac105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.385+0000 7f7ab303a700 1 --2- 172.21.15.31:0/1283074411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7aac105660 0x7f7aac105ae0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7a980098f0 tx=0x7f7a98009c00 comp rx=0 tx=0).stop 2024-04-26T18:04:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.385+0000 7f7ab303a700 1 -- 172.21.15.31:0/1283074411 shutdown_connections 2024-04-26T18:04:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.385+0000 7f7ab303a700 1 --2- 172.21.15.31:0/1283074411 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7aac106020 0x7f7aac112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.385+0000 7f7ab303a700 1 --2- 172.21.15.31:0/1283074411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7aac10b660 0x7f7aac10ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.385+0000 7f7ab303a700 1 --2- 172.21.15.31:0/1283074411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7aac105660 0x7f7aac105ae0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.385+0000 7f7ab303a700 1 -- 172.21.15.31:0/1283074411 >> 172.21.15.31:0/1283074411 conn(0x7f7aac101150 msgr2=0x7f7aac103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:04:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.385+0000 7f7ab303a700 1 -- 172.21.15.31:0/1283074411 shutdown_connections 2024-04-26T18:04:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.385+0000 7f7ab303a700 1 -- 172.21.15.31:0/1283074411 wait complete. 2024-04-26T18:04:48.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.386+0000 7f7ab303a700 1 Processor -- start 2024-04-26T18:04:48.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.387+0000 7f7ab303a700 1 -- start start 2024-04-26T18:04:48.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.388+0000 7f7ab303a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7aac105660 0x7f7aac1a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:48.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.388+0000 7f7ab303a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7aac106020 0x7f7aac1a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:48.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.388+0000 7f7ab303a700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7aac10b660 0x7f7aac19bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:48.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.388+0000 7f7ab0dd6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7aac105660 0x7f7aac1a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:48.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.388+0000 7f7ab0dd6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7aac105660 0x7f7aac1a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:49928/0 (socket says 172.21.15.31:49928) 2024-04-26T18:04:48.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.388+0000 7f7ab0dd6700 1 -- 172.21.15.31:0/754212647 learned_addr learned my addr 172.21.15.31:0/754212647 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:04:48.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.388+0000 7f7ab303a700 1 -- 172.21.15.31:0/754212647 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac1158f0 con 0x7f7aac106020 2024-04-26T18:04:48.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.388+0000 7f7ab303a700 1 -- 172.21.15.31:0/754212647 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac115770 con 0x7f7aac10b660 2024-04-26T18:04:48.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.388+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/754212647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7aac10b660 0x7f7aac19bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:48.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.388+0000 7f7ab303a700 1 -- 172.21.15.31:0/754212647 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac115a70 con 0x7f7aac105660 2024-04-26T18:04:48.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.389+0000 7f7ab0dd6700 1 -- 172.21.15.31:0/754212647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7aac10b660 msgr2=0x7f7aac19bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:48.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.389+0000 7f7ab0dd6700 1 --2- 172.21.15.31:0/754212647 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7aac10b660 0x7f7aac19bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:48.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.389+0000 7f7ab0dd6700 1 -- 172.21.15.31:0/754212647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7aac106020 msgr2=0x7f7aac1a1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:48.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.389+0000 7f7ab0dd6700 1 --2- 172.21.15.31:0/754212647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7aac106020 0x7f7aac1a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:48.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.389+0000 7f7ab0dd6700 1 -- 172.21.15.31:0/754212647 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aac19c3f0 con 0x7f7aac105660 2024-04-26T18:04:48.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.389+0000 7f7ab0dd6700 1 --2- 172.21.15.31:0/754212647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7aac105660 0x7f7aac1a1a40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000e9a0 tx=0x7f7aa000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:48.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.389+0000 7f7aa9ffb700 1 -- 172.21.15.31:0/754212647 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa000cbd0 con 0x7f7aac105660 2024-04-26T18:04:48.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.389+0000 7f7aa9ffb700 1 -- 172.21.15.31:0/754212647 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7aa000cd70 con 0x7f7aac105660 2024-04-26T18:04:48.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.389+0000 7f7aa9ffb700 1 -- 172.21.15.31:0/754212647 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0018680 con 0x7f7aac105660 2024-04-26T18:04:48.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.389+0000 7f7ab303a700 1 -- 172.21.15.31:0/754212647 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aac19c6e0 con 0x7f7aac105660 2024-04-26T18:04:48.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.389+0000 7f7ab303a700 1 -- 172.21.15.31:0/754212647 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aac1a89c0 con 0x7f7aac105660 2024-04-26T18:04:48.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.394+0000 7f7aa9ffb700 1 -- 172.21.15.31:0/754212647 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7aa0018820 con 0x7f7aac105660 2024-04-26T18:04:48.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.395+0000 7f7abd8ccb80 1 Processor -- start 2024-04-26T18:04:48.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.395+0000 7f7abd8ccb80 1 -- start start 2024-04-26T18:04:48.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7abd8ccb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640770eca40 0x5640770ece20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:48.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7abd8ccb80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640770ed360 0x5640770f7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:48.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7abd8ccb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640770f8210 0x5640770fa600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:48.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7ab15d7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640770eca40 0x5640770ece20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:48.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7ab15d7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640770eca40 0x5640770ece20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:49944/0 (socket says 172.21.15.31:49944) 2024-04-26T18:04:48.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7ab15d7700 1 -- 172.21.15.31:0/2770654193 learned_addr learned my addr 172.21.15.31:0/2770654193 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:04:48.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7abd8ccb80 1 -- 172.21.15.31:0/2770654193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5640770378d0 con 0x5640770f8210 2024-04-26T18:04:48.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7abd8ccb80 1 -- 172.21.15.31:0/2770654193 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x564076fd6770 con 0x5640770ed360 2024-04-26T18:04:48.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7abd8ccb80 1 -- 172.21.15.31:0/2770654193 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564077033890 con 0x5640770eca40 2024-04-26T18:04:48.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7aabfff700 1 --2- 172.21.15.31:0/2770654193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640770f8210 0x5640770fa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:48.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7ab15d7700 1 -- 172.21.15.31:0/2770654193 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640770ed360 msgr2=0x5640770f7cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T18:04:48.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/2770654193 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640770ed360 0x5640770f7cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:48.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7ab15d7700 1 -- 172.21.15.31:0/2770654193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640770f8210 msgr2=0x5640770fa600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:04:48.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/2770654193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640770f8210 0x5640770fa600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:48.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7ab15d7700 1 -- 172.21.15.31:0/2770654193 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564076f84a60 con 0x5640770eca40 2024-04-26T18:04:48.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/2770654193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640770eca40 0x5640770ece20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4011670 tx=0x7f7aa40099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adbc61469e3df147 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:48.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.396+0000 7f7aa9ffb700 1 --2- 172.21.15.31:0/754212647 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7a9c0b5040 0x7f7a9c0b7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:48.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.397+0000 7f7a96ffd700 1 -- 172.21.15.31:0/2770654193 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4019070 con 0x5640770eca40 2024-04-26T18:04:48.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.397+0000 7f7a96ffd700 1 -- 172.21.15.31:0/2770654193 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7aa4004020 con 0x5640770eca40 2024-04-26T18:04:48.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.397+0000 7f7a96ffd700 1 -- 172.21.15.31:0/2770654193 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4007820 con 0x5640770eca40 2024-04-26T18:04:48.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.397+0000 7f7aa9ffb700 1 -- 172.21.15.31:0/754212647 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8570+0+0 (secure 0 0 0) 0x7f7aa00d0960 con 0x7f7aac105660 2024-04-26T18:04:48.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.398+0000 7f7abd8ccb80 1 -- 172.21.15.31:0/2770654193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640770eca40 msgr2=0x5640770ece20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:04:48.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.398+0000 7f7abd8ccb80 1 --2- 172.21.15.31:0/2770654193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640770eca40 0x5640770ece20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4011670 tx=0x7f7aa40099e0 comp rx=0 tx=0).stop 2024-04-26T18:04:48.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.397+0000 7f7ab0dd6700 1 --2- 172.21.15.31:0/754212647 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7a9c0b5040 0x7f7a9c0b7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:48.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.398+0000 7f7abd8ccb80 1 -- 172.21.15.31:0/2770654193 shutdown_connections 2024-04-26T18:04:48.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.398+0000 7f7abd8ccb80 1 --2- 172.21.15.31:0/2770654193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640770f8210 0x5640770fa600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.398+0000 7f7abd8ccb80 1 --2- 172.21.15.31:0/2770654193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640770eca40 0x5640770ece20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.398+0000 7f7abd8ccb80 1 --2- 172.21.15.31:0/2770654193 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640770ed360 0x5640770f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.398+0000 7f7abd8ccb80 1 -- 172.21.15.31:0/2770654193 >> 172.21.15.31:0/2770654193 conn(0x564077011ed0 msgr2=0x5640770e0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:04:48.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.398+0000 7f7abd8ccb80 1 -- 172.21.15.31:0/2770654193 shutdown_connections 2024-04-26T18:04:48.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.398+0000 7f7abd8ccb80 1 -- 172.21.15.31:0/2770654193 wait complete. 2024-04-26T18:04:48.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.399+0000 7f7abd8ccb80 1 -- start start 2024-04-26T18:04:48.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.399+0000 7f7ab0dd6700 1 --2- 172.21.15.31:0/754212647 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f7a9c0b5040 0x7f7a9c0b7500 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x5640770e1560 tx=0x7f7aa000cf20 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:48.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.400+0000 7f7abd8ccb80 1 Processor -- start 2024-04-26T18:04:48.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.400+0000 7f7abd8ccb80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640770eca40 0x5640770e7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:48.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.400+0000 7f7abd8ccb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640770ed360 0x5640770e75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:48.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.400+0000 7f7ab15d7700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640770eca40 0x5640770e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:48.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.400+0000 7f7ab15d7700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640770eca40 0x5640770e7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:41678/0 (socket says 172.21.15.31:41678) 2024-04-26T18:04:48.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.401+0000 7f7aabfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640770ed360 0x5640770e75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:48.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.401+0000 7f7aabfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640770ed360 0x5640770e75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56578/0 (socket says 172.21.15.31:56578) 2024-04-26T18:04:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.401+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 learned_addr learned my addr 172.21.15.31:0/69685200 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:04:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.401+0000 7f7abd8ccb80 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640770f8210 0x5640770ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.401+0000 7f7abd8ccb80 1 -- 172.21.15.31:0/69685200 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5640770e2460 con 0x5640770ed360 2024-04-26T18:04:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.401+0000 7f7abd8ccb80 1 -- 172.21.15.31:0/69685200 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x564076f9dae0 con 0x5640770eca40 2024-04-26T18:04:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.401+0000 7f7abd8ccb80 1 -- 172.21.15.31:0/69685200 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5640770e25e0 con 0x5640770f8210 2024-04-26T18:04:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.401+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640770f8210 0x5640770ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.402+0000 7f7abd8ccb80 10 client.0 _ll_register_callbacks cb 0x564076fcb710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:04:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.402+0000 7f7aabfff700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640770f8210 msgr2=0x5640770ebe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:04:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.402+0000 7f7aabfff700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640770f8210 0x5640770ebe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.402+0000 7f7aabfff700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640770eca40 msgr2=0x5640770e7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:04:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.402+0000 7f7aabfff700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5640770eca40 0x5640770e7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:04:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.402+0000 7f7aabfff700 1 -- 172.21.15.31:0/69685200 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640770ec5d0 con 0x5640770ed360 2024-04-26T18:04:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.402+0000 7f7abd8ccb80 1 -- 172.21.15.31:0/69685200 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7aa4004020 con 0x5640770ed360 2024-04-26T18:04:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.402+0000 7f7aabfff700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640770ed360 0x5640770e75c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7a98009230 tx=0x7f7a9800fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f2887375f0ddc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:04:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.402+0000 7f7a8bfff700 10 client.16140 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:04:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.403+0000 7f7a8bfff700 1 -- 172.21.15.31:0/69685200 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a98047060 con 0x5640770ed360 2024-04-26T18:04:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.403+0000 7f7a8bfff700 1 -- 172.21.15.31:0/69685200 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8570+0+0 (secure 0 0 0) 0x7f7a98005450 con 0x5640770ed360 2024-04-26T18:04:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.403+0000 7f7a8bfff700 1 -- 172.21.15.31:0/69685200 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f7a980056d0 con 0x5640770ed360 2024-04-26T18:04:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.403+0000 7f7abd8ccb80 10 client.16140 fetch_fsmap learned FSMap version 43 2024-04-26T18:04:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.403+0000 7f7abd8ccb80 1 -- 172.21.15.31:0/69685200 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=43}) -- 0x56407710e0b0 con 0x5640770ed360 2024-04-26T18:04:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.404+0000 7f7a8bfff700 1 -- 172.21.15.31:0/69685200 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 43) ==== 8342+0+0 (secure 0 0 0) 0x7f7a98041070 con 0x5640770ed360 2024-04-26T18:04:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.404+0000 7f7abd8ccb80 10 client.16140 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T18:04:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.404+0000 7f7abd8ccb80 10 client.16140 resolve_mds: validated gid 24445 aka daemon mds.c 2024-04-26T18:04:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.404+0000 7f7abd8ccb80 20 client.16140 populate_metadata read hostname 'smithi031' 2024-04-26T18:04:48.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.404+0000 7f7abd8ccb80 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:04:48.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.404+0000 7f7abd8ccb80 4 client.16140 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:04:48.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.404+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:6836/1510454985 2024-04-26T18:04:48.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.404+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T18:04:48.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.404+0000 7f7a8bfff700 1 client.16140 ms_handle_refused on v2:172.21.15.186:6836/1510454985 2024-04-26T18:04:48.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.404+0000 7f7abd8ccb80 1 -- 172.21.15.31:0/69685200 --> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640770452b0 con 0x564077110580 2024-04-26T18:04:48.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.605+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:6836/1510454985 2024-04-26T18:04:48.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.605+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T18:04:48.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:48.605+0000 7f7a8bfff700 1 client.16140 ms_handle_refused on v2:172.21.15.186:6836/1510454985 2024-04-26T18:04:49.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:49.006+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:6836/1510454985 2024-04-26T18:04:49.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:49.006+0000 7f7a8bfff700 1 client.16140 ms_handle_refused on v2:172.21.15.186:6836/1510454985 2024-04-26T18:04:49.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:49.006+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T18:04:49.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:49 smithi031 ceph-mon[29164]: pgmap v257: 97 pgs: 97 active+clean; 4.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 787 op/s 2024-04-26T18:04:49.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:49 smithi097 ceph-mon[32419]: pgmap v257: 97 pgs: 97 active+clean; 4.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 787 op/s 2024-04-26T18:04:49.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:49 smithi186 ceph-mon[32797]: pgmap v257: 97 pgs: 97 active+clean; 4.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 787 op/s 2024-04-26T18:04:49.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:49.807+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:6836/1510454985 2024-04-26T18:04:49.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:49.807+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T18:04:49.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:49.807+0000 7f7a8bfff700 1 client.16140 ms_handle_refused on v2:172.21.15.186:6836/1510454985 2024-04-26T18:04:51.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:51 smithi031 ceph-mon[29164]: pgmap v258: 97 pgs: 97 active+clean; 4.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 MiB/s wr, 618 op/s 2024-04-26T18:04:51.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:51 smithi097 ceph-mon[32419]: pgmap v258: 97 pgs: 97 active+clean; 4.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 MiB/s wr, 618 op/s 2024-04-26T18:04:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:51.409+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:6836/1510454985 2024-04-26T18:04:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:51.409+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T18:04:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:51.409+0000 7f7a8bfff700 1 client.16140 ms_handle_refused on v2:172.21.15.186:6836/1510454985 2024-04-26T18:04:51.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:51 smithi186 ceph-mon[32797]: pgmap v258: 97 pgs: 97 active+clean; 4.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 MiB/s wr, 618 op/s 2024-04-26T18:04:53.097 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 450.0 s, 689.9 tps, lat 30.921 ms stddev 31.480 2024-04-26T18:04:53.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:53 smithi031 ceph-mon[29164]: pgmap v259: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.8 MiB/s wr, 887 op/s 2024-04-26T18:04:53.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:53 smithi097 ceph-mon[32419]: pgmap v259: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.8 MiB/s wr, 887 op/s 2024-04-26T18:04:53.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:53 smithi186 ceph-mon[32797]: pgmap v259: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.8 MiB/s wr, 887 op/s 2024-04-26T18:04:54.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:54.613+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:6836/1510454985 2024-04-26T18:04:54.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:54.613+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-26T18:04:54.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:04:54.613+0000 7f7a8bfff700 1 client.16140 ms_handle_refused on v2:172.21.15.186:6836/1510454985 2024-04-26T18:04:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:54.816+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:04:54.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:54.817+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:04:54.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:54.817+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:04:54.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:54.817+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:04:54.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:04:54.817+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:04:55.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:54 smithi186 ceph-mon[32797]: pgmap v260: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.1 MiB/s wr, 607 op/s 2024-04-26T18:04:55.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:54 smithi031 ceph-mon[29164]: pgmap v260: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.1 MiB/s wr, 607 op/s 2024-04-26T18:04:55.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:54 smithi097 ceph-mon[32419]: pgmap v260: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.1 MiB/s wr, 607 op/s 2024-04-26T18:04:55.529 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:55 smithi186 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-04-26T18:04:55.529 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:55 smithi186 systemd[1]: Stopped Ceph mds.c for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T18:04:55.529 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:55 smithi186 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.c.service: Consumed 3.103s CPU time. 2024-04-26T18:04:55.529 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:55 smithi186 systemd[1]: Starting Ceph mds.c for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:04:55.529 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:55 smithi186 podman[68785]: 2024-04-26 18:04:55.406829844 +0000 UTC m=+0.018567366 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T18:04:55.782 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:55 smithi186 podman[68785]: 2024-04-26 18:04:55.529799249 +0000 UTC m=+0.141536774 container create d796aef9d2ae6eaff534cd50ccd9706563816a9c099056c412270a26fe34abb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-26T18:04:56.067 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:55 smithi186 ceph-mon[32797]: daemon mds.c restarted 2024-04-26T18:04:56.067 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:55 smithi186 ceph-mon[32797]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T18:04:56.068 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:55 smithi186 ceph-mon[32797]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T18:04:56.068 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:55 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.186:6836/1162255559,v1:172.21.15.186:6837/1162255559] up:boot 2024-04-26T18:04:56.068 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:55 smithi186 ceph-mon[32797]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T18:04:56.068 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:55 smithi186 ceph-mon[32797]: fsmap cephfs:2/3 {1=i=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T18:04:56.068 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:55 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T18:04:56.068 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:55 smithi186 ceph-mon[32797]: fsmap cephfs:3/3 {0=b=up:replay,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:56.068 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:55 smithi186 podman[68785]: 2024-04-26 18:04:55.783089575 +0000 UTC m=+0.394827098 container init d796aef9d2ae6eaff534cd50ccd9706563816a9c099056c412270a26fe34abb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD) 2024-04-26T18:04:56.068 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:55 smithi186 podman[68785]: 2024-04-26 18:04:55.785897772 +0000 UTC m=+0.397635303 container start d796aef9d2ae6eaff534cd50ccd9706563816a9c099056c412270a26fe34abb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-26T18:04:56.068 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:55 smithi186 ceph-mds[68803]: -- 172.21.15.186:0/2196022347 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561cef241a40 con 0x561cef214800 2024-04-26T18:04:56.068 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:55 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[68799]: starting mds.c at 2024-04-26T18:04:56.068 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:55 smithi186 bash[68785]: d796aef9d2ae6eaff534cd50ccd9706563816a9c099056c412270a26fe34abb2 2024-04-26T18:04:56.069 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:04:55 smithi186 systemd[1]: Started Ceph mds.c for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T18:04:56.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:55 smithi097 ceph-mon[32419]: daemon mds.c restarted 2024-04-26T18:04:56.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:55 smithi097 ceph-mon[32419]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T18:04:56.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:55 smithi097 ceph-mon[32419]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T18:04:56.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:55 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.186:6836/1162255559,v1:172.21.15.186:6837/1162255559] up:boot 2024-04-26T18:04:56.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:55 smithi097 ceph-mon[32419]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T18:04:56.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:55 smithi097 ceph-mon[32419]: fsmap cephfs:2/3 {1=i=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T18:04:56.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:55 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T18:04:56.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:55 smithi097 ceph-mon[32419]: fsmap cephfs:3/3 {0=b=up:replay,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:56.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:55 smithi031 ceph-mon[29164]: daemon mds.c restarted 2024-04-26T18:04:56.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:55 smithi031 ceph-mon[29164]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T18:04:56.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:55 smithi031 ceph-mon[29164]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T18:04:56.289 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:55 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.186:6836/1162255559,v1:172.21.15.186:6837/1162255559] up:boot 2024-04-26T18:04:56.289 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:55 smithi031 ceph-mon[29164]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T18:04:56.289 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:55 smithi031 ceph-mon[29164]: fsmap cephfs:2/3 {1=i=up:active,2=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-26T18:04:56.289 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:55 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T18:04:56.289 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:55 smithi031 ceph-mon[29164]: fsmap cephfs:3/3 {0=b=up:replay,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:57.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:56 smithi097 ceph-mon[32419]: pgmap v262: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.5 MiB/s wr, 495 op/s 2024-04-26T18:04:57.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:56 smithi186 ceph-mon[32797]: pgmap v262: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.5 MiB/s wr, 495 op/s 2024-04-26T18:04:57.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:56 smithi031 ceph-mon[29164]: pgmap v262: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.5 MiB/s wr, 495 op/s 2024-04-26T18:04:58.097 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 455.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-26T18:04:58.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:57 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:resolve 2024-04-26T18:04:58.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:57 smithi186 ceph-mon[32797]: fsmap cephfs:3/3 {0=b=up:resolve,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:58.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:57 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:resolve 2024-04-26T18:04:58.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:57 smithi031 ceph-mon[29164]: fsmap cephfs:3/3 {0=b=up:resolve,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:58.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:57 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:resolve 2024-04-26T18:04:58.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:57 smithi097 ceph-mon[32419]: fsmap cephfs:3/3 {0=b=up:resolve,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:59.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:58 smithi186 ceph-mon[32797]: reconnect by client.15156 192.168.0.1:0/2547335269 after 0.000999983 2024-04-26T18:04:59.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:58 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:reconnect 2024-04-26T18:04:59.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:58 smithi186 ceph-mon[32797]: fsmap cephfs:3/3 {0=b=up:reconnect,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:59.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:58 smithi186 ceph-mon[32797]: reconnect by client.15330 172.21.15.31:0/1539336114 after 0.0149997 2024-04-26T18:04:59.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:58 smithi186 ceph-mon[32797]: pgmap v263: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.5 MiB/s wr, 377 op/s 2024-04-26T18:04:59.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:58 smithi031 ceph-mon[29164]: reconnect by client.15156 192.168.0.1:0/2547335269 after 0.000999983 2024-04-26T18:04:59.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:58 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:reconnect 2024-04-26T18:04:59.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:58 smithi031 ceph-mon[29164]: fsmap cephfs:3/3 {0=b=up:reconnect,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:59.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:58 smithi031 ceph-mon[29164]: reconnect by client.15330 172.21.15.31:0/1539336114 after 0.0149997 2024-04-26T18:04:59.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:58 smithi031 ceph-mon[29164]: pgmap v263: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.5 MiB/s wr, 377 op/s 2024-04-26T18:04:59.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:58 smithi097 ceph-mon[32419]: reconnect by client.15156 192.168.0.1:0/2547335269 after 0.000999983 2024-04-26T18:04:59.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:58 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:reconnect 2024-04-26T18:04:59.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:58 smithi097 ceph-mon[32419]: fsmap cephfs:3/3 {0=b=up:reconnect,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:04:59.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:58 smithi097 ceph-mon[32419]: reconnect by client.15330 172.21.15.31:0/1539336114 after 0.0149997 2024-04-26T18:04:59.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:58 smithi097 ceph-mon[32419]: pgmap v263: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.5 MiB/s wr, 377 op/s 2024-04-26T18:05:00.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:59 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:rejoin 2024-04-26T18:05:00.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:59 smithi186 ceph-mon[32797]: fsmap cephfs:3/3 {0=b=up:rejoin,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:05:00.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:04:59 smithi186 ceph-mon[32797]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T18:05:00.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:59 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:rejoin 2024-04-26T18:05:00.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:59 smithi031 ceph-mon[29164]: fsmap cephfs:3/3 {0=b=up:rejoin,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:05:00.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:04:59 smithi031 ceph-mon[29164]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T18:05:00.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:59 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:rejoin 2024-04-26T18:05:00.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:59 smithi097 ceph-mon[32419]: fsmap cephfs:3/3 {0=b=up:rejoin,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:05:00.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:04:59 smithi097 ceph-mon[32419]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T18:05:01.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:01.019+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:05:01.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:01.020+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T18:05:01.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:01.020+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:05:01.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:01.020+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:05:01.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:01.020+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-26T18:05:01.167 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:00 smithi186 ceph-mon[32797]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T18:05:01.167 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:00 smithi186 ceph-mon[32797]: Cluster is now healthy 2024-04-26T18:05:01.167 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:00 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:active 2024-04-26T18:05:01.167 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:00 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:05:01.168 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:00 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:05:01.168 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:00 smithi186 ceph-mon[32797]: pgmap v264: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.5 MiB/s wr, 377 op/s 2024-04-26T18:05:01.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:00 smithi031 ceph-mon[29164]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T18:05:01.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:00 smithi031 ceph-mon[29164]: Cluster is now healthy 2024-04-26T18:05:01.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:00 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:active 2024-04-26T18:05:01.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:00 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:05:01.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:00 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:05:01.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:00 smithi031 ceph-mon[29164]: pgmap v264: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.5 MiB/s wr, 377 op/s 2024-04-26T18:05:01.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:00 smithi097 ceph-mon[32419]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T18:05:01.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:00 smithi097 ceph-mon[32419]: Cluster is now healthy 2024-04-26T18:05:01.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:00 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] up:active 2024-04-26T18:05:01.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:00 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T18:05:01.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:00 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:05:01.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:00 smithi097 ceph-mon[32419]: pgmap v264: 97 pgs: 97 active+clean; 4.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.5 MiB/s wr, 377 op/s 2024-04-26T18:05:02.634 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:05:02 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[67844]: 2024-04-26T18:05:02.198+0000 7f71cf7b8700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T18:05:02.637 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:05:02 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[67844]: 2024-04-26T18:05:02.198+0000 7f71cf7b8700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T18:05:02.637 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:05:02 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[67844]: 2024-04-26T18:05:02.198+0000 7f71cf7b8700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T18:05:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 460.0 s, 1349.9 tps, lat 206.720 ms stddev 1305.949 2024-04-26T18:05:03.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:03 smithi186 ceph-mon[32797]: pgmap v265: 97 pgs: 97 active+clean; 4.2 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 20 MiB/s wr, 1.24k op/s 2024-04-26T18:05:03.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:03 smithi031 ceph-mon[29164]: pgmap v265: 97 pgs: 97 active+clean; 4.2 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 20 MiB/s wr, 1.24k op/s 2024-04-26T18:05:03.383 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:03 smithi097 ceph-mon[32419]: pgmap v265: 97 pgs: 97 active+clean; 4.2 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 20 MiB/s wr, 1.24k op/s 2024-04-26T18:05:04.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:04 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:05:04.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:04 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:05:04.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:04 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:05:05.285 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:05 smithi097 ceph-mon[32419]: pgmap v266: 97 pgs: 97 active+clean; 4.2 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 20 MiB/s wr, 1.24k op/s 2024-04-26T18:05:05.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:05 smithi031 ceph-mon[29164]: pgmap v266: 97 pgs: 97 active+clean; 4.2 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 20 MiB/s wr, 1.24k op/s 2024-04-26T18:05:05.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:05 smithi186 ceph-mon[32797]: pgmap v266: 97 pgs: 97 active+clean; 4.2 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 20 MiB/s wr, 1.24k op/s 2024-04-26T18:05:07.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:07 smithi031 ceph-mon[29164]: pgmap v267: 97 pgs: 97 active+clean; 4.2 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 22 MiB/s wr, 1.46k op/s 2024-04-26T18:05:07.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:07 smithi097 ceph-mon[32419]: pgmap v267: 97 pgs: 97 active+clean; 4.2 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 22 MiB/s wr, 1.46k op/s 2024-04-26T18:05:07.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:07 smithi186 ceph-mon[32797]: pgmap v267: 97 pgs: 97 active+clean; 4.2 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 22 MiB/s wr, 1.46k op/s 2024-04-26T18:05:08.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 465.0 s, 3203.8 tps, lat 31.233 ms stddev 36.067 2024-04-26T18:05:09.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:09 smithi031 ceph-mon[29164]: pgmap v268: 97 pgs: 97 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 23 MiB/s wr, 1.49k op/s 2024-04-26T18:05:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:09 smithi097 ceph-mon[32419]: pgmap v268: 97 pgs: 97 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 23 MiB/s wr, 1.49k op/s 2024-04-26T18:05:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:09 smithi186 ceph-mon[32797]: pgmap v268: 97 pgs: 97 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 23 MiB/s wr, 1.49k op/s 2024-04-26T18:05:09.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:09.820+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:05:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:09.820+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:05:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:09.820+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:05:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:09.820+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:05:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:09.820+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:05:11.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:11 smithi031 ceph-mon[29164]: pgmap v269: 97 pgs: 97 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 22 MiB/s wr, 1.47k op/s 2024-04-26T18:05:11.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:11 smithi097 ceph-mon[32419]: pgmap v269: 97 pgs: 97 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 22 MiB/s wr, 1.47k op/s 2024-04-26T18:05:11.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:11 smithi186 ceph-mon[32797]: pgmap v269: 97 pgs: 97 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 22 MiB/s wr, 1.47k op/s 2024-04-26T18:05:13.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 470.0 s, 2552.2 tps, lat 39.054 ms stddev 42.116 2024-04-26T18:05:13.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:13 smithi031 ceph-mon[29164]: pgmap v270: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 26 MiB/s wr, 1.77k op/s 2024-04-26T18:05:13.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:13 smithi097 ceph-mon[32419]: pgmap v270: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 26 MiB/s wr, 1.77k op/s 2024-04-26T18:05:13.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:13 smithi186 ceph-mon[32797]: pgmap v270: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 26 MiB/s wr, 1.77k op/s 2024-04-26T18:05:13.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:13.834+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:05:13.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:13.834+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T18:05:13.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:13.834+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:05:13.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:13.834+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:05:13.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:13.834+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:05:15.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:15 smithi031 ceph-mon[29164]: pgmap v271: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 10 MiB/s wr, 785 op/s 2024-04-26T18:05:15.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:15 smithi097 ceph-mon[32419]: pgmap v271: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 10 MiB/s wr, 785 op/s 2024-04-26T18:05:15.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:15 smithi186 ceph-mon[32797]: pgmap v271: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 10 MiB/s wr, 785 op/s 2024-04-26T18:05:17.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:17 smithi031 ceph-mon[29164]: pgmap v272: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 12 MiB/s wr, 908 op/s 2024-04-26T18:05:17.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:17 smithi097 ceph-mon[32419]: pgmap v272: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 12 MiB/s wr, 908 op/s 2024-04-26T18:05:17.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:17 smithi186 ceph-mon[32797]: pgmap v272: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 12 MiB/s wr, 908 op/s 2024-04-26T18:05:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 475.0 s, 1816.6 tps, lat 54.768 ms stddev 57.646 2024-04-26T18:05:19.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:19 smithi031 ceph-mon[29164]: pgmap v273: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 12 MiB/s wr, 945 op/s 2024-04-26T18:05:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:19 smithi097 ceph-mon[32419]: pgmap v273: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 12 MiB/s wr, 945 op/s 2024-04-26T18:05:19.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:19 smithi186 ceph-mon[32797]: pgmap v273: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 12 MiB/s wr, 945 op/s 2024-04-26T18:05:21.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:21 smithi031 ceph-mon[29164]: pgmap v274: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.0 MiB/s wr, 655 op/s 2024-04-26T18:05:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:21 smithi097 ceph-mon[32419]: pgmap v274: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.0 MiB/s wr, 655 op/s 2024-04-26T18:05:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:21 smithi186 ceph-mon[32797]: pgmap v274: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.0 MiB/s wr, 655 op/s 2024-04-26T18:05:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 480.0 s, 2438.6 tps, lat 41.344 ms stddev 46.979 2024-04-26T18:05:23.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:23 smithi031 ceph-mon[29164]: pgmap v275: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 924 op/s 2024-04-26T18:05:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:23 smithi097 ceph-mon[32419]: pgmap v275: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 924 op/s 2024-04-26T18:05:23.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:23 smithi186 ceph-mon[32797]: pgmap v275: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 924 op/s 2024-04-26T18:05:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:24.826+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:05:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:24.827+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:05:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:24.827+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:05:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:24.827+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:05:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:24.827+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:05:25.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:25 smithi031 ceph-mon[29164]: pgmap v276: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.7 MiB/s wr, 627 op/s 2024-04-26T18:05:25.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:25 smithi097 ceph-mon[32419]: pgmap v276: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.7 MiB/s wr, 627 op/s 2024-04-26T18:05:25.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:25 smithi186 ceph-mon[32797]: pgmap v276: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.7 MiB/s wr, 627 op/s 2024-04-26T18:05:27.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:27 smithi031 ceph-mon[29164]: pgmap v277: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.1 MiB/s wr, 769 op/s 2024-04-26T18:05:27.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:27 smithi097 ceph-mon[32419]: pgmap v277: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.1 MiB/s wr, 769 op/s 2024-04-26T18:05:27.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:27 smithi186 ceph-mon[32797]: pgmap v277: 97 pgs: 97 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.1 MiB/s wr, 769 op/s 2024-04-26T18:05:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 485.0 s, 2039.4 tps, lat 48.313 ms stddev 49.958 2024-04-26T18:05:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:28.848+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:05:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:28.849+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T18:05:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:28.849+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:05:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:28.849+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:05:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:28.849+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:05:29.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:29 smithi031 ceph-mon[29164]: pgmap v278: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.9 MiB/s wr, 772 op/s 2024-04-26T18:05:29.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:29 smithi097 ceph-mon[32419]: pgmap v278: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.9 MiB/s wr, 772 op/s 2024-04-26T18:05:29.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:29 smithi186 ceph-mon[32797]: pgmap v278: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.9 MiB/s wr, 772 op/s 2024-04-26T18:05:31.318 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:31 smithi186 ceph-mon[32797]: pgmap v279: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.1 MiB/s wr, 537 op/s 2024-04-26T18:05:31.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:31 smithi097 ceph-mon[32419]: pgmap v279: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.1 MiB/s wr, 537 op/s 2024-04-26T18:05:31.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:31 smithi031 ceph-mon[29164]: pgmap v279: 97 pgs: 97 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.1 MiB/s wr, 537 op/s 2024-04-26T18:05:33.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 490.0 s, 2080.0 tps, lat 48.573 ms stddev 49.595 2024-04-26T18:05:33.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:33 smithi097 ceph-mon[32419]: pgmap v280: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.5 MiB/s wr, 800 op/s 2024-04-26T18:05:33.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:33 smithi186 ceph-mon[32797]: pgmap v280: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.5 MiB/s wr, 800 op/s 2024-04-26T18:05:33.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:33 smithi031 ceph-mon[29164]: pgmap v280: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.5 MiB/s wr, 800 op/s 2024-04-26T18:05:35.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:35 smithi097 ceph-mon[32419]: pgmap v281: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.0 MiB/s wr, 531 op/s 2024-04-26T18:05:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:35 smithi186 ceph-mon[32797]: pgmap v281: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.0 MiB/s wr, 531 op/s 2024-04-26T18:05:35.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:35 smithi031 ceph-mon[29164]: pgmap v281: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.0 MiB/s wr, 531 op/s 2024-04-26T18:05:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:37 smithi097 ceph-mon[32419]: pgmap v282: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.4 MiB/s wr, 585 op/s 2024-04-26T18:05:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:37 smithi186 ceph-mon[32797]: pgmap v282: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.4 MiB/s wr, 585 op/s 2024-04-26T18:05:37.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:37 smithi031 ceph-mon[29164]: pgmap v282: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.4 MiB/s wr, 585 op/s 2024-04-26T18:05:38.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 495.0 s, 2571.6 tps, lat 38.837 ms stddev 42.216 2024-04-26T18:05:39.361 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:39 smithi031 ceph-mon[29164]: pgmap v283: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 661 op/s 2024-04-26T18:05:39.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:39 smithi097 ceph-mon[32419]: pgmap v283: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 661 op/s 2024-04-26T18:05:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:39 smithi186 ceph-mon[32797]: pgmap v283: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 661 op/s 2024-04-26T18:05:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:39.829+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:05:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:39.829+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:05:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:39.829+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:05:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:39.829+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:05:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:39.829+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:05:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:41 smithi097 ceph-mon[32419]: pgmap v284: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.0 MiB/s wr, 535 op/s 2024-04-26T18:05:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:41 smithi186 ceph-mon[32797]: pgmap v284: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.0 MiB/s wr, 535 op/s 2024-04-26T18:05:41.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:41 smithi031 ceph-mon[29164]: pgmap v284: 97 pgs: 97 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.0 MiB/s wr, 535 op/s 2024-04-26T18:05:43.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 500.0 s, 2186.2 tps, lat 45.936 ms stddev 52.423 2024-04-26T18:05:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:43 smithi097 ceph-mon[32419]: pgmap v285: 97 pgs: 97 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 13 MiB/s wr, 902 op/s 2024-04-26T18:05:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:43 smithi186 ceph-mon[32797]: pgmap v285: 97 pgs: 97 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 13 MiB/s wr, 902 op/s 2024-04-26T18:05:43.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:43 smithi031 ceph-mon[29164]: pgmap v285: 97 pgs: 97 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 13 MiB/s wr, 902 op/s 2024-04-26T18:05:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:43.864+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:05:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:43.864+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T18:05:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:43.864+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:05:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:43.864+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:05:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:43.864+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:05:45.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:45 smithi097 ceph-mon[32419]: pgmap v286: 97 pgs: 97 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 10 MiB/s wr, 638 op/s 2024-04-26T18:05:45.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:45 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:05:45.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:45 smithi186 ceph-mon[32797]: pgmap v286: 97 pgs: 97 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 10 MiB/s wr, 638 op/s 2024-04-26T18:05:45.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:45 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:05:45.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:45 smithi031 ceph-mon[29164]: pgmap v286: 97 pgs: 97 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 10 MiB/s wr, 638 op/s 2024-04-26T18:05:45.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:45 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:05:46.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:46 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:05:46.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:46 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:05:46.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:46 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:05:46.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:46 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:05:46.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:46 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:05:46.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:46 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:05:46.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:46 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:05:46.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:46 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:05:46.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:46 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:05:47.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:47 smithi097 ceph-mon[32419]: pgmap v287: 97 pgs: 97 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 14 MiB/s wr, 764 op/s 2024-04-26T18:05:47.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:47 smithi186 ceph-mon[32797]: pgmap v287: 97 pgs: 97 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 14 MiB/s wr, 764 op/s 2024-04-26T18:05:47.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:47 smithi031 ceph-mon[29164]: pgmap v287: 97 pgs: 97 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 14 MiB/s wr, 764 op/s 2024-04-26T18:05:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 505.0 s, 2669.4 tps, lat 37.457 ms stddev 42.253 2024-04-26T18:05:49.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:49 smithi097 ceph-mon[32419]: pgmap v288: 97 pgs: 97 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 21 MiB/s wr, 1.01k op/s 2024-04-26T18:05:49.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:49 smithi186 ceph-mon[32797]: pgmap v288: 97 pgs: 97 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 21 MiB/s wr, 1.01k op/s 2024-04-26T18:05:49.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:49 smithi031 ceph-mon[29164]: pgmap v288: 97 pgs: 97 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 21 MiB/s wr, 1.01k op/s 2024-04-26T18:05:51.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:51 smithi097 ceph-mon[32419]: pgmap v289: 97 pgs: 97 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 19 MiB/s wr, 792 op/s 2024-04-26T18:05:51.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:51 smithi186 ceph-mon[32797]: pgmap v289: 97 pgs: 97 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 19 MiB/s wr, 792 op/s 2024-04-26T18:05:51.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:51 smithi031 ceph-mon[29164]: pgmap v289: 97 pgs: 97 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 19 MiB/s wr, 792 op/s 2024-04-26T18:05:53.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 510.0 s, 2878.0 tps, lat 34.701 ms stddev 40.677 2024-04-26T18:05:53.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:53 smithi097 ceph-mon[32419]: pgmap v290: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 29 MiB/s wr, 1.10k op/s 2024-04-26T18:05:53.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:53 smithi186 ceph-mon[32797]: pgmap v290: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 29 MiB/s wr, 1.10k op/s 2024-04-26T18:05:53.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:53 smithi031 ceph-mon[29164]: pgmap v290: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 29 MiB/s wr, 1.10k op/s 2024-04-26T18:05:54.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:54.836+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:05:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:54.836+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:05:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:54.836+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:05:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:54.836+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:05:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:05:54.836+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:05:55.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:55 smithi097 ceph-mon[32419]: pgmap v291: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 21 MiB/s wr, 731 op/s 2024-04-26T18:05:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:55 smithi186 ceph-mon[32797]: pgmap v291: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 21 MiB/s wr, 731 op/s 2024-04-26T18:05:55.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:55 smithi031 ceph-mon[29164]: pgmap v291: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 21 MiB/s wr, 731 op/s 2024-04-26T18:05:57.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:57 smithi097 ceph-mon[32419]: pgmap v292: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 21 MiB/s wr, 768 op/s 2024-04-26T18:05:57.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:57 smithi186 ceph-mon[32797]: pgmap v292: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 21 MiB/s wr, 768 op/s 2024-04-26T18:05:57.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:57 smithi031 ceph-mon[29164]: pgmap v292: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 21 MiB/s wr, 768 op/s 2024-04-26T18:05:58.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 515.0 s, 2856.0 tps, lat 35.107 ms stddev 38.509 2024-04-26T18:05:58.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:58.879+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:05:58.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:58.880+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T18:05:58.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:58.880+0000 7f7ab15d7700 1 -- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 msgr2=0x564077112970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:05:58.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:58.880+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:05:58.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:05:58.880+0000 7f7ab15d7700 1 --2- 172.21.15.31:0/69685200 >> [v2:172.21.15.186:6836/1510454985,v1:172.21.15.186:6837/1510454985] conn(0x564077110580 0x564077112970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:05:59.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:05:59 smithi031 ceph-mon[29164]: pgmap v293: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 24 MiB/s wr, 899 op/s 2024-04-26T18:05:59.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:05:59 smithi097 ceph-mon[32419]: pgmap v293: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 24 MiB/s wr, 899 op/s 2024-04-26T18:05:59.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:05:59 smithi186 ceph-mon[32797]: pgmap v293: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 24 MiB/s wr, 899 op/s 2024-04-26T18:06:01.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:01 smithi097 ceph-mon[32419]: pgmap v294: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 MiB/s wr, 600 op/s 2024-04-26T18:06:01.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:01 smithi186 ceph-mon[32797]: pgmap v294: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 MiB/s wr, 600 op/s 2024-04-26T18:06:01.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:01 smithi031 ceph-mon[29164]: pgmap v294: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 MiB/s wr, 600 op/s 2024-04-26T18:06:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 520.0 s, 1620.0 tps, lat 58.106 ms stddev 122.654 2024-04-26T18:06:03.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:03 smithi097 ceph-mon[32419]: pgmap v295: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 24 MiB/s wr, 901 op/s 2024-04-26T18:06:03.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:03 smithi186 ceph-mon[32797]: pgmap v295: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 24 MiB/s wr, 901 op/s 2024-04-26T18:06:03.548 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:03 smithi031 ceph-mon[29164]: pgmap v295: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 24 MiB/s wr, 901 op/s 2024-04-26T18:06:03.962 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-26T18:06:03.963 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't parse response with error Expecting value: line 1 column 1 (char 0); rc: 124 (EMEDIUMTYPE); stdout: 2024-04-26T18:06:03.963 ERROR:tasks.quiescer.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/qa/tasks/quiescer.py", line 291, in do_quiesce response = json.loads(stdout) File "/usr/lib/python3.8/json/__init__.py", line 357, in loads return _default_decoder.decode(s) File "/usr/lib/python3.8/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/qa/tasks/quiescer.py", line 354, in _run self.do_quiesce(d) File "/home/teuthworker/src/git.ceph.com_ceph-c_bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/qa/tasks/quiescer.py", line 295, in do_quiesce raise RuntimeError(f"Error parsing quiesce response: {e}") RuntimeError: Error parsing quiesce response: Expecting value: line 1 column 1 (char 0) 2024-04-26T18:06:04.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:04 smithi031 ceph-mon[29164]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T18:06:04.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:04 smithi097 ceph-mon[32419]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T18:06:04.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:04 smithi186 ceph-mon[32797]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T18:06:05.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:05 smithi097 ceph-mon[32419]: pgmap v297: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 17 MiB/s wr, 714 op/s 2024-04-26T18:06:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:05 smithi186 ceph-mon[32797]: pgmap v297: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 17 MiB/s wr, 714 op/s 2024-04-26T18:06:05.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:05 smithi031 ceph-mon[29164]: pgmap v297: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 17 MiB/s wr, 714 op/s 2024-04-26T18:06:07.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:07 smithi097 ceph-mon[32419]: pgmap v298: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 18 MiB/s wr, 725 op/s 2024-04-26T18:06:07.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:07 smithi186 ceph-mon[32797]: pgmap v298: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 18 MiB/s wr, 725 op/s 2024-04-26T18:06:07.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:07 smithi031 ceph-mon[29164]: pgmap v298: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 18 MiB/s wr, 725 op/s 2024-04-26T18:06:08.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 525.0 s, 598.8 tps, lat 176.159 ms stddev 298.956 2024-04-26T18:06:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:09 smithi097 ceph-mon[32419]: pgmap v299: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 28 MiB/s wr, 1.21k op/s 2024-04-26T18:06:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:09 smithi097 ceph-mon[32419]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T18:06:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:09 smithi186 ceph-mon[32797]: pgmap v299: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 28 MiB/s wr, 1.21k op/s 2024-04-26T18:06:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:09 smithi186 ceph-mon[32797]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T18:06:09.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:09 smithi031 ceph-mon[29164]: pgmap v299: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 28 MiB/s wr, 1.21k op/s 2024-04-26T18:06:09.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:09 smithi031 ceph-mon[29164]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T18:06:09.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:09.841+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:06:09.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:09.841+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:06:09.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:09.841+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:06:09.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:09.841+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:06:09.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:09.841+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:06:11.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:11 smithi186 ceph-mon[32797]: pgmap v301: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 24 MiB/s wr, 1.07k op/s 2024-04-26T18:06:11.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:11 smithi031 ceph-mon[29164]: pgmap v301: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 24 MiB/s wr, 1.07k op/s 2024-04-26T18:06:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:11 smithi097 ceph-mon[32419]: pgmap v301: 97 pgs: 97 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 24 MiB/s wr, 1.07k op/s 2024-04-26T18:06:13.095 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 530.0 s, 2614.2 tps, lat 38.359 ms stddev 54.199 2024-04-26T18:06:13.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:13 smithi186 ceph-mon[32797]: pgmap v302: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 54 MiB/s wr, 2.44k op/s 2024-04-26T18:06:13.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:13 smithi031 ceph-mon[29164]: pgmap v302: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 54 MiB/s wr, 2.44k op/s 2024-04-26T18:06:13.638 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:13 smithi097 ceph-mon[32419]: pgmap v302: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 54 MiB/s wr, 2.44k op/s 2024-04-26T18:06:15.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:15 smithi186 ceph-mon[32797]: pgmap v303: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 48 MiB/s wr, 2.16k op/s 2024-04-26T18:06:15.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:15 smithi031 ceph-mon[29164]: pgmap v303: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 48 MiB/s wr, 2.16k op/s 2024-04-26T18:06:15.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:15 smithi097 ceph-mon[32419]: pgmap v303: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 48 MiB/s wr, 2.16k op/s 2024-04-26T18:06:17.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:17 smithi186 ceph-mon[32797]: pgmap v304: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 47 MiB/s wr, 2.12k op/s 2024-04-26T18:06:17.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:17 smithi031 ceph-mon[29164]: pgmap v304: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 47 MiB/s wr, 2.12k op/s 2024-04-26T18:06:17.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:17 smithi097 ceph-mon[32419]: pgmap v304: 97 pgs: 97 active+clean; 4.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 47 MiB/s wr, 2.12k op/s 2024-04-26T18:06:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 535.0 s, 2816.0 tps, lat 35.382 ms stddev 42.740 2024-04-26T18:06:18.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:18 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:06:18.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:18 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:06:18.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:18 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:06:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:19 smithi186 ceph-mon[32797]: pgmap v305: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 35 MiB/s wr, 1.72k op/s 2024-04-26T18:06:19.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:19 smithi031 ceph-mon[29164]: pgmap v305: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 35 MiB/s wr, 1.72k op/s 2024-04-26T18:06:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:19 smithi097 ceph-mon[32419]: pgmap v305: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 35 MiB/s wr, 1.72k op/s 2024-04-26T18:06:21.469 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:21 smithi031 ceph-mon[29164]: pgmap v306: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 31 MiB/s wr, 1.54k op/s 2024-04-26T18:06:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:21 smithi186 ceph-mon[32797]: pgmap v306: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 31 MiB/s wr, 1.54k op/s 2024-04-26T18:06:21.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:21 smithi097 ceph-mon[32419]: pgmap v306: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 31 MiB/s wr, 1.54k op/s 2024-04-26T18:06:23.097 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 540.0 s, 1552.0 tps, lat 28.407 ms stddev 33.545 2024-04-26T18:06:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:23 smithi186 ceph-mon[32797]: pgmap v307: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 34 MiB/s wr, 1.80k op/s 2024-04-26T18:06:23.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:23 smithi031 ceph-mon[29164]: pgmap v307: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 34 MiB/s wr, 1.80k op/s 2024-04-26T18:06:23.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:23 smithi097 ceph-mon[32419]: pgmap v307: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 34 MiB/s wr, 1.80k op/s 2024-04-26T18:06:24.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:24.848+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:06:24.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:24.849+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:06:24.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:24.849+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:06:24.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:24.849+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:06:24.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:24.849+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:06:25.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:25 smithi186 ceph-mon[32797]: pgmap v308: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 10 MiB/s wr, 707 op/s 2024-04-26T18:06:25.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:25 smithi031 ceph-mon[29164]: pgmap v308: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 10 MiB/s wr, 707 op/s 2024-04-26T18:06:25.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:25 smithi097 ceph-mon[32419]: pgmap v308: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 10 MiB/s wr, 707 op/s 2024-04-26T18:06:27.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:27 smithi186 ceph-mon[32797]: pgmap v309: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 11 MiB/s wr, 762 op/s 2024-04-26T18:06:27.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:27 smithi031 ceph-mon[29164]: pgmap v309: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 11 MiB/s wr, 762 op/s 2024-04-26T18:06:27.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:27 smithi097 ceph-mon[32419]: pgmap v309: 97 pgs: 97 active+clean; 4.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 11 MiB/s wr, 762 op/s 2024-04-26T18:06:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 545.0 s, 1463.1 tps, lat 106.736 ms stddev 605.316 2024-04-26T18:06:29.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:29 smithi186 ceph-mon[32797]: pgmap v310: 97 pgs: 97 active+clean; 4.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 23 MiB/s wr, 1.77k op/s 2024-04-26T18:06:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:29 smithi031 ceph-mon[29164]: pgmap v310: 97 pgs: 97 active+clean; 4.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 23 MiB/s wr, 1.77k op/s 2024-04-26T18:06:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:29 smithi097 ceph-mon[32419]: pgmap v310: 97 pgs: 97 active+clean; 4.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 23 MiB/s wr, 1.77k op/s 2024-04-26T18:06:31.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:31 smithi186 ceph-mon[32797]: pgmap v311: 97 pgs: 97 active+clean; 4.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 18 MiB/s wr, 1.44k op/s 2024-04-26T18:06:31.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:31 smithi031 ceph-mon[29164]: pgmap v311: 97 pgs: 97 active+clean; 4.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 18 MiB/s wr, 1.44k op/s 2024-04-26T18:06:31.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:31 smithi097 ceph-mon[32419]: pgmap v311: 97 pgs: 97 active+clean; 4.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 18 MiB/s wr, 1.44k op/s 2024-04-26T18:06:33.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 550.0 s, 2594.6 tps, lat 38.628 ms stddev 43.388 2024-04-26T18:06:33.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:33 smithi186 ceph-mon[32797]: pgmap v312: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 22 MiB/s wr, 1.74k op/s 2024-04-26T18:06:33.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:33 smithi031 ceph-mon[29164]: pgmap v312: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 22 MiB/s wr, 1.74k op/s 2024-04-26T18:06:33.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:33 smithi097 ceph-mon[32419]: pgmap v312: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 22 MiB/s wr, 1.74k op/s 2024-04-26T18:06:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:35 smithi186 ceph-mon[32797]: pgmap v313: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 17 MiB/s wr, 1.38k op/s 2024-04-26T18:06:35.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:35 smithi031 ceph-mon[29164]: pgmap v313: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 17 MiB/s wr, 1.38k op/s 2024-04-26T18:06:35.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:35 smithi097 ceph-mon[32419]: pgmap v313: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 17 MiB/s wr, 1.38k op/s 2024-04-26T18:06:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:37 smithi186 ceph-mon[32797]: pgmap v314: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 17 MiB/s wr, 1.38k op/s 2024-04-26T18:06:37.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:37 smithi031 ceph-mon[29164]: pgmap v314: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 17 MiB/s wr, 1.38k op/s 2024-04-26T18:06:37.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:37 smithi097 ceph-mon[32419]: pgmap v314: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 17 MiB/s wr, 1.38k op/s 2024-04-26T18:06:38.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 555.0 s, 2489.2 tps, lat 40.054 ms stddev 45.368 2024-04-26T18:06:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:39 smithi186 ceph-mon[32797]: pgmap v315: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 20 MiB/s wr, 1.60k op/s 2024-04-26T18:06:39.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:39 smithi031 ceph-mon[29164]: pgmap v315: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 20 MiB/s wr, 1.60k op/s 2024-04-26T18:06:39.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:39 smithi097 ceph-mon[32419]: pgmap v315: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 20 MiB/s wr, 1.60k op/s 2024-04-26T18:06:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:39.852+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:06:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:39.853+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:06:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:39.853+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:06:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:39.853+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:06:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:39.853+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:06:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:41 smithi186 ceph-mon[32797]: pgmap v316: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.7 MiB/s wr, 578 op/s 2024-04-26T18:06:41.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:41 smithi031 ceph-mon[29164]: pgmap v316: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.7 MiB/s wr, 578 op/s 2024-04-26T18:06:41.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:41 smithi097 ceph-mon[32419]: pgmap v316: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.7 MiB/s wr, 578 op/s 2024-04-26T18:06:43.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 560.0 s, 2640.4 tps, lat 37.396 ms stddev 43.190 2024-04-26T18:06:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:43 smithi186 ceph-mon[32797]: pgmap v317: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 870 op/s 2024-04-26T18:06:43.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:43 smithi031 ceph-mon[29164]: pgmap v317: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 870 op/s 2024-04-26T18:06:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:43 smithi097 ceph-mon[32419]: pgmap v317: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 870 op/s 2024-04-26T18:06:45.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:45 smithi186 ceph-mon[32797]: pgmap v318: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 576 op/s 2024-04-26T18:06:45.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:45 smithi031 ceph-mon[29164]: pgmap v318: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 576 op/s 2024-04-26T18:06:45.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:45 smithi097 ceph-mon[32419]: pgmap v318: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 576 op/s 2024-04-26T18:06:46.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:46 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:06:46.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:46 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:06:46.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:46 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:06:47.716 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:47 smithi097 ceph-mon[32419]: pgmap v319: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.2 MiB/s wr, 586 op/s 2024-04-26T18:06:47.716 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:47 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.716 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:47 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.716 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:47 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.717 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:47 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.717 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:47 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.717 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:47 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:47 smithi186 ceph-mon[32797]: pgmap v319: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.2 MiB/s wr, 586 op/s 2024-04-26T18:06:47.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:47 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:47 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:47 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:47 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:47 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:47 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:47 smithi031 ceph-mon[29164]: pgmap v319: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.2 MiB/s wr, 586 op/s 2024-04-26T18:06:47.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:47 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:47 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:47 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:47 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:47 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:47.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:47 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 565.0 s, 2153.4 tps, lat 46.847 ms stddev 53.382 2024-04-26T18:06:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: pgmap v320: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 MiB/s wr, 836 op/s 2024-04-26T18:06:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:49 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: pgmap v320: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 MiB/s wr, 836 op/s 2024-04-26T18:06:49.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:49 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: pgmap v320: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 MiB/s wr, 836 op/s 2024-04-26T18:06:49.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:49.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:49.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:49 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:50 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T18:06:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:50 smithi186 ceph-mon[32797]: Unable to set osd_memory_target on smithi186 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-26T18:06:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:50 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi097 to 176.2M 2024-04-26T18:06:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:50 smithi186 ceph-mon[32797]: Unable to set osd_memory_target on smithi097 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-04-26T18:06:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:50 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:50 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:50 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:50 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:50.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:50 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:06:50.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:50 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:06:50.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:50 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:50.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:50 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T18:06:50.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:50 smithi031 ceph-mon[29164]: Unable to set osd_memory_target on smithi186 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-26T18:06:50.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:50 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi097 to 176.2M 2024-04-26T18:06:50.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:50 smithi031 ceph-mon[29164]: Unable to set osd_memory_target on smithi097 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-04-26T18:06:50.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:50 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:50.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:50 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:50.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:50 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:50.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:50 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:50.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:50 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:06:50.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:50 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:06:50.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:50 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:50.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:50 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T18:06:50.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:50 smithi097 ceph-mon[32419]: Unable to set osd_memory_target on smithi186 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-26T18:06:50.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:50 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi097 to 176.2M 2024-04-26T18:06:50.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:50 smithi097 ceph-mon[32419]: Unable to set osd_memory_target on smithi097 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-04-26T18:06:50.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:50 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:50.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:50 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:50.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:50 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:50.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:50 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T18:06:50.886 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:50 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:06:50.887 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:50 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:06:50.887 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:50 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:06:51.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:51 smithi186 ceph-mon[32797]: pgmap v321: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.5 MiB/s wr, 561 op/s 2024-04-26T18:06:51.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:51 smithi186 ceph-mon[32797]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-26T18:06:51.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:51 smithi186 ceph-mon[32797]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T18:06:51.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:51 smithi031 ceph-mon[29164]: pgmap v321: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.5 MiB/s wr, 561 op/s 2024-04-26T18:06:51.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:51 smithi031 ceph-mon[29164]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-26T18:06:51.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:51 smithi031 ceph-mon[29164]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T18:06:51.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:51 smithi097 ceph-mon[32419]: pgmap v321: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.5 MiB/s wr, 561 op/s 2024-04-26T18:06:51.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:51 smithi097 ceph-mon[32419]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-26T18:06:51.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:51 smithi097 ceph-mon[32419]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T18:06:53.095 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 570.0 s, 2491.0 tps, lat 40.267 ms stddev 42.243 2024-04-26T18:06:53.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:53 smithi186 ceph-mon[32797]: pgmap v322: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 MiB/s wr, 895 op/s 2024-04-26T18:06:53.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:53 smithi031 ceph-mon[29164]: pgmap v322: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 MiB/s wr, 895 op/s 2024-04-26T18:06:53.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:53 smithi097 ceph-mon[32419]: pgmap v322: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 MiB/s wr, 895 op/s 2024-04-26T18:06:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:54.860+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:06:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:54.860+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:06:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:54.860+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:06:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:54.860+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:06:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:06:54.860+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:06:55.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:55 smithi186 ceph-mon[32797]: pgmap v323: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.6 MiB/s wr, 604 op/s 2024-04-26T18:06:55.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:55 smithi031 ceph-mon[29164]: pgmap v323: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.6 MiB/s wr, 604 op/s 2024-04-26T18:06:55.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:55 smithi097 ceph-mon[32419]: pgmap v323: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.6 MiB/s wr, 604 op/s 2024-04-26T18:06:57.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:57 smithi186 ceph-mon[32797]: pgmap v324: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.7 MiB/s wr, 614 op/s 2024-04-26T18:06:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:57 smithi031 ceph-mon[29164]: pgmap v324: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.7 MiB/s wr, 614 op/s 2024-04-26T18:06:57.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:57 smithi097 ceph-mon[32419]: pgmap v324: 97 pgs: 97 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.7 MiB/s wr, 614 op/s 2024-04-26T18:06:58.095 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 575.0 s, 1930.8 tps, lat 51.630 ms stddev 57.278 2024-04-26T18:06:59.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:06:59 smithi186 ceph-mon[32797]: pgmap v325: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.3 MiB/s wr, 858 op/s 2024-04-26T18:06:59.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:06:59 smithi031 ceph-mon[29164]: pgmap v325: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.3 MiB/s wr, 858 op/s 2024-04-26T18:06:59.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:06:59 smithi097 ceph-mon[32419]: pgmap v325: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.3 MiB/s wr, 858 op/s 2024-04-26T18:07:01.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:01 smithi186 ceph-mon[32797]: pgmap v326: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.4 MiB/s wr, 598 op/s 2024-04-26T18:07:01.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:01 smithi031 ceph-mon[29164]: pgmap v326: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.4 MiB/s wr, 598 op/s 2024-04-26T18:07:01.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:01 smithi097 ceph-mon[32419]: pgmap v326: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.4 MiB/s wr, 598 op/s 2024-04-26T18:07:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 580.0 s, 1515.6 tps, lat 62.651 ms stddev 114.334 2024-04-26T18:07:03.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:03 smithi186 ceph-mon[32797]: pgmap v327: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.4 MiB/s wr, 874 op/s 2024-04-26T18:07:03.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:03 smithi031 ceph-mon[29164]: pgmap v327: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.4 MiB/s wr, 874 op/s 2024-04-26T18:07:03.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:03 smithi097 ceph-mon[32419]: pgmap v327: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.4 MiB/s wr, 874 op/s 2024-04-26T18:07:05.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:05 smithi186 ceph-mon[32797]: pgmap v328: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.7 MiB/s wr, 540 op/s 2024-04-26T18:07:05.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:05 smithi031 ceph-mon[29164]: pgmap v328: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.7 MiB/s wr, 540 op/s 2024-04-26T18:07:05.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:05 smithi097 ceph-mon[32419]: pgmap v328: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.7 MiB/s wr, 540 op/s 2024-04-26T18:07:07.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:07 smithi186 ceph-mon[32797]: pgmap v329: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.7 MiB/s wr, 544 op/s 2024-04-26T18:07:07.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:07 smithi031 ceph-mon[29164]: pgmap v329: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.7 MiB/s wr, 544 op/s 2024-04-26T18:07:07.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:07 smithi097 ceph-mon[32419]: pgmap v329: 97 pgs: 97 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.7 MiB/s wr, 544 op/s 2024-04-26T18:07:08.095 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 585.0 s, 229.0 tps, lat 355.543 ms stddev 220.862 2024-04-26T18:07:08.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:08 smithi186 ceph-mon[32797]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T18:07:08.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:08 smithi031 ceph-mon[29164]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T18:07:08.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:08 smithi097 ceph-mon[32419]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T18:07:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:09 smithi186 ceph-mon[32797]: pgmap v331: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.5 MiB/s wr, 514 op/s 2024-04-26T18:07:09.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:09 smithi031 ceph-mon[29164]: pgmap v331: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.5 MiB/s wr, 514 op/s 2024-04-26T18:07:09.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:09.862+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:07:09.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:09.862+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:07:09.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:09.862+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:07:09.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:09.862+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:07:09.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:09.862+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:07:09.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:09 smithi097 ceph-mon[32419]: pgmap v331: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.5 MiB/s wr, 514 op/s 2024-04-26T18:07:11.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:11 smithi186 ceph-mon[32797]: pgmap v332: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.5 MiB/s wr, 514 op/s 2024-04-26T18:07:11.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:11 smithi031 ceph-mon[29164]: pgmap v332: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.5 MiB/s wr, 514 op/s 2024-04-26T18:07:11.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:11 smithi097 ceph-mon[32419]: pgmap v332: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.5 MiB/s wr, 514 op/s 2024-04-26T18:07:13.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 590.0 s, 2160.6 tps, lat 57.423 ms stddev 146.403 2024-04-26T18:07:13.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:13 smithi186 ceph-mon[32797]: pgmap v333: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 29 MiB/s wr, 1.76k op/s 2024-04-26T18:07:13.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:13 smithi031 ceph-mon[29164]: pgmap v333: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 29 MiB/s wr, 1.76k op/s 2024-04-26T18:07:13.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:13 smithi097 ceph-mon[32419]: pgmap v333: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 29 MiB/s wr, 1.76k op/s 2024-04-26T18:07:14.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:14.397+0000 7f75d17fa640 1 -- 172.21.15.31:0/865661195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f75b40034c0 con 0x7f75dc11c110 2024-04-26T18:07:14.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:14.398+0000 7f75d27fc640 1 -- 172.21.15.31:0/865661195 <== mon.0 v2:172.21.15.31:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f75b40034c0 con 0x7f75dc11c110 2024-04-26T18:07:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:14.414+0000 7f75bb7fe640 1 -- 172.21.15.31:0/164021214 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f75ac0034c0 con 0x556108f4ff00 2024-04-26T18:07:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:14.414+0000 7f75c4ff9640 1 -- 172.21.15.31:0/164021214 <== mon.1 v2:172.21.15.186:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f75ac0034c0 con 0x556108f4ff00 2024-04-26T18:07:15.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:14 smithi031 ceph-mon[29164]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T18:07:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:14 smithi031 ceph-mon[29164]: pgmap v335: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 36 MiB/s wr, 2.19k op/s 2024-04-26T18:07:15.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:14 smithi097 ceph-mon[32419]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T18:07:15.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:14 smithi097 ceph-mon[32419]: pgmap v335: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 36 MiB/s wr, 2.19k op/s 2024-04-26T18:07:15.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:14 smithi186 ceph-mon[32797]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T18:07:15.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:14 smithi186 ceph-mon[32797]: pgmap v335: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 36 MiB/s wr, 2.19k op/s 2024-04-26T18:07:17.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:17 smithi031 ceph-mon[29164]: pgmap v336: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 33 MiB/s wr, 1.86k op/s 2024-04-26T18:07:17.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:17 smithi097 ceph-mon[32419]: pgmap v336: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 33 MiB/s wr, 1.86k op/s 2024-04-26T18:07:17.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:17 smithi186 ceph-mon[32797]: pgmap v336: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 33 MiB/s wr, 1.86k op/s 2024-04-26T18:07:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 595.0 s, 2717.6 tps, lat 36.656 ms stddev 42.987 2024-04-26T18:07:19.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:19 smithi031 ceph-mon[29164]: pgmap v337: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 31 MiB/s wr, 1.92k op/s 2024-04-26T18:07:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:19 smithi097 ceph-mon[32419]: pgmap v337: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 31 MiB/s wr, 1.92k op/s 2024-04-26T18:07:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:19 smithi186 ceph-mon[32797]: pgmap v337: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 31 MiB/s wr, 1.92k op/s 2024-04-26T18:07:21.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:21 smithi031 ceph-mon[29164]: pgmap v338: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 31 MiB/s wr, 1.92k op/s 2024-04-26T18:07:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:21 smithi097 ceph-mon[32419]: pgmap v338: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 31 MiB/s wr, 1.92k op/s 2024-04-26T18:07:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:21 smithi186 ceph-mon[32797]: pgmap v338: 97 pgs: 97 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 31 MiB/s wr, 1.92k op/s 2024-04-26T18:07:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 600.0 s, 2637.2 tps, lat 37.883 ms stddev 40.081 2024-04-26T18:07:23.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:23 smithi031 ceph-mon[29164]: pgmap v339: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.3 MiB/s wr, 807 op/s 2024-04-26T18:07:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:23 smithi097 ceph-mon[32419]: pgmap v339: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.3 MiB/s wr, 807 op/s 2024-04-26T18:07:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:23 smithi186 ceph-mon[32797]: pgmap v339: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.3 MiB/s wr, 807 op/s 2024-04-26T18:07:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:24.868+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:07:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:24.868+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:07:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:24.868+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:07:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:24.868+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:07:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:24.868+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:07:25.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:25 smithi031 ceph-mon[29164]: pgmap v340: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 8.1 MiB/s wr, 790 op/s 2024-04-26T18:07:25.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:25 smithi097 ceph-mon[32419]: pgmap v340: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 8.1 MiB/s wr, 790 op/s 2024-04-26T18:07:25.516 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:25 smithi186 ceph-mon[32797]: pgmap v340: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 8.1 MiB/s wr, 790 op/s 2024-04-26T18:07:27.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:27 smithi031 ceph-mon[29164]: pgmap v341: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.6 MiB/s wr, 719 op/s 2024-04-26T18:07:27.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:27 smithi097 ceph-mon[32419]: pgmap v341: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.6 MiB/s wr, 719 op/s 2024-04-26T18:07:27.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:27 smithi186 ceph-mon[32797]: pgmap v341: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.6 MiB/s wr, 719 op/s 2024-04-26T18:07:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 605.0 s, 2799.6 tps, lat 35.838 ms stddev 38.624 2024-04-26T18:07:29.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:29 smithi031 ceph-mon[29164]: pgmap v342: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.8 MiB/s wr, 949 op/s 2024-04-26T18:07:29.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:29 smithi097 ceph-mon[32419]: pgmap v342: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.8 MiB/s wr, 949 op/s 2024-04-26T18:07:29.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:29 smithi186 ceph-mon[32797]: pgmap v342: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.8 MiB/s wr, 949 op/s 2024-04-26T18:07:31.319 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:31 smithi186 ceph-mon[32797]: pgmap v343: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.8 MiB/s wr, 671 op/s 2024-04-26T18:07:31.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:31 smithi097 ceph-mon[32419]: pgmap v343: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.8 MiB/s wr, 671 op/s 2024-04-26T18:07:31.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:31 smithi031 ceph-mon[29164]: pgmap v343: 97 pgs: 97 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.8 MiB/s wr, 671 op/s 2024-04-26T18:07:33.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 610.0 s, 1426.0 tps, lat 70.201 ms stddev 135.016 2024-04-26T18:07:33.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:33 smithi097 ceph-mon[32419]: pgmap v344: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 MiB/s wr, 2.02k op/s 2024-04-26T18:07:33.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:33 smithi186 ceph-mon[32797]: pgmap v344: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 MiB/s wr, 2.02k op/s 2024-04-26T18:07:33.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:33 smithi031 ceph-mon[29164]: pgmap v344: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 MiB/s wr, 2.02k op/s 2024-04-26T18:07:35.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:35 smithi097 ceph-mon[32419]: pgmap v345: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 24 MiB/s wr, 1.63k op/s 2024-04-26T18:07:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:35 smithi186 ceph-mon[32797]: pgmap v345: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 24 MiB/s wr, 1.63k op/s 2024-04-26T18:07:35.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:35 smithi031 ceph-mon[29164]: pgmap v345: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 24 MiB/s wr, 1.63k op/s 2024-04-26T18:07:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:37 smithi097 ceph-mon[32419]: pgmap v346: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 25 MiB/s wr, 1.66k op/s 2024-04-26T18:07:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:37 smithi186 ceph-mon[32797]: pgmap v346: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 25 MiB/s wr, 1.66k op/s 2024-04-26T18:07:37.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:37 smithi031 ceph-mon[29164]: pgmap v346: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 25 MiB/s wr, 1.66k op/s 2024-04-26T18:07:38.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 615.0 s, 2122.6 tps, lat 46.278 ms stddev 51.745 2024-04-26T18:07:39.362 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:39 smithi031 ceph-mon[29164]: pgmap v347: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 26 MiB/s wr, 1.88k op/s 2024-04-26T18:07:39.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:39 smithi097 ceph-mon[32419]: pgmap v347: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 26 MiB/s wr, 1.88k op/s 2024-04-26T18:07:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:39 smithi186 ceph-mon[32797]: pgmap v347: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 26 MiB/s wr, 1.88k op/s 2024-04-26T18:07:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:39.871+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:07:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:39.872+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:07:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:39.872+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:07:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:39.872+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:07:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:39.872+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:07:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:41 smithi097 ceph-mon[32419]: pgmap v348: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 24 MiB/s wr, 1.65k op/s 2024-04-26T18:07:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:41 smithi186 ceph-mon[32797]: pgmap v348: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 24 MiB/s wr, 1.65k op/s 2024-04-26T18:07:41.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:41 smithi031 ceph-mon[29164]: pgmap v348: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 24 MiB/s wr, 1.65k op/s 2024-04-26T18:07:43.095 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 620.0 s, 1908.4 tps, lat 53.327 ms stddev 58.011 2024-04-26T18:07:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:43 smithi097 ceph-mon[32419]: pgmap v349: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 MiB/s wr, 1.92k op/s 2024-04-26T18:07:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:43 smithi186 ceph-mon[32797]: pgmap v349: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 MiB/s wr, 1.92k op/s 2024-04-26T18:07:43.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:43 smithi031 ceph-mon[29164]: pgmap v349: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 MiB/s wr, 1.92k op/s 2024-04-26T18:07:45.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:45 smithi097 ceph-mon[32419]: pgmap v350: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.4 MiB/s wr, 570 op/s 2024-04-26T18:07:45.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:45 smithi186 ceph-mon[32797]: pgmap v350: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.4 MiB/s wr, 570 op/s 2024-04-26T18:07:45.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:45 smithi031 ceph-mon[29164]: pgmap v350: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.4 MiB/s wr, 570 op/s 2024-04-26T18:07:47.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:47 smithi097 ceph-mon[32419]: pgmap v351: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.9 MiB/s wr, 605 op/s 2024-04-26T18:07:47.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:47 smithi186 ceph-mon[32797]: pgmap v351: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.9 MiB/s wr, 605 op/s 2024-04-26T18:07:47.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:47 smithi031 ceph-mon[29164]: pgmap v351: 97 pgs: 97 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.9 MiB/s wr, 605 op/s 2024-04-26T18:07:48.095 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 625.0 s, 2331.8 tps, lat 42.558 ms stddev 47.122 2024-04-26T18:07:49.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:49 smithi097 ceph-mon[32419]: pgmap v352: 97 pgs: 97 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.8 MiB/s wr, 804 op/s 2024-04-26T18:07:49.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:49 smithi186 ceph-mon[32797]: pgmap v352: 97 pgs: 97 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.8 MiB/s wr, 804 op/s 2024-04-26T18:07:49.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:49 smithi031 ceph-mon[29164]: pgmap v352: 97 pgs: 97 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.8 MiB/s wr, 804 op/s 2024-04-26T18:07:51.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:51 smithi097 ceph-mon[32419]: pgmap v353: 97 pgs: 97 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.5 MiB/s wr, 538 op/s 2024-04-26T18:07:51.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:51 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:07:51.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:51 smithi186 ceph-mon[32797]: pgmap v353: 97 pgs: 97 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.5 MiB/s wr, 538 op/s 2024-04-26T18:07:51.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:51 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:07:51.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:51 smithi031 ceph-mon[29164]: pgmap v353: 97 pgs: 97 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.5 MiB/s wr, 538 op/s 2024-04-26T18:07:51.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:51 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:07:52.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:52 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:07:52.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:52 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:07:52.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:52 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:07:52.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:52 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:07:52.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:52 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:07:52.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:52 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:07:52.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:52 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:07:52.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:52 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:07:52.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:52 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:07:53.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 630.0 s, 1627.0 tps, lat 62.060 ms stddev 68.573 2024-04-26T18:07:53.520 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:53 smithi186 ceph-mon[32797]: pgmap v354: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.7 MiB/s wr, 871 op/s 2024-04-26T18:07:53.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:53 smithi031 ceph-mon[29164]: pgmap v354: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.7 MiB/s wr, 871 op/s 2024-04-26T18:07:53.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:53 smithi097 ceph-mon[32419]: pgmap v354: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.7 MiB/s wr, 871 op/s 2024-04-26T18:07:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:54.880+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:07:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:54.880+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:07:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:54.880+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:07:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:54.880+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:07:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:07:54.880+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:07:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:55 smithi186 ceph-mon[32797]: pgmap v355: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.1 MiB/s wr, 601 op/s 2024-04-26T18:07:55.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:55 smithi031 ceph-mon[29164]: pgmap v355: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.1 MiB/s wr, 601 op/s 2024-04-26T18:07:55.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:55 smithi097 ceph-mon[32419]: pgmap v355: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.1 MiB/s wr, 601 op/s 2024-04-26T18:07:57.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:57 smithi186 ceph-mon[32797]: pgmap v356: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.1 MiB/s wr, 612 op/s 2024-04-26T18:07:57.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:57 smithi031 ceph-mon[29164]: pgmap v356: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.1 MiB/s wr, 612 op/s 2024-04-26T18:07:57.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:57 smithi097 ceph-mon[32419]: pgmap v356: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.1 MiB/s wr, 612 op/s 2024-04-26T18:07:58.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 635.0 s, 2561.0 tps, lat 39.012 ms stddev 42.993 2024-04-26T18:07:59.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:07:59 smithi186 ceph-mon[32797]: pgmap v357: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.1 MiB/s wr, 841 op/s 2024-04-26T18:07:59.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:07:59 smithi031 ceph-mon[29164]: pgmap v357: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.1 MiB/s wr, 841 op/s 2024-04-26T18:07:59.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:07:59 smithi097 ceph-mon[32419]: pgmap v357: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.1 MiB/s wr, 841 op/s 2024-04-26T18:08:01.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:01 smithi031 ceph-mon[29164]: pgmap v358: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.8 MiB/s wr, 606 op/s 2024-04-26T18:08:01.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:01 smithi097 ceph-mon[32419]: pgmap v358: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.8 MiB/s wr, 606 op/s 2024-04-26T18:08:01.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:01 smithi186 ceph-mon[32797]: pgmap v358: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.8 MiB/s wr, 606 op/s 2024-04-26T18:08:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 640.0 s, 1345.4 tps, lat 61.539 ms stddev 103.742 2024-04-26T18:08:03.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:03 smithi031 ceph-mon[29164]: pgmap v359: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 8.2 MiB/s wr, 878 op/s 2024-04-26T18:08:03.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:03 smithi097 ceph-mon[32419]: pgmap v359: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 8.2 MiB/s wr, 878 op/s 2024-04-26T18:08:03.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:03 smithi186 ceph-mon[32797]: pgmap v359: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 8.2 MiB/s wr, 878 op/s 2024-04-26T18:08:05.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:05 smithi031 ceph-mon[29164]: pgmap v360: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 5.0 MiB/s wr, 546 op/s 2024-04-26T18:08:05.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:05 smithi097 ceph-mon[32419]: pgmap v360: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 5.0 MiB/s wr, 546 op/s 2024-04-26T18:08:05.766 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:05 smithi186 ceph-mon[32797]: pgmap v360: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 5.0 MiB/s wr, 546 op/s 2024-04-26T18:08:07.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:07 smithi031 ceph-mon[29164]: pgmap v361: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.0 MiB/s wr, 609 op/s 2024-04-26T18:08:07.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:07 smithi097 ceph-mon[32419]: pgmap v361: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.0 MiB/s wr, 609 op/s 2024-04-26T18:08:07.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:07 smithi186 ceph-mon[32797]: pgmap v361: 97 pgs: 97 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.0 MiB/s wr, 609 op/s 2024-04-26T18:08:08.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 645.0 s, 1978.6 tps, lat 57.718 ms stddev 125.303 2024-04-26T18:08:08.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:08 smithi031 ceph-mon[29164]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T18:08:08.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:08 smithi097 ceph-mon[32419]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T18:08:08.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:08 smithi186 ceph-mon[32797]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T18:08:09.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:09 smithi031 ceph-mon[29164]: pgmap v363: 97 pgs: 97 active+clean; 5.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 27 MiB/s wr, 1.96k op/s 2024-04-26T18:08:09.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:09 smithi097 ceph-mon[32419]: pgmap v363: 97 pgs: 97 active+clean; 5.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 27 MiB/s wr, 1.96k op/s 2024-04-26T18:08:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:09 smithi186 ceph-mon[32797]: pgmap v363: 97 pgs: 97 active+clean; 5.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 27 MiB/s wr, 1.96k op/s 2024-04-26T18:08:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:09.882+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:08:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:09.882+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:08:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:09.882+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:08:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:09.882+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:08:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:09.882+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:08:11.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:11 smithi031 ceph-mon[29164]: pgmap v364: 97 pgs: 97 active+clean; 5.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 27 MiB/s wr, 1.96k op/s 2024-04-26T18:08:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:11 smithi097 ceph-mon[32419]: pgmap v364: 97 pgs: 97 active+clean; 5.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 27 MiB/s wr, 1.96k op/s 2024-04-26T18:08:11.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:11 smithi186 ceph-mon[32797]: pgmap v364: 97 pgs: 97 active+clean; 5.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 27 MiB/s wr, 1.96k op/s 2024-04-26T18:08:13.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 650.0 s, 2111.4 tps, lat 48.464 ms stddev 61.729 2024-04-26T18:08:13.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:13 smithi031 ceph-mon[29164]: pgmap v365: 97 pgs: 97 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 28 MiB/s wr, 1.95k op/s 2024-04-26T18:08:13.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:13 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:08:13.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:13 smithi097 ceph-mon[32419]: pgmap v365: 97 pgs: 97 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 28 MiB/s wr, 1.95k op/s 2024-04-26T18:08:13.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:13 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:08:13.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:13 smithi186 ceph-mon[32797]: pgmap v365: 97 pgs: 97 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 28 MiB/s wr, 1.95k op/s 2024-04-26T18:08:13.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:13 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:08:15.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:14 smithi031 ceph-mon[29164]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T18:08:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:14 smithi031 ceph-mon[29164]: pgmap v367: 97 pgs: 97 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 33 MiB/s wr, 2.34k op/s 2024-04-26T18:08:15.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:14 smithi097 ceph-mon[32419]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T18:08:15.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:14 smithi097 ceph-mon[32419]: pgmap v367: 97 pgs: 97 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 33 MiB/s wr, 2.34k op/s 2024-04-26T18:08:15.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:14 smithi186 ceph-mon[32797]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T18:08:15.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:14 smithi186 ceph-mon[32797]: pgmap v367: 97 pgs: 97 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 33 MiB/s wr, 2.34k op/s 2024-04-26T18:08:17.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:17 smithi097 ceph-mon[32419]: pgmap v368: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.0 MiB/s wr, 373 op/s 2024-04-26T18:08:17.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:17 smithi186 ceph-mon[32797]: pgmap v368: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.0 MiB/s wr, 373 op/s 2024-04-26T18:08:17.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:17 smithi031 ceph-mon[29164]: pgmap v368: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.0 MiB/s wr, 373 op/s 2024-04-26T18:08:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 655.0 s, 2009.4 tps, lat 49.717 ms stddev 54.801 2024-04-26T18:08:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:19 smithi097 ceph-mon[32419]: pgmap v369: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.5 MiB/s wr, 626 op/s 2024-04-26T18:08:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:19 smithi186 ceph-mon[32797]: pgmap v369: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.5 MiB/s wr, 626 op/s 2024-04-26T18:08:19.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:19 smithi031 ceph-mon[29164]: pgmap v369: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.5 MiB/s wr, 626 op/s 2024-04-26T18:08:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:21 smithi097 ceph-mon[32419]: pgmap v370: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.5 MiB/s wr, 626 op/s 2024-04-26T18:08:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:21 smithi186 ceph-mon[32797]: pgmap v370: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.5 MiB/s wr, 626 op/s 2024-04-26T18:08:21.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:21 smithi031 ceph-mon[29164]: pgmap v370: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.5 MiB/s wr, 626 op/s 2024-04-26T18:08:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 660.0 s, 2076.8 tps, lat 48.369 ms stddev 51.430 2024-04-26T18:08:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:23 smithi097 ceph-mon[32419]: pgmap v371: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.3 MiB/s wr, 702 op/s 2024-04-26T18:08:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:23 smithi186 ceph-mon[32797]: pgmap v371: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.3 MiB/s wr, 702 op/s 2024-04-26T18:08:23.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:23 smithi031 ceph-mon[29164]: pgmap v371: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.3 MiB/s wr, 702 op/s 2024-04-26T18:08:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:24.889+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:08:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:24.889+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:08:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:24.889+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:08:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:24.889+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:08:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:24.889+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:08:25.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:25 smithi097 ceph-mon[32419]: pgmap v372: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.1 MiB/s wr, 687 op/s 2024-04-26T18:08:25.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:25 smithi186 ceph-mon[32797]: pgmap v372: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.1 MiB/s wr, 687 op/s 2024-04-26T18:08:25.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:25 smithi031 ceph-mon[29164]: pgmap v372: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.1 MiB/s wr, 687 op/s 2024-04-26T18:08:27.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:27 smithi097 ceph-mon[32419]: pgmap v373: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.2 MiB/s wr, 597 op/s 2024-04-26T18:08:27.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:27 smithi186 ceph-mon[32797]: pgmap v373: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.2 MiB/s wr, 597 op/s 2024-04-26T18:08:27.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:27 smithi031 ceph-mon[29164]: pgmap v373: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.2 MiB/s wr, 597 op/s 2024-04-26T18:08:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 665.0 s, 1692.2 tps, lat 58.714 ms stddev 64.017 2024-04-26T18:08:29.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:29 smithi097 ceph-mon[32419]: pgmap v374: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 9.7 MiB/s wr, 928 op/s 2024-04-26T18:08:29.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:29 smithi186 ceph-mon[32797]: pgmap v374: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 9.7 MiB/s wr, 928 op/s 2024-04-26T18:08:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:29 smithi031 ceph-mon[29164]: pgmap v374: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 9.7 MiB/s wr, 928 op/s 2024-04-26T18:08:31.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:31 smithi097 ceph-mon[32419]: pgmap v375: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.2 MiB/s wr, 677 op/s 2024-04-26T18:08:31.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:31 smithi186 ceph-mon[32797]: pgmap v375: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.2 MiB/s wr, 677 op/s 2024-04-26T18:08:31.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:31 smithi031 ceph-mon[29164]: pgmap v375: 97 pgs: 97 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.2 MiB/s wr, 677 op/s 2024-04-26T18:08:33.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 670.0 s, 1732.4 tps, lat 57.867 ms stddev 95.241 2024-04-26T18:08:33.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:33 smithi097 ceph-mon[32419]: pgmap v376: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 32 MiB/s wr, 1.85k op/s 2024-04-26T18:08:33.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:33 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:08:33.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:33 smithi186 ceph-mon[32797]: pgmap v376: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 32 MiB/s wr, 1.85k op/s 2024-04-26T18:08:33.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:33 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:08:33.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:33 smithi031 ceph-mon[29164]: pgmap v376: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 32 MiB/s wr, 1.85k op/s 2024-04-26T18:08:33.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:33 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:08:35.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:35 smithi097 ceph-mon[32419]: pgmap v377: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 MiB/s wr, 1.52k op/s 2024-04-26T18:08:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:35 smithi186 ceph-mon[32797]: pgmap v377: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 MiB/s wr, 1.52k op/s 2024-04-26T18:08:35.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:35 smithi031 ceph-mon[29164]: pgmap v377: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 MiB/s wr, 1.52k op/s 2024-04-26T18:08:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:37 smithi097 ceph-mon[32419]: pgmap v378: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 MiB/s wr, 1.54k op/s 2024-04-26T18:08:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:37 smithi186 ceph-mon[32797]: pgmap v378: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 MiB/s wr, 1.54k op/s 2024-04-26T18:08:37.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:37 smithi031 ceph-mon[29164]: pgmap v378: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 MiB/s wr, 1.54k op/s 2024-04-26T18:08:38.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 675.0 s, 1420.0 tps, lat 70.297 ms stddev 77.465 2024-04-26T18:08:39.362 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:39 smithi031 ceph-mon[29164]: pgmap v379: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 31 MiB/s wr, 1.80k op/s 2024-04-26T18:08:39.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:39 smithi097 ceph-mon[32419]: pgmap v379: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 31 MiB/s wr, 1.80k op/s 2024-04-26T18:08:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:39 smithi186 ceph-mon[32797]: pgmap v379: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 31 MiB/s wr, 1.80k op/s 2024-04-26T18:08:39.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:39.890+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:08:39.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:39.890+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:08:39.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:39.890+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:08:39.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:39.890+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:08:39.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:39.890+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:08:41.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:41 smithi097 ceph-mon[32419]: pgmap v380: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 28 MiB/s wr, 1.46k op/s 2024-04-26T18:08:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:41 smithi186 ceph-mon[32797]: pgmap v380: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 28 MiB/s wr, 1.46k op/s 2024-04-26T18:08:41.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:41 smithi031 ceph-mon[29164]: pgmap v380: 97 pgs: 97 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 28 MiB/s wr, 1.46k op/s 2024-04-26T18:08:43.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 680.0 s, 1969.4 tps, lat 51.163 ms stddev 55.380 2024-04-26T18:08:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:43 smithi097 ceph-mon[32419]: pgmap v381: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 MiB/s wr, 1.72k op/s 2024-04-26T18:08:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:43 smithi186 ceph-mon[32797]: pgmap v381: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 MiB/s wr, 1.72k op/s 2024-04-26T18:08:43.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:43 smithi031 ceph-mon[29164]: pgmap v381: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 MiB/s wr, 1.72k op/s 2024-04-26T18:08:45.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:45 smithi097 ceph-mon[32419]: pgmap v382: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.9 MiB/s wr, 543 op/s 2024-04-26T18:08:45.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:45 smithi186 ceph-mon[32797]: pgmap v382: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.9 MiB/s wr, 543 op/s 2024-04-26T18:08:45.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:45 smithi031 ceph-mon[29164]: pgmap v382: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.9 MiB/s wr, 543 op/s 2024-04-26T18:08:47.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:47 smithi097 ceph-mon[32419]: pgmap v383: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.0 MiB/s wr, 551 op/s 2024-04-26T18:08:47.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:47 smithi186 ceph-mon[32797]: pgmap v383: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.0 MiB/s wr, 551 op/s 2024-04-26T18:08:47.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:47 smithi031 ceph-mon[29164]: pgmap v383: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.0 MiB/s wr, 551 op/s 2024-04-26T18:08:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 685.0 s, 2660.8 tps, lat 37.582 ms stddev 38.867 2024-04-26T18:08:49.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:49 smithi097 ceph-mon[32419]: pgmap v384: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.5 MiB/s wr, 807 op/s 2024-04-26T18:08:49.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:49 smithi186 ceph-mon[32797]: pgmap v384: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.5 MiB/s wr, 807 op/s 2024-04-26T18:08:49.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:49 smithi031 ceph-mon[29164]: pgmap v384: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.5 MiB/s wr, 807 op/s 2024-04-26T18:08:51.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:51 smithi097 ceph-mon[32419]: pgmap v385: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.0 MiB/s wr, 530 op/s 2024-04-26T18:08:51.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:51 smithi186 ceph-mon[32797]: pgmap v385: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.0 MiB/s wr, 530 op/s 2024-04-26T18:08:51.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:51 smithi031 ceph-mon[29164]: pgmap v385: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.0 MiB/s wr, 530 op/s 2024-04-26T18:08:52.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:52 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:08:52.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:52 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:08:52.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:52 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:08:53.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 690.0 s, 2125.0 tps, lat 46.786 ms stddev 50.368 2024-04-26T18:08:53.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:53 smithi186 ceph-mon[32797]: pgmap v386: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.8 MiB/s wr, 826 op/s 2024-04-26T18:08:53.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:53 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:08:53.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:53 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:08:53.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:53 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:08:53.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:53 smithi031 ceph-mon[29164]: pgmap v386: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.8 MiB/s wr, 826 op/s 2024-04-26T18:08:53.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:53 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:08:53.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:53 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:08:53.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:53 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:08:53.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:53 smithi097 ceph-mon[32419]: pgmap v386: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.8 MiB/s wr, 826 op/s 2024-04-26T18:08:53.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:53 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:08:53.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:53 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:08:53.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:53 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:08:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:54.900+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:08:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:54.901+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:08:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:54.901+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:08:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:54.901+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:08:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:08:54.901+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:08:55.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:54 smithi031 ceph-mon[29164]: pgmap v387: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.5 MiB/s wr, 571 op/s 2024-04-26T18:08:55.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:54 smithi097 ceph-mon[32419]: pgmap v387: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.5 MiB/s wr, 571 op/s 2024-04-26T18:08:55.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:54 smithi186 ceph-mon[32797]: pgmap v387: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.5 MiB/s wr, 571 op/s 2024-04-26T18:08:57.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:57 smithi097 ceph-mon[32419]: pgmap v388: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.2 MiB/s wr, 612 op/s 2024-04-26T18:08:57.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:57 smithi186 ceph-mon[32797]: pgmap v388: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.2 MiB/s wr, 612 op/s 2024-04-26T18:08:57.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:57 smithi031 ceph-mon[29164]: pgmap v388: 97 pgs: 97 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.2 MiB/s wr, 612 op/s 2024-04-26T18:08:58.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 695.0 s, 2292.2 tps, lat 43.445 ms stddev 46.236 2024-04-26T18:08:59.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:08:59 smithi097 ceph-mon[32419]: pgmap v389: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.7 MiB/s wr, 876 op/s 2024-04-26T18:08:59.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:08:59 smithi186 ceph-mon[32797]: pgmap v389: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.7 MiB/s wr, 876 op/s 2024-04-26T18:08:59.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:08:59 smithi031 ceph-mon[29164]: pgmap v389: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.7 MiB/s wr, 876 op/s 2024-04-26T18:09:01.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:01 smithi097 ceph-mon[32419]: pgmap v390: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 609 op/s 2024-04-26T18:09:01.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:01 smithi186 ceph-mon[32797]: pgmap v390: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 609 op/s 2024-04-26T18:09:01.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:01 smithi031 ceph-mon[29164]: pgmap v390: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 609 op/s 2024-04-26T18:09:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 700.0 s, 1058.2 tps, lat 91.754 ms stddev 124.592 2024-04-26T18:09:03.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:03 smithi097 ceph-mon[32419]: pgmap v391: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.3 MiB/s wr, 867 op/s 2024-04-26T18:09:03.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:03 smithi186 ceph-mon[32797]: pgmap v391: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.3 MiB/s wr, 867 op/s 2024-04-26T18:09:03.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:03 smithi031 ceph-mon[29164]: pgmap v391: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.3 MiB/s wr, 867 op/s 2024-04-26T18:09:05.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:05 smithi097 ceph-mon[32419]: pgmap v392: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.4 MiB/s wr, 571 op/s 2024-04-26T18:09:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:05 smithi186 ceph-mon[32797]: pgmap v392: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.4 MiB/s wr, 571 op/s 2024-04-26T18:09:05.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:05 smithi031 ceph-mon[29164]: pgmap v392: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.4 MiB/s wr, 571 op/s 2024-04-26T18:09:07.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:07 smithi097 ceph-mon[32419]: pgmap v393: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.5 MiB/s wr, 577 op/s 2024-04-26T18:09:07.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:07 smithi186 ceph-mon[32797]: pgmap v393: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.5 MiB/s wr, 577 op/s 2024-04-26T18:09:07.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:07 smithi031 ceph-mon[29164]: pgmap v393: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.5 MiB/s wr, 577 op/s 2024-04-26T18:09:08.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 705.0 s, 498.0 tps, lat 205.566 ms stddev 300.692 2024-04-26T18:09:09.362 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:09 smithi031 ceph-mon[29164]: pgmap v394: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-26T18:09:09.362 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:09 smithi031 ceph-mon[29164]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T18:09:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:09 smithi097 ceph-mon[32419]: pgmap v394: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-26T18:09:09.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:09 smithi097 ceph-mon[32419]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T18:09:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:09 smithi186 ceph-mon[32797]: pgmap v394: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-26T18:09:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:09 smithi186 ceph-mon[32797]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T18:09:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:09.903+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:09:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:09.903+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:09:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:09.903+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:09:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:09.903+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:09:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:09.903+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:09:11.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:11 smithi097 ceph-mon[32419]: pgmap v396: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 11 MiB/s wr, 981 op/s 2024-04-26T18:09:11.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:11 smithi186 ceph-mon[32797]: pgmap v396: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 11 MiB/s wr, 981 op/s 2024-04-26T18:09:11.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:11 smithi031 ceph-mon[29164]: pgmap v396: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 11 MiB/s wr, 981 op/s 2024-04-26T18:09:13.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 710.0 s, 1633.4 tps, lat 61.894 ms stddev 80.346 2024-04-26T18:09:13.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:13 smithi097 ceph-mon[32419]: pgmap v397: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 23 MiB/s wr, 1.72k op/s 2024-04-26T18:09:13.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:13 smithi186 ceph-mon[32797]: pgmap v397: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 23 MiB/s wr, 1.72k op/s 2024-04-26T18:09:13.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:13 smithi031 ceph-mon[29164]: pgmap v397: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 23 MiB/s wr, 1.72k op/s 2024-04-26T18:09:15.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:15 smithi097 ceph-mon[32419]: pgmap v398: 97 pgs: 97 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 24 MiB/s wr, 1.82k op/s 2024-04-26T18:09:15.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:15 smithi186 ceph-mon[32797]: pgmap v398: 97 pgs: 97 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 24 MiB/s wr, 1.82k op/s 2024-04-26T18:09:15.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:15 smithi031 ceph-mon[29164]: pgmap v398: 97 pgs: 97 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 24 MiB/s wr, 1.82k op/s 2024-04-26T18:09:17.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:17 smithi097 ceph-mon[32419]: pgmap v399: 97 pgs: 97 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 24 MiB/s wr, 1.82k op/s 2024-04-26T18:09:17.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:17 smithi186 ceph-mon[32797]: pgmap v399: 97 pgs: 97 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 24 MiB/s wr, 1.82k op/s 2024-04-26T18:09:17.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:17 smithi031 ceph-mon[29164]: pgmap v399: 97 pgs: 97 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 24 MiB/s wr, 1.82k op/s 2024-04-26T18:09:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 715.0 s, 1726.6 tps, lat 58.128 ms stddev 66.985 2024-04-26T18:09:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:19 smithi097 ceph-mon[32419]: pgmap v400: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 19 MiB/s wr, 1.45k op/s 2024-04-26T18:09:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:19 smithi097 ceph-mon[32419]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T18:09:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:19 smithi186 ceph-mon[32797]: pgmap v400: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 19 MiB/s wr, 1.45k op/s 2024-04-26T18:09:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:19 smithi186 ceph-mon[32797]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T18:09:19.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:19 smithi031 ceph-mon[29164]: pgmap v400: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 19 MiB/s wr, 1.45k op/s 2024-04-26T18:09:19.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:19 smithi031 ceph-mon[29164]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T18:09:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:21 smithi097 ceph-mon[32419]: pgmap v402: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 19 MiB/s wr, 1.45k op/s 2024-04-26T18:09:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:21 smithi186 ceph-mon[32797]: pgmap v402: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 19 MiB/s wr, 1.45k op/s 2024-04-26T18:09:21.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:21 smithi031 ceph-mon[29164]: pgmap v402: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 19 MiB/s wr, 1.45k op/s 2024-04-26T18:09:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 720.0 s, 1715.8 tps, lat 57.875 ms stddev 64.725 2024-04-26T18:09:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:23 smithi097 ceph-mon[32419]: pgmap v403: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 709 op/s 2024-04-26T18:09:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:23 smithi186 ceph-mon[32797]: pgmap v403: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 709 op/s 2024-04-26T18:09:23.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:23 smithi031 ceph-mon[29164]: pgmap v403: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 709 op/s 2024-04-26T18:09:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:24.911+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:09:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:24.911+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:09:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:24.911+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:09:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:24.911+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:09:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:24.911+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:09:25.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:25 smithi097 ceph-mon[32419]: pgmap v404: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.4 MiB/s wr, 676 op/s 2024-04-26T18:09:25.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:25 smithi186 ceph-mon[32797]: pgmap v404: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.4 MiB/s wr, 676 op/s 2024-04-26T18:09:25.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:25 smithi031 ceph-mon[29164]: pgmap v404: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.4 MiB/s wr, 676 op/s 2024-04-26T18:09:27.519 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:27 smithi186 ceph-mon[32797]: pgmap v405: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.1 MiB/s wr, 680 op/s 2024-04-26T18:09:27.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:27 smithi031 ceph-mon[29164]: pgmap v405: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.1 MiB/s wr, 680 op/s 2024-04-26T18:09:27.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:27 smithi097 ceph-mon[32419]: pgmap v405: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.1 MiB/s wr, 680 op/s 2024-04-26T18:09:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 725.0 s, 1513.4 tps, lat 66.524 ms stddev 66.183 2024-04-26T18:09:29.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:29 smithi186 ceph-mon[32797]: pgmap v406: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.9 MiB/s wr, 725 op/s 2024-04-26T18:09:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:29 smithi031 ceph-mon[29164]: pgmap v406: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.9 MiB/s wr, 725 op/s 2024-04-26T18:09:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:29 smithi097 ceph-mon[32419]: pgmap v406: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.9 MiB/s wr, 725 op/s 2024-04-26T18:09:31.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:31 smithi186 ceph-mon[32797]: pgmap v407: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.1 MiB/s wr, 646 op/s 2024-04-26T18:09:31.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:31 smithi031 ceph-mon[29164]: pgmap v407: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.1 MiB/s wr, 646 op/s 2024-04-26T18:09:31.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:31 smithi097 ceph-mon[32419]: pgmap v407: 97 pgs: 97 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.1 MiB/s wr, 646 op/s 2024-04-26T18:09:33.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 730.0 s, 1434.2 tps, lat 69.344 ms stddev 73.385 2024-04-26T18:09:33.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:33 smithi186 ceph-mon[32797]: pgmap v408: 97 pgs: 97 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 8.5 MiB/s wr, 805 op/s 2024-04-26T18:09:33.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:33 smithi031 ceph-mon[29164]: pgmap v408: 97 pgs: 97 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 8.5 MiB/s wr, 805 op/s 2024-04-26T18:09:33.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:33 smithi097 ceph-mon[32419]: pgmap v408: 97 pgs: 97 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 8.5 MiB/s wr, 805 op/s 2024-04-26T18:09:35.435 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:35 smithi097 ceph-mon[32419]: pgmap v409: 97 pgs: 97 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.3 MiB/s wr, 594 op/s 2024-04-26T18:09:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:35 smithi186 ceph-mon[32797]: pgmap v409: 97 pgs: 97 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.3 MiB/s wr, 594 op/s 2024-04-26T18:09:35.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:35 smithi031 ceph-mon[29164]: pgmap v409: 97 pgs: 97 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.3 MiB/s wr, 594 op/s 2024-04-26T18:09:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:37 smithi186 ceph-mon[32797]: pgmap v410: 97 pgs: 97 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.5 MiB/s wr, 594 op/s 2024-04-26T18:09:37.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:37 smithi031 ceph-mon[29164]: pgmap v410: 97 pgs: 97 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.5 MiB/s wr, 594 op/s 2024-04-26T18:09:37.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:37 smithi097 ceph-mon[32419]: pgmap v410: 97 pgs: 97 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.5 MiB/s wr, 594 op/s 2024-04-26T18:09:38.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 735.0 s, 1179.4 tps, lat 84.532 ms stddev 128.624 2024-04-26T18:09:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:39 smithi186 ceph-mon[32797]: pgmap v411: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 20 MiB/s wr, 1.76k op/s 2024-04-26T18:09:39.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:39 smithi031 ceph-mon[29164]: pgmap v411: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 20 MiB/s wr, 1.76k op/s 2024-04-26T18:09:39.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:39 smithi097 ceph-mon[32419]: pgmap v411: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 20 MiB/s wr, 1.76k op/s 2024-04-26T18:09:39.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:39.913+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:09:39.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:39.913+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:09:39.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:39.913+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:09:39.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:39.913+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:09:39.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:39.913+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:09:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:41 smithi186 ceph-mon[32797]: pgmap v412: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 1.49k op/s 2024-04-26T18:09:41.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:41 smithi031 ceph-mon[29164]: pgmap v412: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 1.49k op/s 2024-04-26T18:09:41.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:41 smithi097 ceph-mon[32419]: pgmap v412: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 1.49k op/s 2024-04-26T18:09:43.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 740.0 s, 2251.2 tps, lat 44.591 ms stddev 49.108 2024-04-26T18:09:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:43 smithi186 ceph-mon[32797]: pgmap v413: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 20 MiB/s wr, 1.73k op/s 2024-04-26T18:09:43.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:43 smithi031 ceph-mon[29164]: pgmap v413: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 20 MiB/s wr, 1.73k op/s 2024-04-26T18:09:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:43 smithi097 ceph-mon[32419]: pgmap v413: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 20 MiB/s wr, 1.73k op/s 2024-04-26T18:09:45.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:45 smithi186 ceph-mon[32797]: pgmap v414: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 MiB/s wr, 1.57k op/s 2024-04-26T18:09:45.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:45 smithi031 ceph-mon[29164]: pgmap v414: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 MiB/s wr, 1.57k op/s 2024-04-26T18:09:45.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:45 smithi097 ceph-mon[32419]: pgmap v414: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 MiB/s wr, 1.57k op/s 2024-04-26T18:09:47.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:47 smithi186 ceph-mon[32797]: pgmap v415: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 MiB/s wr, 1.54k op/s 2024-04-26T18:09:47.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:47 smithi031 ceph-mon[29164]: pgmap v415: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 MiB/s wr, 1.54k op/s 2024-04-26T18:09:47.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:47 smithi097 ceph-mon[32419]: pgmap v415: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 MiB/s wr, 1.54k op/s 2024-04-26T18:09:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 745.0 s, 1825.8 tps, lat 54.766 ms stddev 61.533 2024-04-26T18:09:49.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:49 smithi186 ceph-mon[32797]: pgmap v416: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 20 MiB/s wr, 1.74k op/s 2024-04-26T18:09:49.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:49 smithi031 ceph-mon[29164]: pgmap v416: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 20 MiB/s wr, 1.74k op/s 2024-04-26T18:09:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:49 smithi097 ceph-mon[32419]: pgmap v416: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 20 MiB/s wr, 1.74k op/s 2024-04-26T18:09:51.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:51 smithi186 ceph-mon[32797]: pgmap v417: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.3 MiB/s wr, 557 op/s 2024-04-26T18:09:51.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:51 smithi031 ceph-mon[29164]: pgmap v417: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.3 MiB/s wr, 557 op/s 2024-04-26T18:09:51.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:51 smithi097 ceph-mon[32419]: pgmap v417: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.3 MiB/s wr, 557 op/s 2024-04-26T18:09:53.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 750.0 s, 1340.8 tps, lat 74.033 ms stddev 81.346 2024-04-26T18:09:53.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:53 smithi186 ceph-mon[32797]: pgmap v418: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.7 MiB/s wr, 732 op/s 2024-04-26T18:09:53.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:53 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:09:53.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:53 smithi031 ceph-mon[29164]: pgmap v418: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.7 MiB/s wr, 732 op/s 2024-04-26T18:09:53.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:53 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:09:53.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:53 smithi097 ceph-mon[32419]: pgmap v418: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.7 MiB/s wr, 732 op/s 2024-04-26T18:09:53.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:53 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:09:54.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:54 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:09:54.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:54 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:09:54.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:54 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:09:54.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:54 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:09:54.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:54 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:09:54.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:54 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:09:54.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:54 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:09:54.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:54 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:09:54.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:54 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:09:54.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:54.922+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:09:54.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:54.923+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:09:54.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:54.923+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:09:54.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:54.923+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:09:54.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:09:54.923+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:09:55.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:55 smithi097 ceph-mon[32419]: pgmap v419: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.4 MiB/s wr, 586 op/s 2024-04-26T18:09:56.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:55 smithi186 ceph-mon[32797]: pgmap v419: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.4 MiB/s wr, 586 op/s 2024-04-26T18:09:56.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:55 smithi031 ceph-mon[29164]: pgmap v419: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.4 MiB/s wr, 586 op/s 2024-04-26T18:09:57.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:57 smithi097 ceph-mon[32419]: pgmap v420: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.1 MiB/s wr, 547 op/s 2024-04-26T18:09:58.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:57 smithi186 ceph-mon[32797]: pgmap v420: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.1 MiB/s wr, 547 op/s 2024-04-26T18:09:58.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:57 smithi031 ceph-mon[29164]: pgmap v420: 97 pgs: 97 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.1 MiB/s wr, 547 op/s 2024-04-26T18:09:58.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 755.0 s, 1332.2 tps, lat 75.937 ms stddev 79.430 2024-04-26T18:09:59.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:09:59 smithi097 ceph-mon[32419]: pgmap v421: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.6 MiB/s wr, 811 op/s 2024-04-26T18:10:00.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:09:59 smithi186 ceph-mon[32797]: pgmap v421: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.6 MiB/s wr, 811 op/s 2024-04-26T18:10:00.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:09:59 smithi031 ceph-mon[29164]: pgmap v421: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.6 MiB/s wr, 811 op/s 2024-04-26T18:10:01.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:00 smithi186 ceph-mon[32797]: overall HEALTH_OK 2024-04-26T18:10:01.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:00 smithi031 ceph-mon[29164]: overall HEALTH_OK 2024-04-26T18:10:01.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:00 smithi097 ceph-mon[32419]: overall HEALTH_OK 2024-04-26T18:10:01.917 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:01 smithi186 ceph-mon[32797]: pgmap v422: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.1 MiB/s wr, 550 op/s 2024-04-26T18:10:02.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:01 smithi031 ceph-mon[29164]: pgmap v422: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.1 MiB/s wr, 550 op/s 2024-04-26T18:10:02.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:01 smithi097 ceph-mon[32419]: pgmap v422: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.1 MiB/s wr, 550 op/s 2024-04-26T18:10:03.095 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 760.0 s, 1176.8 tps, lat 82.688 ms stddev 98.196 2024-04-26T18:10:04.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:03 smithi186 ceph-mon[32797]: pgmap v423: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.3 MiB/s wr, 722 op/s 2024-04-26T18:10:04.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:03 smithi031 ceph-mon[29164]: pgmap v423: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.3 MiB/s wr, 722 op/s 2024-04-26T18:10:04.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:03 smithi097 ceph-mon[32419]: pgmap v423: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.3 MiB/s wr, 722 op/s 2024-04-26T18:10:05.935 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:05 smithi097 ceph-mon[32419]: pgmap v424: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.8 MiB/s wr, 638 op/s 2024-04-26T18:10:06.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:05 smithi186 ceph-mon[32797]: pgmap v424: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.8 MiB/s wr, 638 op/s 2024-04-26T18:10:06.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:05 smithi031 ceph-mon[29164]: pgmap v424: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.8 MiB/s wr, 638 op/s 2024-04-26T18:10:08.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:07 smithi186 ceph-mon[32797]: pgmap v425: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.4 MiB/s wr, 597 op/s 2024-04-26T18:10:08.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:07 smithi031 ceph-mon[29164]: pgmap v425: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.4 MiB/s wr, 597 op/s 2024-04-26T18:10:08.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 765.0 s, 976.6 tps, lat 96.335 ms stddev 172.072 2024-04-26T18:10:08.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:07 smithi097 ceph-mon[32419]: pgmap v425: 97 pgs: 97 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.4 MiB/s wr, 597 op/s 2024-04-26T18:10:09.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:08 smithi186 ceph-mon[32797]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T18:10:09.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:08 smithi186 ceph-mon[32797]: pgmap v427: 97 pgs: 97 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 20 MiB/s wr, 1.74k op/s 2024-04-26T18:10:09.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:08 smithi031 ceph-mon[29164]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T18:10:09.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:08 smithi031 ceph-mon[29164]: pgmap v427: 97 pgs: 97 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 20 MiB/s wr, 1.74k op/s 2024-04-26T18:10:09.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:08 smithi097 ceph-mon[32419]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T18:10:09.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:08 smithi097 ceph-mon[32419]: pgmap v427: 97 pgs: 97 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 20 MiB/s wr, 1.74k op/s 2024-04-26T18:10:09.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:09.926+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:10:09.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:09.926+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:10:09.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:09.926+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:10:09.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:09.926+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:10:09.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:09.926+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:10:11.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:11 smithi097 ceph-mon[32419]: pgmap v428: 97 pgs: 97 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 20 MiB/s wr, 1.74k op/s 2024-04-26T18:10:11.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:11 smithi186 ceph-mon[32797]: pgmap v428: 97 pgs: 97 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 20 MiB/s wr, 1.74k op/s 2024-04-26T18:10:11.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:11 smithi031 ceph-mon[29164]: pgmap v428: 97 pgs: 97 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 20 MiB/s wr, 1.74k op/s 2024-04-26T18:10:13.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 770.0 s, 1676.4 tps, lat 64.572 ms stddev 97.656 2024-04-26T18:10:13.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:13 smithi097 ceph-mon[32419]: pgmap v429: 97 pgs: 97 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 25 MiB/s wr, 1.84k op/s 2024-04-26T18:10:13.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:13 smithi186 ceph-mon[32797]: pgmap v429: 97 pgs: 97 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 25 MiB/s wr, 1.84k op/s 2024-04-26T18:10:13.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:13 smithi031 ceph-mon[29164]: pgmap v429: 97 pgs: 97 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 25 MiB/s wr, 1.84k op/s 2024-04-26T18:10:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:14.412+0000 7f75d17fa640 1 -- 172.21.15.31:0/865661195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f75b4005850 con 0x7f75dc11c110 2024-04-26T18:10:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:14.412+0000 7f75d27fc640 1 -- 172.21.15.31:0/865661195 <== mon.0 v2:172.21.15.31:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f75b4005850 con 0x7f75dc11c110 2024-04-26T18:10:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:14.428+0000 7f75bb7fe640 1 -- 172.21.15.31:0/164021214 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f75ac005850 con 0x556108f4ff00 2024-04-26T18:10:14.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:14.429+0000 7f75c4ff9640 1 -- 172.21.15.31:0/164021214 <== mon.1 v2:172.21.15.186:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f75ac005850 con 0x556108f4ff00 2024-04-26T18:10:15.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:15 smithi097 ceph-mon[32419]: pgmap v430: 97 pgs: 97 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 26 MiB/s wr, 1.82k op/s 2024-04-26T18:10:15.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:15 smithi186 ceph-mon[32797]: pgmap v430: 97 pgs: 97 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 26 MiB/s wr, 1.82k op/s 2024-04-26T18:10:15.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:15 smithi031 ceph-mon[29164]: pgmap v430: 97 pgs: 97 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 26 MiB/s wr, 1.82k op/s 2024-04-26T18:10:17.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:17 smithi097 ceph-mon[32419]: pgmap v431: 97 pgs: 97 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 26 MiB/s wr, 1.78k op/s 2024-04-26T18:10:17.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:17 smithi186 ceph-mon[32797]: pgmap v431: 97 pgs: 97 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 26 MiB/s wr, 1.78k op/s 2024-04-26T18:10:17.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:17 smithi031 ceph-mon[29164]: pgmap v431: 97 pgs: 97 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 26 MiB/s wr, 1.78k op/s 2024-04-26T18:10:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 775.0 s, 1816.8 tps, lat 55.103 ms stddev 61.607 2024-04-26T18:10:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:19 smithi097 ceph-mon[32419]: pgmap v432: 97 pgs: 97 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 17 MiB/s wr, 699 op/s 2024-04-26T18:10:19.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:19 smithi097 ceph-mon[32419]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T18:10:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:19 smithi186 ceph-mon[32797]: pgmap v432: 97 pgs: 97 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 17 MiB/s wr, 699 op/s 2024-04-26T18:10:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:19 smithi186 ceph-mon[32797]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T18:10:19.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:19 smithi031 ceph-mon[29164]: pgmap v432: 97 pgs: 97 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 17 MiB/s wr, 699 op/s 2024-04-26T18:10:19.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:19 smithi031 ceph-mon[29164]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T18:10:21.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:21 smithi097 ceph-mon[32419]: pgmap v434: 97 pgs: 97 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 MiB/s wr, 726 op/s 2024-04-26T18:10:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:21 smithi186 ceph-mon[32797]: pgmap v434: 97 pgs: 97 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 MiB/s wr, 726 op/s 2024-04-26T18:10:21.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:21 smithi031 ceph-mon[29164]: pgmap v434: 97 pgs: 97 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 MiB/s wr, 726 op/s 2024-04-26T18:10:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 780.0 s, 2005.3 tps, lat 46.587 ms stddev 55.645 2024-04-26T18:10:23.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:23 smithi097 ceph-mon[32419]: pgmap v435: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 17 MiB/s wr, 677 op/s 2024-04-26T18:10:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:23 smithi186 ceph-mon[32797]: pgmap v435: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 17 MiB/s wr, 677 op/s 2024-04-26T18:10:23.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:23 smithi031 ceph-mon[29164]: pgmap v435: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 17 MiB/s wr, 677 op/s 2024-04-26T18:10:24.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:24.934+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:10:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:24.934+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:10:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:24.934+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:10:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:24.934+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:10:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:24.934+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:10:25.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:25 smithi097 ceph-mon[32419]: pgmap v436: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 17 MiB/s wr, 673 op/s 2024-04-26T18:10:25.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:25 smithi186 ceph-mon[32797]: pgmap v436: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 17 MiB/s wr, 673 op/s 2024-04-26T18:10:25.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:25 smithi031 ceph-mon[29164]: pgmap v436: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 17 MiB/s wr, 673 op/s 2024-04-26T18:10:27.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:27 smithi097 ceph-mon[32419]: pgmap v437: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 16 MiB/s wr, 652 op/s 2024-04-26T18:10:27.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:27 smithi186 ceph-mon[32797]: pgmap v437: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 16 MiB/s wr, 652 op/s 2024-04-26T18:10:27.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:27 smithi031 ceph-mon[29164]: pgmap v437: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 16 MiB/s wr, 652 op/s 2024-04-26T18:10:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 785.0 s, 1208.1 tps, lat 88.214 ms stddev 294.224 2024-04-26T18:10:29.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:29 smithi097 ceph-mon[32419]: pgmap v438: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 14 MiB/s wr, 668 op/s 2024-04-26T18:10:29.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:29 smithi186 ceph-mon[32797]: pgmap v438: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 14 MiB/s wr, 668 op/s 2024-04-26T18:10:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:29 smithi031 ceph-mon[29164]: pgmap v438: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 14 MiB/s wr, 668 op/s 2024-04-26T18:10:31.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:31 smithi097 ceph-mon[32419]: pgmap v439: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 595 op/s 2024-04-26T18:10:31.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:31 smithi186 ceph-mon[32797]: pgmap v439: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 595 op/s 2024-04-26T18:10:31.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:31 smithi031 ceph-mon[29164]: pgmap v439: 97 pgs: 97 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 595 op/s 2024-04-26T18:10:33.103 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 790.0 s, 1825.8 tps, lat 54.837 ms stddev 59.864 2024-04-26T18:10:33.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:33 smithi186 ceph-mon[32797]: pgmap v440: 97 pgs: 97 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 16 MiB/s wr, 803 op/s 2024-04-26T18:10:33.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:33 smithi031 ceph-mon[29164]: pgmap v440: 97 pgs: 97 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 16 MiB/s wr, 803 op/s 2024-04-26T18:10:33.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:33 smithi097 ceph-mon[32419]: pgmap v440: 97 pgs: 97 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 16 MiB/s wr, 803 op/s 2024-04-26T18:10:35.436 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:35 smithi097 ceph-mon[32419]: pgmap v441: 97 pgs: 97 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 645 op/s 2024-04-26T18:10:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:35 smithi186 ceph-mon[32797]: pgmap v441: 97 pgs: 97 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 645 op/s 2024-04-26T18:10:35.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:35 smithi031 ceph-mon[29164]: pgmap v441: 97 pgs: 97 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 645 op/s 2024-04-26T18:10:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:37 smithi186 ceph-mon[32797]: pgmap v442: 97 pgs: 97 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 10 MiB/s wr, 575 op/s 2024-04-26T18:10:37.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:37 smithi031 ceph-mon[29164]: pgmap v442: 97 pgs: 97 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 10 MiB/s wr, 575 op/s 2024-04-26T18:10:37.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:37 smithi097 ceph-mon[32419]: pgmap v442: 97 pgs: 97 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 10 MiB/s wr, 575 op/s 2024-04-26T18:10:38.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 795.0 s, 1402.4 tps, lat 71.662 ms stddev 173.158 2024-04-26T18:10:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:39 smithi186 ceph-mon[32797]: pgmap v443: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 29 MiB/s wr, 1.83k op/s 2024-04-26T18:10:39.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:39 smithi031 ceph-mon[29164]: pgmap v443: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 29 MiB/s wr, 1.83k op/s 2024-04-26T18:10:39.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:39 smithi097 ceph-mon[32419]: pgmap v443: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 29 MiB/s wr, 1.83k op/s 2024-04-26T18:10:39.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:39.937+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:10:39.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:39.937+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:10:39.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:39.937+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:10:39.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:39.937+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:10:39.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:39.937+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:10:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:41 smithi186 ceph-mon[32797]: pgmap v444: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 24 MiB/s wr, 1.56k op/s 2024-04-26T18:10:41.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:41 smithi031 ceph-mon[29164]: pgmap v444: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 24 MiB/s wr, 1.56k op/s 2024-04-26T18:10:41.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:41 smithi097 ceph-mon[32419]: pgmap v444: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 24 MiB/s wr, 1.56k op/s 2024-04-26T18:10:43.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 800.0 s, 2191.4 tps, lat 45.236 ms stddev 59.363 2024-04-26T18:10:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:43 smithi186 ceph-mon[32797]: pgmap v445: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 MiB/s wr, 1.79k op/s 2024-04-26T18:10:43.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:43 smithi031 ceph-mon[29164]: pgmap v445: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 MiB/s wr, 1.79k op/s 2024-04-26T18:10:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:43 smithi097 ceph-mon[32419]: pgmap v445: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 MiB/s wr, 1.79k op/s 2024-04-26T18:10:45.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:45 smithi186 ceph-mon[32797]: pgmap v446: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 25 MiB/s wr, 1.60k op/s 2024-04-26T18:10:45.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:45 smithi031 ceph-mon[29164]: pgmap v446: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 25 MiB/s wr, 1.60k op/s 2024-04-26T18:10:45.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:45 smithi097 ceph-mon[32419]: pgmap v446: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 25 MiB/s wr, 1.60k op/s 2024-04-26T18:10:47.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:47 smithi186 ceph-mon[32797]: pgmap v447: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 MiB/s wr, 1.55k op/s 2024-04-26T18:10:47.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:47 smithi031 ceph-mon[29164]: pgmap v447: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 MiB/s wr, 1.55k op/s 2024-04-26T18:10:47.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:47 smithi097 ceph-mon[32419]: pgmap v447: 97 pgs: 97 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 MiB/s wr, 1.55k op/s 2024-04-26T18:10:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 805.0 s, 1670.6 tps, lat 60.377 ms stddev 114.839 2024-04-26T18:10:49.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:49 smithi186 ceph-mon[32797]: pgmap v448: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 33 MiB/s wr, 2.33k op/s 2024-04-26T18:10:49.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:49 smithi031 ceph-mon[29164]: pgmap v448: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 33 MiB/s wr, 2.33k op/s 2024-04-26T18:10:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:49 smithi097 ceph-mon[32419]: pgmap v448: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 33 MiB/s wr, 2.33k op/s 2024-04-26T18:10:51.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:51 smithi186 ceph-mon[32797]: pgmap v449: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 15 MiB/s wr, 1.08k op/s 2024-04-26T18:10:51.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:51 smithi031 ceph-mon[29164]: pgmap v449: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 15 MiB/s wr, 1.08k op/s 2024-04-26T18:10:51.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:51 smithi097 ceph-mon[32419]: pgmap v449: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 15 MiB/s wr, 1.08k op/s 2024-04-26T18:10:53.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 810.0 s, 2028.6 tps, lat 48.288 ms stddev 57.541 2024-04-26T18:10:53.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:53 smithi186 ceph-mon[32797]: pgmap v450: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 20 MiB/s wr, 1.57k op/s 2024-04-26T18:10:53.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:53 smithi031 ceph-mon[29164]: pgmap v450: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 20 MiB/s wr, 1.57k op/s 2024-04-26T18:10:53.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:53 smithi097 ceph-mon[32419]: pgmap v450: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 20 MiB/s wr, 1.57k op/s 2024-04-26T18:10:54.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:54 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:10:54.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:54 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:10:54.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:54 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:10:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:54.943+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:10:54.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:54.943+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:10:54.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:54.943+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:10:54.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:54.943+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:10:54.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:10:54.943+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:10:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:55 smithi186 ceph-mon[32797]: pgmap v451: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 18 MiB/s wr, 1.42k op/s 2024-04-26T18:10:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:55 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:10:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:55 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:10:55.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:55 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:10:55.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:55 smithi031 ceph-mon[29164]: pgmap v451: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 18 MiB/s wr, 1.42k op/s 2024-04-26T18:10:55.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:55 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:10:55.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:55 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:10:55.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:55 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:10:55.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:55 smithi097 ceph-mon[32419]: pgmap v451: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 18 MiB/s wr, 1.42k op/s 2024-04-26T18:10:55.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:55 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:10:55.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:55 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:10:55.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:55 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:10:57.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:57 smithi186 ceph-mon[32797]: pgmap v452: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 16 MiB/s wr, 1.35k op/s 2024-04-26T18:10:57.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:57 smithi031 ceph-mon[29164]: pgmap v452: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 16 MiB/s wr, 1.35k op/s 2024-04-26T18:10:57.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:57 smithi097 ceph-mon[32419]: pgmap v452: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 16 MiB/s wr, 1.35k op/s 2024-04-26T18:10:58.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 815.0 s, 1841.6 tps, lat 54.840 ms stddev 59.689 2024-04-26T18:10:59.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:10:59 smithi186 ceph-mon[32797]: pgmap v453: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 21 MiB/s wr, 1.73k op/s 2024-04-26T18:10:59.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:10:59 smithi031 ceph-mon[29164]: pgmap v453: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 21 MiB/s wr, 1.73k op/s 2024-04-26T18:10:59.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:10:59 smithi097 ceph-mon[32419]: pgmap v453: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 21 MiB/s wr, 1.73k op/s 2024-04-26T18:11:01.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:01 smithi186 ceph-mon[32797]: pgmap v454: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 12 MiB/s wr, 943 op/s 2024-04-26T18:11:01.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:01 smithi031 ceph-mon[29164]: pgmap v454: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 12 MiB/s wr, 943 op/s 2024-04-26T18:11:01.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:01 smithi097 ceph-mon[32419]: pgmap v454: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 12 MiB/s wr, 943 op/s 2024-04-26T18:11:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 820.0 s, 1076.4 tps, lat 92.644 ms stddev 113.464 2024-04-26T18:11:03.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:03 smithi186 ceph-mon[32797]: pgmap v455: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 14 MiB/s wr, 1.16k op/s 2024-04-26T18:11:03.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:03 smithi031 ceph-mon[29164]: pgmap v455: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 14 MiB/s wr, 1.16k op/s 2024-04-26T18:11:03.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:03 smithi097 ceph-mon[32419]: pgmap v455: 97 pgs: 97 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 14 MiB/s wr, 1.16k op/s 2024-04-26T18:11:05.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:05 smithi186 ceph-mon[32797]: pgmap v456: 97 pgs: 97 active+clean; 5.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 10 MiB/s wr, 793 op/s 2024-04-26T18:11:05.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:05 smithi031 ceph-mon[29164]: pgmap v456: 97 pgs: 97 active+clean; 5.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 10 MiB/s wr, 793 op/s 2024-04-26T18:11:05.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:05 smithi097 ceph-mon[32419]: pgmap v456: 97 pgs: 97 active+clean; 5.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 10 MiB/s wr, 793 op/s 2024-04-26T18:11:07.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:07 smithi186 ceph-mon[32797]: pgmap v457: 97 pgs: 97 active+clean; 5.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 8.7 MiB/s wr, 722 op/s 2024-04-26T18:11:07.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:07 smithi031 ceph-mon[29164]: pgmap v457: 97 pgs: 97 active+clean; 5.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 8.7 MiB/s wr, 722 op/s 2024-04-26T18:11:07.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:07 smithi097 ceph-mon[32419]: pgmap v457: 97 pgs: 97 active+clean; 5.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 8.7 MiB/s wr, 722 op/s 2024-04-26T18:11:08.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 825.0 s, 1451.6 tps, lat 64.977 ms stddev 89.952 2024-04-26T18:11:08.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:08 smithi186 ceph-mon[32797]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T18:11:08.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:08 smithi031 ceph-mon[29164]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T18:11:08.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:08 smithi097 ceph-mon[32419]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T18:11:09.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:09 smithi186 ceph-mon[32797]: pgmap v459: 97 pgs: 4 active+clean+snaptrim, 82 active+clean, 11 active+clean+snaptrim_wait; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 17 MiB/s wr, 1.55k op/s 2024-04-26T18:11:09.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:09 smithi031 ceph-mon[29164]: pgmap v459: 97 pgs: 4 active+clean+snaptrim, 82 active+clean, 11 active+clean+snaptrim_wait; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 17 MiB/s wr, 1.55k op/s 2024-04-26T18:11:09.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:09 smithi097 ceph-mon[32419]: pgmap v459: 97 pgs: 4 active+clean+snaptrim, 82 active+clean, 11 active+clean+snaptrim_wait; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 17 MiB/s wr, 1.55k op/s 2024-04-26T18:11:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:09.945+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:11:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:09.945+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:11:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:09.945+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:11:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:09.945+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:11:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:09.945+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:11:11.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:11 smithi186 ceph-mon[32797]: pgmap v460: 97 pgs: 4 active+clean+snaptrim, 82 active+clean, 11 active+clean+snaptrim_wait; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 17 MiB/s wr, 1.55k op/s 2024-04-26T18:11:11.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:11 smithi031 ceph-mon[29164]: pgmap v460: 97 pgs: 4 active+clean+snaptrim, 82 active+clean, 11 active+clean+snaptrim_wait; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 17 MiB/s wr, 1.55k op/s 2024-04-26T18:11:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:11 smithi097 ceph-mon[32419]: pgmap v460: 97 pgs: 4 active+clean+snaptrim, 82 active+clean, 11 active+clean+snaptrim_wait; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 17 MiB/s wr, 1.55k op/s 2024-04-26T18:11:13.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 830.0 s, 1912.4 tps, lat 55.650 ms stddev 80.772 2024-04-26T18:11:13.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:13 smithi186 ceph-mon[32797]: pgmap v461: 97 pgs: 4 active+clean+snaptrim, 82 active+clean, 11 active+clean+snaptrim_wait; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 18 MiB/s wr, 1.54k op/s 2024-04-26T18:11:13.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:13 smithi031 ceph-mon[29164]: pgmap v461: 97 pgs: 4 active+clean+snaptrim, 82 active+clean, 11 active+clean+snaptrim_wait; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 18 MiB/s wr, 1.54k op/s 2024-04-26T18:11:13.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:13 smithi097 ceph-mon[32419]: pgmap v461: 97 pgs: 4 active+clean+snaptrim, 82 active+clean, 11 active+clean+snaptrim_wait; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 18 MiB/s wr, 1.54k op/s 2024-04-26T18:11:15.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:15 smithi186 ceph-mon[32797]: pgmap v462: 97 pgs: 97 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 16 MiB/s wr, 1.44k op/s 2024-04-26T18:11:15.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:15 smithi031 ceph-mon[29164]: pgmap v462: 97 pgs: 97 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 16 MiB/s wr, 1.44k op/s 2024-04-26T18:11:15.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:15 smithi097 ceph-mon[32419]: pgmap v462: 97 pgs: 97 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 16 MiB/s wr, 1.44k op/s 2024-04-26T18:11:17.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:17 smithi186 ceph-mon[32797]: pgmap v463: 97 pgs: 97 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 16 MiB/s wr, 1.44k op/s 2024-04-26T18:11:17.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:17 smithi031 ceph-mon[29164]: pgmap v463: 97 pgs: 97 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 16 MiB/s wr, 1.44k op/s 2024-04-26T18:11:17.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:17 smithi097 ceph-mon[32419]: pgmap v463: 97 pgs: 97 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 16 MiB/s wr, 1.44k op/s 2024-04-26T18:11:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 835.0 s, 1829.2 tps, lat 54.640 ms stddev 60.492 2024-04-26T18:11:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:19 smithi186 ceph-mon[32797]: pgmap v464: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 9.2 MiB/s wr, 746 op/s 2024-04-26T18:11:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:19 smithi186 ceph-mon[32797]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T18:11:19.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:19 smithi031 ceph-mon[29164]: pgmap v464: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 9.2 MiB/s wr, 746 op/s 2024-04-26T18:11:19.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:19 smithi031 ceph-mon[29164]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T18:11:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:19 smithi097 ceph-mon[32419]: pgmap v464: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 9.2 MiB/s wr, 746 op/s 2024-04-26T18:11:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:19 smithi097 ceph-mon[32419]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T18:11:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:21 smithi186 ceph-mon[32797]: pgmap v466: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.4 MiB/s wr, 613 op/s 2024-04-26T18:11:21.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:21 smithi031 ceph-mon[29164]: pgmap v466: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.4 MiB/s wr, 613 op/s 2024-04-26T18:11:21.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:21 smithi097 ceph-mon[32419]: pgmap v466: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.4 MiB/s wr, 613 op/s 2024-04-26T18:11:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 840.0 s, 1656.2 tps, lat 59.833 ms stddev 63.287 2024-04-26T18:11:23.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:23 smithi031 ceph-mon[29164]: pgmap v467: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.9 MiB/s wr, 598 op/s 2024-04-26T18:11:23.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:23 smithi097 ceph-mon[32419]: pgmap v467: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.9 MiB/s wr, 598 op/s 2024-04-26T18:11:23.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:23 smithi186 ceph-mon[32797]: pgmap v467: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.9 MiB/s wr, 598 op/s 2024-04-26T18:11:24.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:24.953+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:11:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:24.955+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:11:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:24.955+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:11:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:24.955+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:11:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:24.955+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:11:25.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:25 smithi031 ceph-mon[29164]: pgmap v468: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.2 MiB/s wr, 610 op/s 2024-04-26T18:11:25.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:25 smithi097 ceph-mon[32419]: pgmap v468: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.2 MiB/s wr, 610 op/s 2024-04-26T18:11:25.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:25 smithi186 ceph-mon[32797]: pgmap v468: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.2 MiB/s wr, 610 op/s 2024-04-26T18:11:27.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:27 smithi031 ceph-mon[29164]: pgmap v469: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.2 MiB/s wr, 610 op/s 2024-04-26T18:11:27.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:27 smithi097 ceph-mon[32419]: pgmap v469: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.2 MiB/s wr, 610 op/s 2024-04-26T18:11:27.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:27 smithi186 ceph-mon[32797]: pgmap v469: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.2 MiB/s wr, 610 op/s 2024-04-26T18:11:28.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 845.0 s, 1946.6 tps, lat 51.559 ms stddev 54.692 2024-04-26T18:11:29.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:29 smithi031 ceph-mon[29164]: pgmap v470: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.3 MiB/s wr, 648 op/s 2024-04-26T18:11:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:29 smithi097 ceph-mon[32419]: pgmap v470: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.3 MiB/s wr, 648 op/s 2024-04-26T18:11:29.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:29 smithi186 ceph-mon[32797]: pgmap v470: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.3 MiB/s wr, 648 op/s 2024-04-26T18:11:31.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:31 smithi031 ceph-mon[29164]: pgmap v471: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.5 MiB/s wr, 577 op/s 2024-04-26T18:11:31.568 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:31 smithi186 ceph-mon[32797]: pgmap v471: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.5 MiB/s wr, 577 op/s 2024-04-26T18:11:31.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:31 smithi097 ceph-mon[32419]: pgmap v471: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.5 MiB/s wr, 577 op/s 2024-04-26T18:11:33.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 850.0 s, 2527.6 tps, lat 39.973 ms stddev 47.792 2024-04-26T18:11:33.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:33 smithi031 ceph-mon[29164]: pgmap v472: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.1 MiB/s wr, 800 op/s 2024-04-26T18:11:33.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:33 smithi097 ceph-mon[32419]: pgmap v472: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.1 MiB/s wr, 800 op/s 2024-04-26T18:11:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:33 smithi186 ceph-mon[32797]: pgmap v472: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.1 MiB/s wr, 800 op/s 2024-04-26T18:11:35.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:35 smithi031 ceph-mon[29164]: pgmap v473: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.4 MiB/s wr, 659 op/s 2024-04-26T18:11:35.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:35 smithi097 ceph-mon[32419]: pgmap v473: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.4 MiB/s wr, 659 op/s 2024-04-26T18:11:35.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:35 smithi186 ceph-mon[32797]: pgmap v473: 97 pgs: 97 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.4 MiB/s wr, 659 op/s 2024-04-26T18:11:37.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:37 smithi031 ceph-mon[29164]: pgmap v474: 97 pgs: 97 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.2 MiB/s wr, 700 op/s 2024-04-26T18:11:37.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:37 smithi097 ceph-mon[32419]: pgmap v474: 97 pgs: 97 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.2 MiB/s wr, 700 op/s 2024-04-26T18:11:37.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:37 smithi186 ceph-mon[32797]: pgmap v474: 97 pgs: 97 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.2 MiB/s wr, 700 op/s 2024-04-26T18:11:38.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 855.0 s, 1486.8 tps, lat 66.689 ms stddev 135.254 2024-04-26T18:11:39.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:39 smithi031 ceph-mon[29164]: pgmap v475: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 27 MiB/s wr, 1.95k op/s 2024-04-26T18:11:39.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:39 smithi097 ceph-mon[32419]: pgmap v475: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 27 MiB/s wr, 1.95k op/s 2024-04-26T18:11:39.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:39 smithi186 ceph-mon[32797]: pgmap v475: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 27 MiB/s wr, 1.95k op/s 2024-04-26T18:11:39.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:39.958+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:11:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:39.958+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:11:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:39.958+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:11:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:39.958+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:11:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:39.958+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:11:41.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:41 smithi031 ceph-mon[29164]: pgmap v476: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 24 MiB/s wr, 1.65k op/s 2024-04-26T18:11:41.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:41 smithi097 ceph-mon[32419]: pgmap v476: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 24 MiB/s wr, 1.65k op/s 2024-04-26T18:11:41.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:41 smithi186 ceph-mon[32797]: pgmap v476: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 24 MiB/s wr, 1.65k op/s 2024-04-26T18:11:43.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 860.0 s, 2318.0 tps, lat 43.306 ms stddev 54.888 2024-04-26T18:11:43.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:43 smithi031 ceph-mon[29164]: pgmap v477: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 26 MiB/s wr, 1.83k op/s 2024-04-26T18:11:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:43 smithi097 ceph-mon[32419]: pgmap v477: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 26 MiB/s wr, 1.83k op/s 2024-04-26T18:11:43.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:43 smithi186 ceph-mon[32797]: pgmap v477: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 26 MiB/s wr, 1.83k op/s 2024-04-26T18:11:45.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:45 smithi031 ceph-mon[29164]: pgmap v478: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 24 MiB/s wr, 1.66k op/s 2024-04-26T18:11:45.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:45 smithi097 ceph-mon[32419]: pgmap v478: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 24 MiB/s wr, 1.66k op/s 2024-04-26T18:11:45.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:45 smithi186 ceph-mon[32797]: pgmap v478: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 24 MiB/s wr, 1.66k op/s 2024-04-26T18:11:47.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:47 smithi031 ceph-mon[29164]: pgmap v479: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 24 MiB/s wr, 1.64k op/s 2024-04-26T18:11:47.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:47 smithi097 ceph-mon[32419]: pgmap v479: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 24 MiB/s wr, 1.64k op/s 2024-04-26T18:11:47.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:47 smithi186 ceph-mon[32797]: pgmap v479: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 24 MiB/s wr, 1.64k op/s 2024-04-26T18:11:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 865.0 s, 1537.0 tps, lat 65.234 ms stddev 75.055 2024-04-26T18:11:49.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:49 smithi031 ceph-mon[29164]: pgmap v480: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 25 MiB/s wr, 1.84k op/s 2024-04-26T18:11:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:49 smithi097 ceph-mon[32419]: pgmap v480: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 25 MiB/s wr, 1.84k op/s 2024-04-26T18:11:49.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:49 smithi186 ceph-mon[32797]: pgmap v480: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 25 MiB/s wr, 1.84k op/s 2024-04-26T18:11:51.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:51 smithi097 ceph-mon[32419]: pgmap v481: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 6.1 MiB/s wr, 594 op/s 2024-04-26T18:11:51.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:51 smithi186 ceph-mon[32797]: pgmap v481: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 6.1 MiB/s wr, 594 op/s 2024-04-26T18:11:51.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:51 smithi031 ceph-mon[29164]: pgmap v481: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 6.1 MiB/s wr, 594 op/s 2024-04-26T18:11:53.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 870.0 s, 2036.6 tps, lat 48.294 ms stddev 57.165 2024-04-26T18:11:53.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:53 smithi097 ceph-mon[32419]: pgmap v482: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 8.3 MiB/s wr, 793 op/s 2024-04-26T18:11:53.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:53 smithi186 ceph-mon[32797]: pgmap v482: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 8.3 MiB/s wr, 793 op/s 2024-04-26T18:11:53.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:53 smithi031 ceph-mon[29164]: pgmap v482: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 8.3 MiB/s wr, 793 op/s 2024-04-26T18:11:54.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:54.965+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:11:54.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:54.965+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:11:54.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:54.965+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:11:54.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:54.965+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:11:54.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:11:54.965+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:11:55.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:55 smithi097 ceph-mon[32419]: pgmap v483: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 7.1 MiB/s wr, 677 op/s 2024-04-26T18:11:55.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:55 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:11:55.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:55 smithi186 ceph-mon[32797]: pgmap v483: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 7.1 MiB/s wr, 677 op/s 2024-04-26T18:11:55.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:55 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:11:55.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:55 smithi031 ceph-mon[29164]: pgmap v483: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 7.1 MiB/s wr, 677 op/s 2024-04-26T18:11:55.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:55 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:11:56.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:56 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:11:56.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:56 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:11:56.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:56 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:11:56.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:56 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:11:56.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:56 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:11:56.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:56 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:11:56.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:56 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:11:56.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:56 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:11:56.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:56 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:11:57.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:57 smithi097 ceph-mon[32419]: pgmap v484: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 6.2 MiB/s wr, 604 op/s 2024-04-26T18:11:57.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:57 smithi186 ceph-mon[32797]: pgmap v484: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 6.2 MiB/s wr, 604 op/s 2024-04-26T18:11:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:57 smithi031 ceph-mon[29164]: pgmap v484: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 6.2 MiB/s wr, 604 op/s 2024-04-26T18:11:58.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 875.0 s, 1324.2 tps, lat 75.814 ms stddev 82.082 2024-04-26T18:11:59.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:11:59 smithi097 ceph-mon[32419]: pgmap v485: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 8.3 MiB/s wr, 818 op/s 2024-04-26T18:11:59.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:11:59 smithi186 ceph-mon[32797]: pgmap v485: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 8.3 MiB/s wr, 818 op/s 2024-04-26T18:11:59.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:11:59 smithi031 ceph-mon[29164]: pgmap v485: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 8.3 MiB/s wr, 818 op/s 2024-04-26T18:12:01.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:01 smithi097 ceph-mon[32419]: pgmap v486: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.5 MiB/s wr, 548 op/s 2024-04-26T18:12:01.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:01 smithi186 ceph-mon[32797]: pgmap v486: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.5 MiB/s wr, 548 op/s 2024-04-26T18:12:01.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:01 smithi031 ceph-mon[29164]: pgmap v486: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.5 MiB/s wr, 548 op/s 2024-04-26T18:12:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 880.0 s, 976.0 tps, lat 99.767 ms stddev 113.156 2024-04-26T18:12:03.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:03 smithi097 ceph-mon[32419]: pgmap v487: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 7.3 MiB/s wr, 751 op/s 2024-04-26T18:12:03.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:03 smithi186 ceph-mon[32797]: pgmap v487: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 7.3 MiB/s wr, 751 op/s 2024-04-26T18:12:03.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:03 smithi031 ceph-mon[29164]: pgmap v487: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 7.3 MiB/s wr, 751 op/s 2024-04-26T18:12:05.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:05 smithi097 ceph-mon[32419]: pgmap v488: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 5.6 MiB/s wr, 619 op/s 2024-04-26T18:12:05.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:05 smithi186 ceph-mon[32797]: pgmap v488: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 5.6 MiB/s wr, 619 op/s 2024-04-26T18:12:05.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:05 smithi031 ceph-mon[29164]: pgmap v488: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 5.6 MiB/s wr, 619 op/s 2024-04-26T18:12:07.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:07 smithi097 ceph-mon[32419]: pgmap v489: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.3 MiB/s wr, 640 op/s 2024-04-26T18:12:07.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:07 smithi186 ceph-mon[32797]: pgmap v489: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.3 MiB/s wr, 640 op/s 2024-04-26T18:12:07.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:07 smithi031 ceph-mon[29164]: pgmap v489: 97 pgs: 97 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.3 MiB/s wr, 640 op/s 2024-04-26T18:12:08.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 885.0 s, 1400.8 tps, lat 71.220 ms stddev 154.341 2024-04-26T18:12:08.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:08 smithi097 ceph-mon[32419]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T18:12:08.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:08 smithi186 ceph-mon[32797]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T18:12:08.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:08 smithi031 ceph-mon[29164]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T18:12:09.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:09 smithi097 ceph-mon[32419]: pgmap v491: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 18 MiB/s wr, 1.61k op/s 2024-04-26T18:12:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:09 smithi186 ceph-mon[32797]: pgmap v491: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 18 MiB/s wr, 1.61k op/s 2024-04-26T18:12:09.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:09 smithi031 ceph-mon[29164]: pgmap v491: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 18 MiB/s wr, 1.61k op/s 2024-04-26T18:12:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:09.967+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:09.968+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:12:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:09.968+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:12:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:09.968+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:12:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:09.968+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:12:11.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:11 smithi097 ceph-mon[32419]: pgmap v492: 97 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 89 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 20 MiB/s wr, 1.77k op/s 2024-04-26T18:12:11.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:11 smithi186 ceph-mon[32797]: pgmap v492: 97 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 89 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 20 MiB/s wr, 1.77k op/s 2024-04-26T18:12:11.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:11 smithi031 ceph-mon[29164]: pgmap v492: 97 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 89 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 20 MiB/s wr, 1.77k op/s 2024-04-26T18:12:13.095 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 890.0 s, 1982.0 tps, lat 51.872 ms stddev 73.020 2024-04-26T18:12:13.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:13 smithi097 ceph-mon[32419]: pgmap v493: 97 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 89 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 20 MiB/s wr, 1.71k op/s 2024-04-26T18:12:13.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:13 smithi186 ceph-mon[32797]: pgmap v493: 97 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 89 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 20 MiB/s wr, 1.71k op/s 2024-04-26T18:12:13.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:13 smithi031 ceph-mon[29164]: pgmap v493: 97 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 89 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 20 MiB/s wr, 1.71k op/s 2024-04-26T18:12:15.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:14 smithi097 ceph-mon[32419]: pgmap v494: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 21 MiB/s wr, 1.77k op/s 2024-04-26T18:12:15.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:14 smithi186 ceph-mon[32797]: pgmap v494: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 21 MiB/s wr, 1.77k op/s 2024-04-26T18:12:15.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:14 smithi031 ceph-mon[29164]: pgmap v494: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 21 MiB/s wr, 1.77k op/s 2024-04-26T18:12:17.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:17 smithi186 ceph-mon[32797]: pgmap v495: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 20 MiB/s wr, 1.68k op/s 2024-04-26T18:12:17.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:17 smithi031 ceph-mon[29164]: pgmap v495: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 20 MiB/s wr, 1.68k op/s 2024-04-26T18:12:17.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:17 smithi097 ceph-mon[32419]: pgmap v495: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 20 MiB/s wr, 1.68k op/s 2024-04-26T18:12:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 895.0 s, 1807.8 tps, lat 55.896 ms stddev 73.586 2024-04-26T18:12:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:19 smithi186 ceph-mon[32797]: pgmap v496: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.7 MiB/s wr, 807 op/s 2024-04-26T18:12:19.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:19 smithi186 ceph-mon[32797]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T18:12:19.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:19 smithi031 ceph-mon[29164]: pgmap v496: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.7 MiB/s wr, 807 op/s 2024-04-26T18:12:19.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:19 smithi031 ceph-mon[29164]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T18:12:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:19 smithi097 ceph-mon[32419]: pgmap v496: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.7 MiB/s wr, 807 op/s 2024-04-26T18:12:19.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:19 smithi097 ceph-mon[32419]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T18:12:21.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:21 smithi186 ceph-mon[32797]: pgmap v498: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 6.4 MiB/s wr, 602 op/s 2024-04-26T18:12:21.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:21 smithi031 ceph-mon[29164]: pgmap v498: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 6.4 MiB/s wr, 602 op/s 2024-04-26T18:12:21.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:21 smithi097 ceph-mon[32419]: pgmap v498: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 6.4 MiB/s wr, 602 op/s 2024-04-26T18:12:23.095 INFO:teuthology.orchestra.run.smithi031.stderr:progress: 900.0 s, 2015.6 tps, lat 49.506 ms stddev 53.670 2024-04-26T18:12:23.304 INFO:teuthology.orchestra.run.smithi031.stdout:transaction type: 2024-04-26T18:12:23.304 INFO:teuthology.orchestra.run.smithi031.stdout:scaling factor: 32 2024-04-26T18:12:23.304 INFO:teuthology.orchestra.run.smithi031.stdout:query mode: simple 2024-04-26T18:12:23.304 INFO:teuthology.orchestra.run.smithi031.stdout:number of clients: 100 2024-04-26T18:12:23.304 INFO:teuthology.orchestra.run.smithi031.stdout:number of threads: 4 2024-04-26T18:12:23.304 INFO:teuthology.orchestra.run.smithi031.stdout:duration: 900 s 2024-04-26T18:12:23.304 INFO:teuthology.orchestra.run.smithi031.stdout:number of transactions actually processed: 1689892 2024-04-26T18:12:23.304 INFO:teuthology.orchestra.run.smithi031.stdout:latency average = 53.255 ms 2024-04-26T18:12:23.304 INFO:teuthology.orchestra.run.smithi031.stdout:latency stddev = 581.762 ms 2024-04-26T18:12:23.304 INFO:teuthology.orchestra.run.smithi031.stdout:tps = 1877.221305 (including connections establishing) 2024-04-26T18:12:23.305 INFO:teuthology.orchestra.run.smithi031.stdout:tps = 1877.226994 (excluding connections establishing) 2024-04-26T18:12:23.307 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl stop postgresql' 2024-04-26T18:12:23.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:23 smithi186 ceph-mon[32797]: pgmap v499: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 7.3 MiB/s wr, 656 op/s 2024-04-26T18:12:23.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:23 smithi031 ceph-mon[29164]: pgmap v499: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 7.3 MiB/s wr, 656 op/s 2024-04-26T18:12:23.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:23 smithi097 ceph-mon[32419]: pgmap v499: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 7.3 MiB/s wr, 656 op/s 2024-04-26T18:12:24.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:24 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:12:24.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:24 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:12:24.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:24 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:12:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:24.975+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:24.976+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:12:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:24.976+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:12:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:24.976+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:12:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:24.976+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:12:25.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:25 smithi186 ceph-mon[32797]: pgmap v500: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.1 MiB/s wr, 601 op/s 2024-04-26T18:12:25.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:25 smithi031 ceph-mon[29164]: pgmap v500: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.1 MiB/s wr, 601 op/s 2024-04-26T18:12:25.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:25 smithi097 ceph-mon[32419]: pgmap v500: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.1 MiB/s wr, 601 op/s 2024-04-26T18:12:27.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:27 smithi186 ceph-mon[32797]: pgmap v501: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.8 MiB/s wr, 663 op/s 2024-04-26T18:12:27.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:27 smithi031 ceph-mon[29164]: pgmap v501: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.8 MiB/s wr, 663 op/s 2024-04-26T18:12:27.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:27 smithi097 ceph-mon[32419]: pgmap v501: 97 pgs: 97 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.8 MiB/s wr, 663 op/s 2024-04-26T18:12:29.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:29.424+0000 7f75d17fa640 1 -- 172.21.15.31:0/865661195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f75b4005b80 con 0x7f75dc11c110 2024-04-26T18:12:29.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:29.425+0000 7f75d27fc640 1 -- 172.21.15.31:0/865661195 <== mon.0 v2:172.21.15.31:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f75b4005b80 con 0x7f75dc11c110 2024-04-26T18:12:29.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:29.440+0000 7f75bb7fe640 1 -- 172.21.15.31:0/164021214 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f75ac005b50 con 0x556108f4ff00 2024-04-26T18:12:29.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:29.441+0000 7f75c4ff9640 1 -- 172.21.15.31:0/164021214 <== mon.1 v2:172.21.15.186:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f75ac005b50 con 0x556108f4ff00 2024-04-26T18:12:29.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:29 smithi186 ceph-mon[32797]: pgmap v502: 97 pgs: 97 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.8 MiB/s wr, 943 op/s 2024-04-26T18:12:29.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:29 smithi031 ceph-mon[29164]: pgmap v502: 97 pgs: 97 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.8 MiB/s wr, 943 op/s 2024-04-26T18:12:29.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:29 smithi097 ceph-mon[32419]: pgmap v502: 97 pgs: 97 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.8 MiB/s wr, 943 op/s 2024-04-26T18:12:31.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:31 smithi186 ceph-mon[32797]: pgmap v503: 97 pgs: 97 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.8 MiB/s wr, 876 op/s 2024-04-26T18:12:31.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:31 smithi031 ceph-mon[29164]: pgmap v503: 97 pgs: 97 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.8 MiB/s wr, 876 op/s 2024-04-26T18:12:31.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:31 smithi097 ceph-mon[32419]: pgmap v503: 97 pgs: 97 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.8 MiB/s wr, 876 op/s 2024-04-26T18:12:32.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-26T18:12:32.795 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/: 2024-04-26T18:12:32.795 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.795 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 . 2024-04-26T18:12:32.795 INFO:teuthology.orchestra.run.smithi031.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 .. 2024-04-26T18:12:32.795 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 data 2024-04-26T18:12:32.795 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.795 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data: 2024-04-26T18:12:32.795 INFO:teuthology.orchestra.run.smithi031.stdout:total 36 2024-04-26T18:12:32.795 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 . 2024-04-26T18:12:32.795 INFO:teuthology.orchestra.run.smithi031.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 .. 2024-04-26T18:12:32.796 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 26 17:57 base 2024-04-26T18:12:32.796 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 30 Apr 26 17:56 current_logfiles 2024-04-26T18:12:32.796 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Apr 26 17:56 global 2024-04-26T18:12:32.796 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 log 2024-04-26T18:12:32.796 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_commit_ts 2024-04-26T18:12:32.796 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_dynshmem 2024-04-26T18:12:32.796 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 26 17:56 pg_hba.conf 2024-04-26T18:12:32.796 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 26 17:56 pg_ident.conf 2024-04-26T18:12:32.796 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 18:12 pg_logical 2024-04-26T18:12:32.796 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 17:56 pg_multixact 2024-04-26T18:12:32.796 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_notify 2024-04-26T18:12:32.797 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_replslot 2024-04-26T18:12:32.797 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_serial 2024-04-26T18:12:32.797 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_snapshots 2024-04-26T18:12:32.797 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 18:12 pg_stat 2024-04-26T18:12:32.797 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 18:12 pg_stat_tmp 2024-04-26T18:12:32.797 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 18:12 pg_subtrans 2024-04-26T18:12:32.797 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_tblspc 2024-04-26T18:12:32.798 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 pg_twophase 2024-04-26T18:12:32.798 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 PG_VERSION 2024-04-26T18:12:32.798 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 26 18:12 pg_wal 2024-04-26T18:12:32.798 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 18:10 pg_xact 2024-04-26T18:12:32.798 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 26 17:56 postgresql.auto.conf 2024-04-26T18:12:32.799 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 26 17:56 postgresql.conf 2024-04-26T18:12:32.799 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 51 Apr 26 17:56 postmaster.opts 2024-04-26T18:12:32.799 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.799 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/base: 2024-04-26T18:12:32.799 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.799 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 26 17:57 . 2024-04-26T18:12:32.799 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.800 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 17:56 1 2024-04-26T18:12:32.800 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 17:56 13493 2024-04-26T18:12:32.800 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Apr 26 18:02 13494 2024-04-26T18:12:32.800 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:57 pgsql_tmp 2024-04-26T18:12:32.800 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.800 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-26T18:12:32.801 INFO:teuthology.orchestra.run.smithi031.stdout:total 7777 2024-04-26T18:12:32.801 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 17:56 . 2024-04-26T18:12:32.801 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 26 17:57 .. 2024-04-26T18:12:32.801 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 112 2024-04-26T18:12:32.801 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 113 2024-04-26T18:12:32.801 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 17:56 1247 2024-04-26T18:12:32.802 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1247_fsm 2024-04-26T18:12:32.802 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1247_vm 2024-04-26T18:12:32.802 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 17:56 1249 2024-04-26T18:12:32.802 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1249_fsm 2024-04-26T18:12:32.802 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1249_vm 2024-04-26T18:12:32.802 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 17:56 1255 2024-04-26T18:12:32.803 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1255_fsm 2024-04-26T18:12:32.803 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1255_vm 2024-04-26T18:12:32.803 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 17:56 1259 2024-04-26T18:12:32.803 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1259_fsm 2024-04-26T18:12:32.803 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1259_vm 2024-04-26T18:12:32.803 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 17:56 13344 2024-04-26T18:12:32.803 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13344_fsm 2024-04-26T18:12:32.804 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13344_vm 2024-04-26T18:12:32.804 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13346 2024-04-26T18:12:32.804 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13348 2024-04-26T18:12:32.804 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13349 2024-04-26T18:12:32.804 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13349_fsm 2024-04-26T18:12:32.804 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13349_vm 2024-04-26T18:12:32.805 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13351 2024-04-26T18:12:32.805 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13353 2024-04-26T18:12:32.805 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13354 2024-04-26T18:12:32.805 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13354_fsm 2024-04-26T18:12:32.805 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13354_vm 2024-04-26T18:12:32.805 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13356 2024-04-26T18:12:32.805 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13358 2024-04-26T18:12:32.806 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13359 2024-04-26T18:12:32.806 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13359_fsm 2024-04-26T18:12:32.806 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13359_vm 2024-04-26T18:12:32.806 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13361 2024-04-26T18:12:32.806 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13363 2024-04-26T18:12:32.806 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 1417 2024-04-26T18:12:32.807 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 1418 2024-04-26T18:12:32.807 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 174 2024-04-26T18:12:32.807 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 175 2024-04-26T18:12:32.807 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2187 2024-04-26T18:12:32.807 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2224 2024-04-26T18:12:32.807 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2328 2024-04-26T18:12:32.807 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2336 2024-04-26T18:12:32.808 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2337 2024-04-26T18:12:32.808 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2579 2024-04-26T18:12:32.808 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2600 2024-04-26T18:12:32.808 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2600_fsm 2024-04-26T18:12:32.808 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2600_vm 2024-04-26T18:12:32.809 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2601 2024-04-26T18:12:32.809 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2601_fsm 2024-04-26T18:12:32.809 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2601_vm 2024-04-26T18:12:32.809 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2602 2024-04-26T18:12:32.809 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2602_fsm 2024-04-26T18:12:32.809 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2602_vm 2024-04-26T18:12:32.809 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2603 2024-04-26T18:12:32.810 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2603_fsm 2024-04-26T18:12:32.810 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2603_vm 2024-04-26T18:12:32.810 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2604 2024-04-26T18:12:32.810 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2605 2024-04-26T18:12:32.810 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2605_fsm 2024-04-26T18:12:32.810 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2605_vm 2024-04-26T18:12:32.811 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2606 2024-04-26T18:12:32.811 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2606_fsm 2024-04-26T18:12:32.811 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2606_vm 2024-04-26T18:12:32.811 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2607 2024-04-26T18:12:32.811 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2607_fsm 2024-04-26T18:12:32.812 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2607_vm 2024-04-26T18:12:32.812 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 17:56 2608 2024-04-26T18:12:32.812 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2608_fsm 2024-04-26T18:12:32.812 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2608_vm 2024-04-26T18:12:32.812 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 17:56 2609 2024-04-26T18:12:32.812 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2609_fsm 2024-04-26T18:12:32.813 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2609_vm 2024-04-26T18:12:32.813 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2610 2024-04-26T18:12:32.813 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2610_fsm 2024-04-26T18:12:32.813 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2610_vm 2024-04-26T18:12:32.813 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2611 2024-04-26T18:12:32.813 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2612 2024-04-26T18:12:32.814 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2612_fsm 2024-04-26T18:12:32.814 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2612_vm 2024-04-26T18:12:32.814 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2613 2024-04-26T18:12:32.814 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2615 2024-04-26T18:12:32.814 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2615_fsm 2024-04-26T18:12:32.814 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2615_vm 2024-04-26T18:12:32.815 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2616 2024-04-26T18:12:32.815 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2616_fsm 2024-04-26T18:12:32.815 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2616_vm 2024-04-26T18:12:32.815 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 17:56 2617 2024-04-26T18:12:32.815 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2617_fsm 2024-04-26T18:12:32.815 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2617_vm 2024-04-26T18:12:32.815 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 17:56 2618 2024-04-26T18:12:32.816 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2618_fsm 2024-04-26T18:12:32.816 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2618_vm 2024-04-26T18:12:32.816 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 26 17:56 2619 2024-04-26T18:12:32.816 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2619_fsm 2024-04-26T18:12:32.816 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2619_vm 2024-04-26T18:12:32.816 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2620 2024-04-26T18:12:32.816 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2650 2024-04-26T18:12:32.817 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2651 2024-04-26T18:12:32.817 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2652 2024-04-26T18:12:32.817 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2653 2024-04-26T18:12:32.817 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2654 2024-04-26T18:12:32.817 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2655 2024-04-26T18:12:32.817 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2656 2024-04-26T18:12:32.818 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2657 2024-04-26T18:12:32.818 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 17:56 2658 2024-04-26T18:12:32.818 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 17:56 2659 2024-04-26T18:12:32.818 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2660 2024-04-26T18:12:32.818 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2661 2024-04-26T18:12:32.819 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2662 2024-04-26T18:12:32.819 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2663 2024-04-26T18:12:32.819 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2664 2024-04-26T18:12:32.819 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2665 2024-04-26T18:12:32.819 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2666 2024-04-26T18:12:32.819 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2667 2024-04-26T18:12:32.820 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2668 2024-04-26T18:12:32.820 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2669 2024-04-26T18:12:32.820 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2670 2024-04-26T18:12:32.820 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 17:56 2673 2024-04-26T18:12:32.820 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 17:56 2674 2024-04-26T18:12:32.820 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 17:56 2675 2024-04-26T18:12:32.821 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2678 2024-04-26T18:12:32.821 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2679 2024-04-26T18:12:32.821 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2680 2024-04-26T18:12:32.821 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2681 2024-04-26T18:12:32.821 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2682 2024-04-26T18:12:32.821 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2683 2024-04-26T18:12:32.822 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2684 2024-04-26T18:12:32.822 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2685 2024-04-26T18:12:32.822 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2686 2024-04-26T18:12:32.822 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2687 2024-04-26T18:12:32.822 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2688 2024-04-26T18:12:32.823 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2689 2024-04-26T18:12:32.823 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 17:56 2690 2024-04-26T18:12:32.823 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 17:56 2691 2024-04-26T18:12:32.823 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2692 2024-04-26T18:12:32.823 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2693 2024-04-26T18:12:32.824 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2696 2024-04-26T18:12:32.824 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2699 2024-04-26T18:12:32.824 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2701 2024-04-26T18:12:32.824 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2702 2024-04-26T18:12:32.824 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2703 2024-04-26T18:12:32.825 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2704 2024-04-26T18:12:32.825 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2753 2024-04-26T18:12:32.825 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2753_fsm 2024-04-26T18:12:32.825 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2753_vm 2024-04-26T18:12:32.825 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2754 2024-04-26T18:12:32.826 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2755 2024-04-26T18:12:32.826 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2756 2024-04-26T18:12:32.826 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2757 2024-04-26T18:12:32.826 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2830 2024-04-26T18:12:32.829 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2831 2024-04-26T18:12:32.829 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2832 2024-04-26T18:12:32.829 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2833 2024-04-26T18:12:32.829 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2834 2024-04-26T18:12:32.830 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2835 2024-04-26T18:12:32.830 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2836 2024-04-26T18:12:32.830 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2837 2024-04-26T18:12:32.830 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 17:56 2838 2024-04-26T18:12:32.830 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2838_fsm 2024-04-26T18:12:32.830 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2838_vm 2024-04-26T18:12:32.831 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2839 2024-04-26T18:12:32.831 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2840 2024-04-26T18:12:32.831 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2840_fsm 2024-04-26T18:12:32.831 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2840_vm 2024-04-26T18:12:32.831 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2841 2024-04-26T18:12:32.831 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2995 2024-04-26T18:12:32.832 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2996 2024-04-26T18:12:32.832 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3079 2024-04-26T18:12:32.832 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3079_fsm 2024-04-26T18:12:32.832 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3079_vm 2024-04-26T18:12:32.832 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3080 2024-04-26T18:12:32.832 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3081 2024-04-26T18:12:32.832 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 3085 2024-04-26T18:12:32.833 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3118 2024-04-26T18:12:32.833 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3119 2024-04-26T18:12:32.833 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 17:56 3164 2024-04-26T18:12:32.833 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3256 2024-04-26T18:12:32.833 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3257 2024-04-26T18:12:32.833 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3258 2024-04-26T18:12:32.834 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3350 2024-04-26T18:12:32.834 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3351 2024-04-26T18:12:32.834 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3379 2024-04-26T18:12:32.834 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3380 2024-04-26T18:12:32.834 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3381 2024-04-26T18:12:32.834 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3394 2024-04-26T18:12:32.835 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3394_fsm 2024-04-26T18:12:32.835 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3394_vm 2024-04-26T18:12:32.835 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3395 2024-04-26T18:12:32.835 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3429 2024-04-26T18:12:32.835 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3430 2024-04-26T18:12:32.835 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3431 2024-04-26T18:12:32.836 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3433 2024-04-26T18:12:32.836 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3439 2024-04-26T18:12:32.836 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3440 2024-04-26T18:12:32.836 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 3455 2024-04-26T18:12:32.836 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 17:56 3456 2024-04-26T18:12:32.836 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3456_fsm 2024-04-26T18:12:32.837 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3456_vm 2024-04-26T18:12:32.837 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3466 2024-04-26T18:12:32.837 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3467 2024-04-26T18:12:32.837 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3468 2024-04-26T18:12:32.837 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3501 2024-04-26T18:12:32.837 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3502 2024-04-26T18:12:32.838 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3503 2024-04-26T18:12:32.838 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3534 2024-04-26T18:12:32.838 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3541 2024-04-26T18:12:32.838 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3541_fsm 2024-04-26T18:12:32.838 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3541_vm 2024-04-26T18:12:32.838 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3542 2024-04-26T18:12:32.839 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3574 2024-04-26T18:12:32.839 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3575 2024-04-26T18:12:32.839 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3576 2024-04-26T18:12:32.839 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3596 2024-04-26T18:12:32.839 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3597 2024-04-26T18:12:32.839 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3598 2024-04-26T18:12:32.839 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3599 2024-04-26T18:12:32.840 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3600 2024-04-26T18:12:32.840 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3600_fsm 2024-04-26T18:12:32.840 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3600_vm 2024-04-26T18:12:32.840 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3601 2024-04-26T18:12:32.840 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3601_fsm 2024-04-26T18:12:32.840 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3601_vm 2024-04-26T18:12:32.841 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3602 2024-04-26T18:12:32.841 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3602_fsm 2024-04-26T18:12:32.841 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3602_vm 2024-04-26T18:12:32.841 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3603 2024-04-26T18:12:32.841 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3603_fsm 2024-04-26T18:12:32.841 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3603_vm 2024-04-26T18:12:32.841 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3604 2024-04-26T18:12:32.842 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3605 2024-04-26T18:12:32.842 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3606 2024-04-26T18:12:32.842 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3607 2024-04-26T18:12:32.842 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3608 2024-04-26T18:12:32.842 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 3609 2024-04-26T18:12:32.842 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3712 2024-04-26T18:12:32.843 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3764 2024-04-26T18:12:32.843 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3764_fsm 2024-04-26T18:12:32.843 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3764_vm 2024-04-26T18:12:32.843 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3766 2024-04-26T18:12:32.843 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3767 2024-04-26T18:12:32.843 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3997 2024-04-26T18:12:32.844 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4143 2024-04-26T18:12:32.844 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4144 2024-04-26T18:12:32.844 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4145 2024-04-26T18:12:32.844 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4146 2024-04-26T18:12:32.844 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4147 2024-04-26T18:12:32.844 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4148 2024-04-26T18:12:32.845 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4149 2024-04-26T18:12:32.845 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4150 2024-04-26T18:12:32.845 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4151 2024-04-26T18:12:32.845 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4152 2024-04-26T18:12:32.845 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4153 2024-04-26T18:12:32.845 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4154 2024-04-26T18:12:32.845 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4155 2024-04-26T18:12:32.845 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4156 2024-04-26T18:12:32.845 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4157 2024-04-26T18:12:32.845 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4158 2024-04-26T18:12:32.846 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4159 2024-04-26T18:12:32.846 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4160 2024-04-26T18:12:32.846 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4161 2024-04-26T18:12:32.846 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4162 2024-04-26T18:12:32.849 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4163 2024-04-26T18:12:32.850 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4164 2024-04-26T18:12:32.850 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4165 2024-04-26T18:12:32.850 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4166 2024-04-26T18:12:32.850 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4167 2024-04-26T18:12:32.850 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4168 2024-04-26T18:12:32.850 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4169 2024-04-26T18:12:32.850 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4170 2024-04-26T18:12:32.850 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4171 2024-04-26T18:12:32.851 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4172 2024-04-26T18:12:32.851 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4173 2024-04-26T18:12:32.851 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4174 2024-04-26T18:12:32.851 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 5002 2024-04-26T18:12:32.851 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 548 2024-04-26T18:12:32.851 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 549 2024-04-26T18:12:32.851 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6102 2024-04-26T18:12:32.851 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6104 2024-04-26T18:12:32.851 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6106 2024-04-26T18:12:32.851 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6110 2024-04-26T18:12:32.851 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6111 2024-04-26T18:12:32.851 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6112 2024-04-26T18:12:32.852 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6113 2024-04-26T18:12:32.852 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6117 2024-04-26T18:12:32.852 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 826 2024-04-26T18:12:32.852 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 827 2024-04-26T18:12:32.852 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 828 2024-04-26T18:12:32.852 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 17:56 pg_filenode.map 2024-04-26T18:12:32.852 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 PG_VERSION 2024-04-26T18:12:32.852 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.852 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-26T18:12:32.852 INFO:teuthology.orchestra.run.smithi031.stdout:total 7777 2024-04-26T18:12:32.852 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 26 17:56 . 2024-04-26T18:12:32.852 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 26 17:57 .. 2024-04-26T18:12:32.852 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 112 2024-04-26T18:12:32.852 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 113 2024-04-26T18:12:32.853 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 17:56 1247 2024-04-26T18:12:32.853 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1247_fsm 2024-04-26T18:12:32.853 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1247_vm 2024-04-26T18:12:32.853 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 17:56 1249 2024-04-26T18:12:32.853 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1249_fsm 2024-04-26T18:12:32.853 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1249_vm 2024-04-26T18:12:32.853 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 17:56 1255 2024-04-26T18:12:32.853 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1255_fsm 2024-04-26T18:12:32.853 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1255_vm 2024-04-26T18:12:32.853 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 17:56 1259 2024-04-26T18:12:32.853 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1259_fsm 2024-04-26T18:12:32.853 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1259_vm 2024-04-26T18:12:32.853 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 17:56 13344 2024-04-26T18:12:32.853 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13344_fsm 2024-04-26T18:12:32.854 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13344_vm 2024-04-26T18:12:32.854 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13346 2024-04-26T18:12:32.854 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13348 2024-04-26T18:12:32.854 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13349 2024-04-26T18:12:32.854 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13349_fsm 2024-04-26T18:12:32.854 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13349_vm 2024-04-26T18:12:32.854 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13351 2024-04-26T18:12:32.854 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13353 2024-04-26T18:12:32.854 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13354 2024-04-26T18:12:32.854 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13354_fsm 2024-04-26T18:12:32.854 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13354_vm 2024-04-26T18:12:32.854 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13356 2024-04-26T18:12:32.854 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13358 2024-04-26T18:12:32.854 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13359 2024-04-26T18:12:32.855 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13359_fsm 2024-04-26T18:12:32.855 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13359_vm 2024-04-26T18:12:32.855 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13361 2024-04-26T18:12:32.855 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13363 2024-04-26T18:12:32.855 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 1417 2024-04-26T18:12:32.855 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 1418 2024-04-26T18:12:32.855 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 174 2024-04-26T18:12:32.855 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 175 2024-04-26T18:12:32.855 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2187 2024-04-26T18:12:32.855 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2224 2024-04-26T18:12:32.855 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2328 2024-04-26T18:12:32.855 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2336 2024-04-26T18:12:32.856 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2337 2024-04-26T18:12:32.856 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2579 2024-04-26T18:12:32.856 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2600 2024-04-26T18:12:32.856 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2600_fsm 2024-04-26T18:12:32.856 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2600_vm 2024-04-26T18:12:32.856 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2601 2024-04-26T18:12:32.856 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2601_fsm 2024-04-26T18:12:32.856 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2601_vm 2024-04-26T18:12:32.856 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2602 2024-04-26T18:12:32.857 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2602_fsm 2024-04-26T18:12:32.857 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2602_vm 2024-04-26T18:12:32.857 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2603 2024-04-26T18:12:32.857 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2603_fsm 2024-04-26T18:12:32.857 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2603_vm 2024-04-26T18:12:32.857 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2604 2024-04-26T18:12:32.857 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2605 2024-04-26T18:12:32.857 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2605_fsm 2024-04-26T18:12:32.857 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2605_vm 2024-04-26T18:12:32.857 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2606 2024-04-26T18:12:32.857 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2606_fsm 2024-04-26T18:12:32.857 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2606_vm 2024-04-26T18:12:32.857 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2607 2024-04-26T18:12:32.857 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2607_fsm 2024-04-26T18:12:32.858 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2607_vm 2024-04-26T18:12:32.858 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 17:56 2608 2024-04-26T18:12:32.858 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2608_fsm 2024-04-26T18:12:32.858 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2608_vm 2024-04-26T18:12:32.858 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 17:56 2609 2024-04-26T18:12:32.858 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2609_fsm 2024-04-26T18:12:32.858 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2609_vm 2024-04-26T18:12:32.858 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2610 2024-04-26T18:12:32.858 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2610_fsm 2024-04-26T18:12:32.858 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2610_vm 2024-04-26T18:12:32.858 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2611 2024-04-26T18:12:32.858 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2612 2024-04-26T18:12:32.858 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2612_fsm 2024-04-26T18:12:32.859 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2612_vm 2024-04-26T18:12:32.859 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2613 2024-04-26T18:12:32.859 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2615 2024-04-26T18:12:32.859 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2615_fsm 2024-04-26T18:12:32.859 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2615_vm 2024-04-26T18:12:32.859 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2616 2024-04-26T18:12:32.859 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2616_fsm 2024-04-26T18:12:32.859 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2616_vm 2024-04-26T18:12:32.860 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 17:56 2617 2024-04-26T18:12:32.860 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2617_fsm 2024-04-26T18:12:32.860 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2617_vm 2024-04-26T18:12:32.860 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 17:56 2618 2024-04-26T18:12:32.860 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2618_fsm 2024-04-26T18:12:32.860 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2618_vm 2024-04-26T18:12:32.860 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 26 17:56 2619 2024-04-26T18:12:32.860 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2619_fsm 2024-04-26T18:12:32.860 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2619_vm 2024-04-26T18:12:32.860 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2620 2024-04-26T18:12:32.860 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2650 2024-04-26T18:12:32.860 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2651 2024-04-26T18:12:32.860 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2652 2024-04-26T18:12:32.860 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2653 2024-04-26T18:12:32.861 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2654 2024-04-26T18:12:32.861 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2655 2024-04-26T18:12:32.861 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2656 2024-04-26T18:12:32.861 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2657 2024-04-26T18:12:32.861 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 17:56 2658 2024-04-26T18:12:32.861 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 17:56 2659 2024-04-26T18:12:32.861 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2660 2024-04-26T18:12:32.861 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2661 2024-04-26T18:12:32.861 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2662 2024-04-26T18:12:32.861 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2663 2024-04-26T18:12:32.861 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2664 2024-04-26T18:12:32.861 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2665 2024-04-26T18:12:32.861 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2666 2024-04-26T18:12:32.861 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2667 2024-04-26T18:12:32.862 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2668 2024-04-26T18:12:32.862 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2669 2024-04-26T18:12:32.862 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2670 2024-04-26T18:12:32.862 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 17:56 2673 2024-04-26T18:12:32.862 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 17:56 2674 2024-04-26T18:12:32.862 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 17:56 2675 2024-04-26T18:12:32.862 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2678 2024-04-26T18:12:32.862 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2679 2024-04-26T18:12:32.862 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2680 2024-04-26T18:12:32.862 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2681 2024-04-26T18:12:32.862 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2682 2024-04-26T18:12:32.862 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2683 2024-04-26T18:12:32.862 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2684 2024-04-26T18:12:32.862 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2685 2024-04-26T18:12:32.863 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2686 2024-04-26T18:12:32.863 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2687 2024-04-26T18:12:32.863 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2688 2024-04-26T18:12:32.863 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2689 2024-04-26T18:12:32.863 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 17:56 2690 2024-04-26T18:12:32.863 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 17:56 2691 2024-04-26T18:12:32.863 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2692 2024-04-26T18:12:32.863 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2693 2024-04-26T18:12:32.863 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2696 2024-04-26T18:12:32.863 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2699 2024-04-26T18:12:32.863 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2701 2024-04-26T18:12:32.863 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2702 2024-04-26T18:12:32.863 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2703 2024-04-26T18:12:32.864 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2704 2024-04-26T18:12:32.864 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2753 2024-04-26T18:12:32.864 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2753_fsm 2024-04-26T18:12:32.864 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2753_vm 2024-04-26T18:12:32.864 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2754 2024-04-26T18:12:32.864 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2755 2024-04-26T18:12:32.864 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2756 2024-04-26T18:12:32.864 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2757 2024-04-26T18:12:32.864 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2830 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2831 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2832 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2833 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2834 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2835 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2836 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2837 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 17:56 2838 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2838_fsm 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2838_vm 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2839 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2840 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2840_fsm 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2840_vm 2024-04-26T18:12:32.865 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2841 2024-04-26T18:12:32.866 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2995 2024-04-26T18:12:32.866 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2996 2024-04-26T18:12:32.866 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3079 2024-04-26T18:12:32.866 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3079_fsm 2024-04-26T18:12:32.866 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3079_vm 2024-04-26T18:12:32.866 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3080 2024-04-26T18:12:32.866 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3081 2024-04-26T18:12:32.866 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 3085 2024-04-26T18:12:32.866 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3118 2024-04-26T18:12:32.866 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3119 2024-04-26T18:12:32.866 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 17:56 3164 2024-04-26T18:12:32.866 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3256 2024-04-26T18:12:32.866 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3257 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3258 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3350 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3351 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3379 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3380 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3381 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3394 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3394_fsm 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3394_vm 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3395 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3429 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3430 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3431 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3433 2024-04-26T18:12:32.867 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3439 2024-04-26T18:12:32.868 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3440 2024-04-26T18:12:32.868 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 3455 2024-04-26T18:12:32.868 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 17:56 3456 2024-04-26T18:12:32.868 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3456_fsm 2024-04-26T18:12:32.868 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3456_vm 2024-04-26T18:12:32.868 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3466 2024-04-26T18:12:32.868 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3467 2024-04-26T18:12:32.868 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3468 2024-04-26T18:12:32.868 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3501 2024-04-26T18:12:32.868 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3502 2024-04-26T18:12:32.868 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3503 2024-04-26T18:12:32.868 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3534 2024-04-26T18:12:32.868 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3541 2024-04-26T18:12:32.868 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3541_fsm 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3541_vm 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3542 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3574 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3575 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3576 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3596 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3597 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3598 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3599 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3600 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3600_fsm 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3600_vm 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3601 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3601_fsm 2024-04-26T18:12:32.869 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3601_vm 2024-04-26T18:12:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3602 2024-04-26T18:12:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3602_fsm 2024-04-26T18:12:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3602_vm 2024-04-26T18:12:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3603 2024-04-26T18:12:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3603_fsm 2024-04-26T18:12:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3603_vm 2024-04-26T18:12:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3604 2024-04-26T18:12:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3605 2024-04-26T18:12:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3606 2024-04-26T18:12:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3607 2024-04-26T18:12:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3608 2024-04-26T18:12:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 3609 2024-04-26T18:12:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3712 2024-04-26T18:12:32.870 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3764 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3764_fsm 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3764_vm 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3766 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3767 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3997 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4143 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4144 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4145 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4146 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4147 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4148 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4149 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4150 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4151 2024-04-26T18:12:32.871 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4152 2024-04-26T18:12:32.872 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4153 2024-04-26T18:12:32.872 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4154 2024-04-26T18:12:32.872 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4155 2024-04-26T18:12:32.872 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4156 2024-04-26T18:12:32.872 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4157 2024-04-26T18:12:32.872 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4158 2024-04-26T18:12:32.872 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4159 2024-04-26T18:12:32.872 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4160 2024-04-26T18:12:32.872 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4161 2024-04-26T18:12:32.872 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4162 2024-04-26T18:12:32.872 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4163 2024-04-26T18:12:32.872 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4164 2024-04-26T18:12:32.872 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4165 2024-04-26T18:12:32.872 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4166 2024-04-26T18:12:32.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4167 2024-04-26T18:12:32.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4168 2024-04-26T18:12:32.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4169 2024-04-26T18:12:32.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4170 2024-04-26T18:12:32.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4171 2024-04-26T18:12:32.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4172 2024-04-26T18:12:32.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4173 2024-04-26T18:12:32.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4174 2024-04-26T18:12:32.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 5002 2024-04-26T18:12:32.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 548 2024-04-26T18:12:32.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 549 2024-04-26T18:12:32.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6102 2024-04-26T18:12:32.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6104 2024-04-26T18:12:32.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6106 2024-04-26T18:12:32.874 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6110 2024-04-26T18:12:32.874 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6111 2024-04-26T18:12:32.874 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6112 2024-04-26T18:12:32.874 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6113 2024-04-26T18:12:32.874 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6117 2024-04-26T18:12:32.874 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 826 2024-04-26T18:12:32.874 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 827 2024-04-26T18:12:32.874 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 828 2024-04-26T18:12:32.874 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 17:56 pg_filenode.map 2024-04-26T18:12:32.874 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 PG_VERSION 2024-04-26T18:12:32.874 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.874 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-26T18:12:32.874 INFO:teuthology.orchestra.run.smithi031.stdout:total 600662 2024-04-26T18:12:32.874 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Apr 26 18:02 . 2024-04-26T18:12:32.875 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 26 17:57 .. 2024-04-26T18:12:32.875 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 112 2024-04-26T18:12:32.875 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 113 2024-04-26T18:12:32.875 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 17:58 1247 2024-04-26T18:12:32.875 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1247_fsm 2024-04-26T18:12:32.875 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:58 1247_vm 2024-04-26T18:12:32.875 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 26 17:58 1249 2024-04-26T18:12:32.875 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:58 1249_fsm 2024-04-26T18:12:32.875 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:58 1249_vm 2024-04-26T18:12:32.875 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 26 17:56 1255 2024-04-26T18:12:32.875 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1255_fsm 2024-04-26T18:12:32.875 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1255_vm 2024-04-26T18:12:32.875 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 26 18:12 1259 2024-04-26T18:12:32.875 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1259_fsm 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:58 1259_vm 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 26 17:56 13344 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13344_fsm 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13344_vm 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13346 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13348 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13349 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13349_fsm 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13349_vm 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13351 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13353 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13354 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13354_fsm 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13354_vm 2024-04-26T18:12:32.876 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13356 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13358 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13359 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 13359_fsm 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13359_vm 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 13361 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 13363 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 1417 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 1418 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 443121664 Apr 26 18:12 16396 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Apr 26 18:12 16396_fsm 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 18:03 16396_vm 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1613824 Apr 26 18:12 16397 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 18:12 16397_fsm 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 18:12 16397_vm 2024-04-26T18:12:32.877 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1638400 Apr 26 18:12 16399 2024-04-26T18:12:32.878 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 18:12 16399_fsm 2024-04-26T18:12:32.878 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 18:12 16399_vm 2024-04-26T18:12:32.878 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 18:12 16400 2024-04-26T18:12:32.878 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 18:12 16402 2024-04-26T18:12:32.878 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 71901184 Apr 26 18:12 16404 2024-04-26T18:12:32.878 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88317952 Apr 26 18:12 16406 2024-04-26T18:12:32.878 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 18:12 16406_fsm 2024-04-26T18:12:32.878 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 18:09 16406_vm 2024-04-26T18:12:32.878 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 174 2024-04-26T18:12:32.878 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 175 2024-04-26T18:12:32.878 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2187 2024-04-26T18:12:32.878 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2224 2024-04-26T18:12:32.878 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2328 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2336 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2337 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:58 2579 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2600 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2600_fsm 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2600_vm 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2601 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2601_fsm 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2601_vm 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2602 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2602_fsm 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2602_vm 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:57 2603 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2603_fsm 2024-04-26T18:12:32.879 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2603_vm 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2604 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2605 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2605_fsm 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2605_vm 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:58 2606 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2606_fsm 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:58 2606_vm 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2607 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2607_fsm 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2607_vm 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 26 17:58 2608 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2608_fsm 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:58 2608_vm 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 26 17:56 2609 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2609_fsm 2024-04-26T18:12:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2609_vm 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:58 2610 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2610_fsm 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:58 2610_vm 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2611 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2612 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2612_fsm 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2612_vm 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2613 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2615 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2615_fsm 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2615_vm 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2616 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2616_fsm 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2616_vm 2024-04-26T18:12:32.881 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 26 17:56 2617 2024-04-26T18:12:32.882 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2617_fsm 2024-04-26T18:12:32.882 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2617_vm 2024-04-26T18:12:32.882 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 26 17:56 2618 2024-04-26T18:12:32.882 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2618_fsm 2024-04-26T18:12:32.882 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2618_vm 2024-04-26T18:12:32.882 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 163840 Apr 26 18:12 2619 2024-04-26T18:12:32.882 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 18:01 2619_fsm 2024-04-26T18:12:32.882 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:58 2619_vm 2024-04-26T18:12:32.882 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2620 2024-04-26T18:12:32.882 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2650 2024-04-26T18:12:32.882 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2651 2024-04-26T18:12:32.882 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2652 2024-04-26T18:12:32.882 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:57 2653 2024-04-26T18:12:32.883 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2654 2024-04-26T18:12:32.883 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:57 2655 2024-04-26T18:12:32.883 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2656 2024-04-26T18:12:32.883 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2657 2024-04-26T18:12:32.883 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 26 17:58 2658 2024-04-26T18:12:32.883 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 26 17:58 2659 2024-04-26T18:12:32.883 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2660 2024-04-26T18:12:32.883 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2661 2024-04-26T18:12:32.883 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:58 2662 2024-04-26T18:12:32.883 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:58 2663 2024-04-26T18:12:32.883 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:58 2664 2024-04-26T18:12:32.883 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:58 2665 2024-04-26T18:12:32.883 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:58 2666 2024-04-26T18:12:32.883 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:58 2667 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2668 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2669 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2670 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 26 17:58 2673 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 26 17:58 2674 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 26 17:56 2675 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:58 2678 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:58 2679 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2680 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2681 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2682 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2683 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2684 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:57 2685 2024-04-26T18:12:32.884 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:57 2686 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2687 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 2688 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 26 17:56 2689 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 26 17:56 2690 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 26 17:56 2691 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2692 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2693 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 18:01 2696 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2699 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2701 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2702 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:58 2703 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:58 2704 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2753 2024-04-26T18:12:32.885 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2753_fsm 2024-04-26T18:12:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2753_vm 2024-04-26T18:12:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2754 2024-04-26T18:12:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2755 2024-04-26T18:12:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2756 2024-04-26T18:12:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 2757 2024-04-26T18:12:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2830 2024-04-26T18:12:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2831 2024-04-26T18:12:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2832 2024-04-26T18:12:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2833 2024-04-26T18:12:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2834 2024-04-26T18:12:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2835 2024-04-26T18:12:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2836 2024-04-26T18:12:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2837 2024-04-26T18:12:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 26 17:56 2838 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2838_fsm 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2838_vm 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2839 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2840 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2840_fsm 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2840_vm 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2841 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2995 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2996 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3079 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3079_fsm 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3079_vm 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3080 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3081 2024-04-26T18:12:32.887 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:56 3085 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3118 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3119 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 26 17:56 3164 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3256 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3257 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3258 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3350 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3351 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3379 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3380 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3381 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3394 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3394_fsm 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3394_vm 2024-04-26T18:12:32.888 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3395 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3429 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3430 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3431 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3433 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3439 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3440 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 26 17:58 3455 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 26 17:56 3456 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3456_fsm 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3456_vm 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3466 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3467 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3468 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3501 2024-04-26T18:12:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3502 2024-04-26T18:12:32.890 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3503 2024-04-26T18:12:32.890 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3534 2024-04-26T18:12:32.890 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3541 2024-04-26T18:12:32.890 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3541_fsm 2024-04-26T18:12:32.890 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3541_vm 2024-04-26T18:12:32.890 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3542 2024-04-26T18:12:32.890 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3574 2024-04-26T18:12:32.890 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3575 2024-04-26T18:12:32.890 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3576 2024-04-26T18:12:32.890 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3596 2024-04-26T18:12:32.891 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3597 2024-04-26T18:12:32.891 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3598 2024-04-26T18:12:32.891 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3599 2024-04-26T18:12:32.891 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3600 2024-04-26T18:12:32.891 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3600_fsm 2024-04-26T18:12:32.891 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3600_vm 2024-04-26T18:12:32.891 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3601 2024-04-26T18:12:32.891 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3601_fsm 2024-04-26T18:12:32.891 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3601_vm 2024-04-26T18:12:32.891 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3602 2024-04-26T18:12:32.891 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3602_fsm 2024-04-26T18:12:32.891 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3602_vm 2024-04-26T18:12:32.891 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3603 2024-04-26T18:12:32.891 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3603_fsm 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3603_vm 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3604 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3605 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3606 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3607 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3608 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 26 17:56 3609 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3712 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3764 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 3764_fsm 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3764_vm 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3766 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 3767 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3997 2024-04-26T18:12:32.892 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4143 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4144 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4145 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4146 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4147 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4148 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4149 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4150 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4151 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4152 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4153 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4154 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4155 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4156 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4157 2024-04-26T18:12:32.893 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4158 2024-04-26T18:12:32.894 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4159 2024-04-26T18:12:32.894 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4160 2024-04-26T18:12:32.894 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4161 2024-04-26T18:12:32.894 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4162 2024-04-26T18:12:32.894 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4163 2024-04-26T18:12:32.894 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4164 2024-04-26T18:12:32.894 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4165 2024-04-26T18:12:32.894 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4166 2024-04-26T18:12:32.894 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4167 2024-04-26T18:12:32.894 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4168 2024-04-26T18:12:32.894 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4169 2024-04-26T18:12:32.894 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4170 2024-04-26T18:12:32.894 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4171 2024-04-26T18:12:32.895 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4172 2024-04-26T18:12:32.895 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4173 2024-04-26T18:12:32.895 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4174 2024-04-26T18:12:32.895 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 5002 2024-04-26T18:12:32.895 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 548 2024-04-26T18:12:32.895 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 549 2024-04-26T18:12:32.895 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6102 2024-04-26T18:12:32.895 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6104 2024-04-26T18:12:32.895 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6106 2024-04-26T18:12:32.895 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6110 2024-04-26T18:12:32.895 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6111 2024-04-26T18:12:32.895 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6112 2024-04-26T18:12:32.895 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6113 2024-04-26T18:12:32.895 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6117 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 826 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 827 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 828 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 17:56 pg_filenode.map 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 151596 Apr 26 17:56 pg_internal.init 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 17:56 PG_VERSION 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/base/pgsql_tmp: 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:57 . 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 26 17:57 .. 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/global: 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout:total 560 2024-04-26T18:12:32.896 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Apr 26 17:56 . 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1213 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1213_fsm 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1213_vm 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1214 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1214_fsm 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1214_vm 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 1232 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 1233 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:57 1260 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1260_fsm 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1260_vm 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1261 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1261_fsm 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1261_vm 2024-04-26T18:12:32.897 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1262 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 1262_fsm 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 1262_vm 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2396 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 26 17:56 2396_fsm 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2396_vm 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2397 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:57 2671 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:57 2672 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2676 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:57 2677 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2694 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2695 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2697 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 26 17:56 2698 2024-04-26T18:12:32.898 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2846 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2847 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2964 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:57 2965 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 2966 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 2967 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 3592 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 3593 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4060 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4061 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4175 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4176 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4177 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4178 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4181 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4182 2024-04-26T18:12:32.899 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4183 2024-04-26T18:12:32.900 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4184 2024-04-26T18:12:32.900 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 4185 2024-04-26T18:12:32.901 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 4186 2024-04-26T18:12:32.901 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6000 2024-04-26T18:12:32.901 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6001 2024-04-26T18:12:32.901 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6002 2024-04-26T18:12:32.901 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 6100 2024-04-26T18:12:32.901 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6114 2024-04-26T18:12:32.901 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 6115 2024-04-26T18:12:32.901 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 18:12 pg_control 2024-04-26T18:12:32.901 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 26 17:56 pg_filenode.map 2024-04-26T18:12:32.901 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 23144 Apr 26 17:56 pg_internal.init 2024-04-26T18:12:32.902 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.902 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/log: 2024-04-26T18:12:32.902 INFO:teuthology.orchestra.run.smithi031.stdout:total 2 2024-04-26T18:12:32.902 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 . 2024-04-26T18:12:32.902 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.902 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1466 Apr 26 18:12 postgresql-Fri.log 2024-04-26T18:12:32.902 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.902 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-26T18:12:32.902 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.902 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T18:12:32.902 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.902 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.902 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout:total 1 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 18:12 . 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 mappings 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 26 18:12 replorigin_checkpoint 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 snapshots 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T18:12:32.903 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 18:12 .. 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 18:12 .. 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 17:56 . 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 members 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 offsets 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout:total 8 2024-04-26T18:12:32.904 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 . 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 17:56 .. 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 17:56 0000 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout:total 8 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 17:56 . 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 17:56 .. 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 26 18:01 0000 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.905 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.906 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout:total 15 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 26 18:12 . 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1952 Apr 26 18:12 db_0.stat 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 10979 Apr 26 18:12 db_13494.stat 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1151 Apr 26 18:12 global.stat 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 18:12 . 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout:total 208 2024-04-26T18:12:32.907 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 26 18:12 . 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 212992 Apr 26 18:12 0019 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout:total 1048576 2024-04-26T18:12:32.908 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 26 18:12 . 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:12 00000001000000000000009C 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:06 00000001000000000000009D 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:05 00000001000000000000009E 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:05 00000001000000000000009F 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:06 0000000100000000000000A0 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:07 0000000100000000000000A1 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:05 0000000100000000000000A2 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:06 0000000100000000000000A3 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:05 0000000100000000000000A4 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:05 0000000100000000000000A5 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:06 0000000100000000000000A6 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:07 0000000100000000000000A7 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:07 0000000100000000000000A8 2024-04-26T18:12:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:05 0000000100000000000000A9 2024-04-26T18:12:32.910 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:06 0000000100000000000000AA 2024-04-26T18:12:32.910 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:05 0000000100000000000000AB 2024-04-26T18:12:32.910 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:06 0000000100000000000000AC 2024-04-26T18:12:32.910 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:06 0000000100000000000000AD 2024-04-26T18:12:32.910 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:05 0000000100000000000000AE 2024-04-26T18:12:32.910 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:06 0000000100000000000000AF 2024-04-26T18:12:32.910 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:05 0000000100000000000000B0 2024-04-26T18:12:32.910 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:05 0000000100000000000000B1 2024-04-26T18:12:32.910 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:06 0000000100000000000000B2 2024-04-26T18:12:32.910 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:05 0000000100000000000000B3 2024-04-26T18:12:32.910 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:05 0000000100000000000000B4 2024-04-26T18:12:32.910 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:05 0000000100000000000000B5 2024-04-26T18:12:32.910 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:09 0000000100000000000000B6 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:07 0000000100000000000000B7 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000B8 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:08 0000000100000000000000B9 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:05 0000000100000000000000BA 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000BB 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000BC 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000BD 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000BE 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000BF 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:11 0000000100000000000000C0 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000C1 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000C2 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:11 0000000100000000000000C3 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000C4 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000C5 2024-04-26T18:12:32.911 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000C6 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000C7 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:11 0000000100000000000000C8 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000C9 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:12 0000000100000000000000CA 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:11 0000000100000000000000CB 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000CC 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000CD 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000CE 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000CF 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000D0 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000D1 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:11 0000000100000000000000D2 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000D3 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:11 0000000100000000000000D4 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000D5 2024-04-26T18:12:32.912 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:12 0000000100000000000000D6 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000D7 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000D8 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000D9 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:11 0000000100000000000000DA 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 26 18:10 0000000100000000000000DB 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 archive_status 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout:total 0 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 26 17:56 . 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 26 18:12 .. 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout:total 416 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 26 18:10 . 2024-04-26T18:12:32.913 INFO:teuthology.orchestra.run.smithi031.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 26 18:12 .. 2024-04-26T18:12:32.914 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 262144 Apr 26 18:10 0000 2024-04-26T18:12:32.914 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 163840 Apr 26 18:12 0001 2024-04-26T18:12:32.914 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-26T18:12:32.924 INFO:tasks.quiescer:joining Quiescers 2024-04-26T18:12:32.924 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-04-26T18:12:32.924 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T18:12:33.354 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:12:33.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:33 smithi031 ceph-mon[29164]: pgmap v504: 97 pgs: 97 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 21 MiB/s wr, 1.69k op/s 2024-04-26T18:12:33.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:33 smithi186 ceph-mon[32797]: pgmap v504: 97 pgs: 97 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 21 MiB/s wr, 1.69k op/s 2024-04-26T18:12:33.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:33 smithi097 ceph-mon[32419]: pgmap v504: 97 pgs: 97 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 21 MiB/s wr, 1.69k op/s 2024-04-26T18:12:35.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:35 smithi186 ceph-mon[32797]: pgmap v505: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 27 MiB/s wr, 2.02k op/s 2024-04-26T18:12:35.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:35 smithi031 ceph-mon[29164]: pgmap v505: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 27 MiB/s wr, 2.02k op/s 2024-04-26T18:12:35.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:35 smithi097 ceph-mon[32419]: pgmap v505: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 27 MiB/s wr, 2.02k op/s 2024-04-26T18:12:36.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.869+0000 7fc9fd50b700 1 -- 172.21.15.31:0/1948307446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f806c490 msgr2=0x7fc9f8110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:36.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.869+0000 7fc9fd50b700 1 --2- 172.21.15.31:0/1948307446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f806c490 0x7fc9f8110300 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000b0a0 tx=0x7fc9f000b3b0 comp rx=0 tx=0).stop 2024-04-26T18:12:36.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.869+0000 7fc9fd50b700 1 -- 172.21.15.31:0/1948307446 shutdown_connections 2024-04-26T18:12:36.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.869+0000 7fc9fd50b700 1 --2- 172.21.15.31:0/1948307446 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f81094c0 0x7fc9f81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.869+0000 7fc9fd50b700 1 --2- 172.21.15.31:0/1948307446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f806c490 0x7fc9f8110300 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.869+0000 7fc9fd50b700 1 --2- 172.21.15.31:0/1948307446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9f806baf0 0x7fc9f806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.869+0000 7fc9fd50b700 1 -- 172.21.15.31:0/1948307446 >> 172.21.15.31:0/1948307446 conn(0x7fc9f80792d0 msgr2=0x7fc9f80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.869+0000 7fc9fd50b700 1 -- 172.21.15.31:0/1948307446 shutdown_connections 2024-04-26T18:12:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.869+0000 7fc9fd50b700 1 -- 172.21.15.31:0/1948307446 wait complete. 2024-04-26T18:12:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.869+0000 7fc9fd50b700 1 Processor -- start 2024-04-26T18:12:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9fd50b700 1 -- start start 2024-04-26T18:12:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9fd50b700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f806baf0 0x7fc9f81027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9fd50b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9f806c490 0x7fc9f8102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9fd50b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81094c0 0x7fc9f8103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9fd50b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f81135e0 con 0x7fc9f81094c0 2024-04-26T18:12:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9fd50b700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f8113460 con 0x7fc9f806baf0 2024-04-26T18:12:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9fd50b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f8113760 con 0x7fc9f806c490 2024-04-26T18:12:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9f77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81094c0 0x7fc9f8103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9f77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81094c0 0x7fc9f8103250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36300/0 (socket says 172.21.15.31:36300) 2024-04-26T18:12:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9f77fe700 1 -- 172.21.15.31:0/387781966 learned_addr learned my addr 172.21.15.31:0/387781966 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:12:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9f6ffd700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f806baf0 0x7fc9f81027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9f67fc700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9f806c490 0x7fc9f8102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9f77fe700 1 -- 172.21.15.31:0/387781966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9f806c490 msgr2=0x7fc9f8102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9f77fe700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9f806c490 0x7fc9f8102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9f77fe700 1 -- 172.21.15.31:0/387781966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f806baf0 msgr2=0x7fc9f81027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9f77fe700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f806baf0 0x7fc9f81027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9f77fe700 1 -- 172.21.15.31:0/387781966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f81a8060 con 0x7fc9f81094c0 2024-04-26T18:12:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.870+0000 7fc9f67fc700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9f806c490 0x7fc9f8102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:12:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.871+0000 7fc9f77fe700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81094c0 0x7fc9f8103250 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000b780 tx=0x7fc9f0004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.871+0000 7fc9dffff700 1 -- 172.21.15.31:0/387781966 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0021070 con 0x7fc9f81094c0 2024-04-26T18:12:36.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.871+0000 7fc9dffff700 1 -- 172.21.15.31:0/387781966 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9f0007e30 con 0x7fc9f81094c0 2024-04-26T18:12:36.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.871+0000 7fc9dffff700 1 -- 172.21.15.31:0/387781966 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f00047b0 con 0x7fc9f81094c0 2024-04-26T18:12:36.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.871+0000 7fc9fd50b700 1 -- 172.21.15.31:0/387781966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f81a82f0 con 0x7fc9f81094c0 2024-04-26T18:12:36.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.871+0000 7fc9fd50b700 1 -- 172.21.15.31:0/387781966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f81a87a0 con 0x7fc9f81094c0 2024-04-26T18:12:36.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.872+0000 7fc9fd50b700 1 -- 172.21.15.31:0/387781966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9f8104b70 con 0x7fc9f81094c0 2024-04-26T18:12:36.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.873+0000 7fc9dffff700 1 -- 172.21.15.31:0/387781966 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc9f0007980 con 0x7fc9f81094c0 2024-04-26T18:12:36.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.874+0000 7fc9dffff700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc9e00a3db0 0x7fc9e00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:36.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.874+0000 7fc9dffff700 1 -- 172.21.15.31:0/387781966 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8656+0+0 (secure 0 0 0) 0x7fc9f000f020 con 0x7fc9f81094c0 2024-04-26T18:12:36.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.874+0000 7fc9f6ffd700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc9e00a3db0 0x7fc9e00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:36.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.874+0000 7fc9f6ffd700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc9e00a3db0 0x7fc9e00a6270 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009be0 tx=0x7fc9e8009260 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:36.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:36.879+0000 7fc9dffff700 1 -- 172.21.15.31:0/387781966 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc9f002a020 con 0x7fc9f81094c0 2024-04-26T18:12:37.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:37.134+0000 7fc9fd50b700 1 -- 172.21.15.31:0/387781966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9f802e490 con 0x7fc9f81094c0 2024-04-26T18:12:37.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:37.135+0000 7fc9dffff700 1 -- 172.21.15.31:0/387781966 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+11877 (secure 0 0 0) 0x7fc9f009e300 con 0x7fc9f81094c0 2024-04-26T18:12:37.135 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:37.136 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T18:12:22.205584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":25799,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25747":{"gid":25747,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1162255559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1162255559},{"type":"v1","addr":"172.21.15.186:6837","nonce":1162255559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25799":{"gid":25799,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":14,"addr":"172.21.15.97:6839/2506986520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2506986520},{"type":"v1","addr":"172.21.15.97:6839","nonce":2506986520}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":25799,"qdb_cluster":[24457,25799,24446]},"id":1}]} 2024-04-26T18:12:37.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:37.137+0000 7fc9fd50b700 1 -- 172.21.15.31:0/387781966 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc9e00a3db0 msgr2=0x7fc9e00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:37.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:37.137+0000 7fc9fd50b700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc9e00a3db0 0x7fc9e00a6270 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009be0 tx=0x7fc9e8009260 comp rx=0 tx=0).stop 2024-04-26T18:12:37.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:37.137+0000 7fc9fd50b700 1 -- 172.21.15.31:0/387781966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81094c0 msgr2=0x7fc9f8103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:37.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:37.137+0000 7fc9fd50b700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81094c0 0x7fc9f8103250 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000b780 tx=0x7fc9f0004d00 comp rx=0 tx=0).stop 2024-04-26T18:12:37.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:37.137+0000 7fc9fd50b700 1 -- 172.21.15.31:0/387781966 shutdown_connections 2024-04-26T18:12:37.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:37.137+0000 7fc9fd50b700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc9e00a3db0 0x7fc9e00a6270 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:37.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:37.137+0000 7fc9fd50b700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f806baf0 0x7fc9f81027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:37.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:37.137+0000 7fc9fd50b700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81094c0 0x7fc9f8103250 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:37.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:37.137+0000 7fc9fd50b700 1 --2- 172.21.15.31:0/387781966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9f806c490 0x7fc9f8102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:37.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:37.137+0000 7fc9fd50b700 1 -- 172.21.15.31:0/387781966 >> 172.21.15.31:0/387781966 conn(0x7fc9f80792d0 msgr2=0x7fc9f8100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:37.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:37.137+0000 7fc9fd50b700 1 -- 172.21.15.31:0/387781966 shutdown_connections 2024-04-26T18:12:37.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:37.137+0000 7fc9fd50b700 1 -- 172.21.15.31:0/387781966 wait complete. 2024-04-26T18:12:37.139 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 55 2024-04-26T18:12:37.347 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:37 smithi031 ceph-mon[29164]: pgmap v506: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 27 MiB/s wr, 1.95k op/s 2024-04-26T18:12:37.347 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:37 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/387781966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:12:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:37 smithi186 ceph-mon[32797]: pgmap v506: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 27 MiB/s wr, 1.95k op/s 2024-04-26T18:12:37.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:37 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/387781966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:12:37.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:37 smithi097 ceph-mon[32419]: pgmap v506: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 27 MiB/s wr, 1.95k op/s 2024-04-26T18:12:37.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:37 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/387781966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:12:37.901 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.25799 quiesce db --cancel --all' 2024-04-26T18:12:37.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.25799 quiesce db --cancel --all 2024-04-26T18:12:38.166 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:12:39.495 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:39 smithi031 ceph-mon[29164]: pgmap v507: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 28 MiB/s wr, 1.99k op/s 2024-04-26T18:12:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:39 smithi186 ceph-mon[32797]: pgmap v507: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 28 MiB/s wr, 1.99k op/s 2024-04-26T18:12:39.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:39 smithi097 ceph-mon[32419]: pgmap v507: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 28 MiB/s wr, 1.99k op/s 2024-04-26T18:12:39.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:39.980+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:39.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:39.981+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T18:12:39.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:39.981+0000 7f75e0e8c640 1 -- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 msgr2=0x556108f6dd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T18:12:39.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:39.981+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T18:12:39.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-26T18:12:39.981+0000 7f75e0e8c640 1 --2- 172.21.15.31:0/164021214 >> [v2:172.21.15.97:6838/3765855684,v1:172.21.15.97:6839/3765855684] conn(0x556108f6b8d0 0x556108f6dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-26T18:12:40.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.611+0000 7fa8ce5a0700 1 -- 172.21.15.31:0/3240099870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8c8104580 msgr2=0x7fa8c8104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:40.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.611+0000 7fa8ce5a0700 1 --2- 172.21.15.31:0/3240099870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8c8104580 0x7fa8c8104960 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4009990 tx=0x7fa8c4009ca0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.611+0000 7fa8ce5a0700 1 -- 172.21.15.31:0/3240099870 shutdown_connections 2024-04-26T18:12:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.611+0000 7fa8ce5a0700 1 --2- 172.21.15.31:0/3240099870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8c810d9b0 0x7fa8c810fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.611+0000 7fa8ce5a0700 1 --2- 172.21.15.31:0/3240099870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8c8104580 0x7fa8c8104960 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.611+0000 7fa8ce5a0700 1 --2- 172.21.15.31:0/3240099870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8c8104ea0 0x7fa8c810d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.611+0000 7fa8ce5a0700 1 -- 172.21.15.31:0/3240099870 >> 172.21.15.31:0/3240099870 conn(0x7fa8c80fddd0 msgr2=0x7fa8c81001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.611+0000 7fa8ce5a0700 1 -- 172.21.15.31:0/3240099870 shutdown_connections 2024-04-26T18:12:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.611+0000 7fa8ce5a0700 1 -- 172.21.15.31:0/3240099870 wait complete. 2024-04-26T18:12:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.612+0000 7fa8ce5a0700 1 Processor -- start 2024-04-26T18:12:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.612+0000 7fa8ce5a0700 1 -- start start 2024-04-26T18:12:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.612+0000 7fa8ce5a0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8c8104580 0x7fa8c819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.612+0000 7fa8ce5a0700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8c8104ea0 0x7fa8c819fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.612+0000 7fa8ce5a0700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8c810d9b0 0x7fa8c8199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.612+0000 7fa8ce5a0700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c8113620 con 0x7fa8c8104ea0 2024-04-26T18:12:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.612+0000 7fa8ce5a0700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81134a0 con 0x7fa8c810d9b0 2024-04-26T18:12:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.612+0000 7fa8ce5a0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81137a0 con 0x7fa8c8104580 2024-04-26T18:12:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.613+0000 7fa8ccd9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8c8104ea0 0x7fa8c819fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.613+0000 7fa8ccd9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8c8104ea0 0x7fa8c819fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36320/0 (socket says 172.21.15.31:36320) 2024-04-26T18:12:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.613+0000 7fa8ccd9d700 1 -- 172.21.15.31:0/2465679336 learned_addr learned my addr 172.21.15.31:0/2465679336 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:12:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.613+0000 7fa8cd59e700 1 --2- 172.21.15.31:0/2465679336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8c8104580 0x7fa8c819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.613+0000 7fa8cdd9f700 1 --2- 172.21.15.31:0/2465679336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8c810d9b0 0x7fa8c8199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.613+0000 7fa8ccd9d700 1 -- 172.21.15.31:0/2465679336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8c8104580 msgr2=0x7fa8c819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.613+0000 7fa8ccd9d700 1 --2- 172.21.15.31:0/2465679336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8c8104580 0x7fa8c819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.613+0000 7fa8ccd9d700 1 -- 172.21.15.31:0/2465679336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8c810d9b0 msgr2=0x7fa8c8199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.613+0000 7fa8ccd9d700 1 --2- 172.21.15.31:0/2465679336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8c810d9b0 0x7fa8c8199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.613+0000 7fa8ccd9d700 1 -- 172.21.15.31:0/2465679336 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c819a0a0 con 0x7fa8c8104ea0 2024-04-26T18:12:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.613+0000 7fa8ccd9d700 1 --2- 172.21.15.31:0/2465679336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8c8104ea0 0x7fa8c819fcc0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800d720 tx=0x7fa8b800da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.613+0000 7fa8be7fc700 1 -- 172.21.15.31:0/2465679336 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8016070 con 0x7fa8c8104ea0 2024-04-26T18:12:40.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.613+0000 7fa8be7fc700 1 -- 172.21.15.31:0/2465679336 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8b8005020 con 0x7fa8c8104ea0 2024-04-26T18:12:40.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.614+0000 7fa8ce5a0700 1 -- 172.21.15.31:0/2465679336 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c819a390 con 0x7fa8c8104ea0 2024-04-26T18:12:40.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.614+0000 7fa8be7fc700 1 -- 172.21.15.31:0/2465679336 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b800fa90 con 0x7fa8c8104ea0 2024-04-26T18:12:40.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.614+0000 7fa8ce5a0700 1 -- 172.21.15.31:0/2465679336 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c81a6650 con 0x7fa8c8104ea0 2024-04-26T18:12:40.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.616+0000 7fa8be7fc700 1 -- 172.21.15.31:0/2465679336 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa8b80054f0 con 0x7fa8c8104ea0 2024-04-26T18:12:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.617+0000 7fa8be7fc700 1 --2- 172.21.15.31:0/2465679336 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa8b40a3db0 0x7fa8b40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.617+0000 7fa8cd59e700 1 --2- 172.21.15.31:0/2465679336 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa8b40a3db0 0x7fa8b40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.617+0000 7fa8be7fc700 1 -- 172.21.15.31:0/2465679336 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8656+0+0 (secure 0 0 0) 0x7fa8b80d0210 con 0x7fa8c8104ea0 2024-04-26T18:12:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.618+0000 7fa8cd59e700 1 --2- 172.21.15.31:0/2465679336 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa8b40a3db0 0x7fa8b40a6270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa8c819aa80 tx=0x7fa8c4005fd0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.618+0000 7fa8da094b80 1 Processor -- start 2024-04-26T18:12:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.618+0000 7fa8da094b80 1 -- start start 2024-04-26T18:12:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.618+0000 7fa8da094b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115aaa53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.618+0000 7fa8da094b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5920 0x56115aa9f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.618+0000 7fa8da094b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 0x56115aa9fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.618+0000 7fa8da094b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56115a9f18a0 con 0x56115aaa5000 2024-04-26T18:12:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.618+0000 7fa8da094b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56115a9579a0 con 0x56115aaa5920 2024-04-26T18:12:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.618+0000 7fa8da094b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56115a9ed860 con 0x56115aa9fb90 2024-04-26T18:12:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.618+0000 7fa8cd59e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115aaa53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.618+0000 7fa8cd59e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115aaa53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36328/0 (socket says 172.21.15.31:36328) 2024-04-26T18:12:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.618+0000 7fa8cd59e700 1 -- 172.21.15.31:0/119353835 learned_addr learned my addr 172.21.15.31:0/119353835 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:12:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.619+0000 7fa8ccd9d700 1 --2- 172.21.15.31:0/119353835 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5920 0x56115aa9f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.619+0000 7fa8cdd9f700 1 --2- 172.21.15.31:0/119353835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 0x56115aa9fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.619+0000 7fa8cd59e700 1 -- 172.21.15.31:0/119353835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 msgr2=0x56115aa9fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:12:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.619+0000 7fa8cd59e700 1 --2- 172.21.15.31:0/119353835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 0x56115aa9fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.619+0000 7fa8cd59e700 1 -- 172.21.15.31:0/119353835 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5920 msgr2=0x56115aa9f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:12:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.619+0000 7fa8cd59e700 1 --2- 172.21.15.31:0/119353835 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5920 0x56115aa9f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.619+0000 7fa8cd59e700 1 -- 172.21.15.31:0/119353835 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56115a93e940 con 0x56115aaa5000 2024-04-26T18:12:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.619+0000 7fa8cd59e700 1 --2- 172.21.15.31:0/119353835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115aaa53e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4015000 tx=0x7fa8c4020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d22158604de0556d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.619+0000 7fa8b37fe700 1 -- 172.21.15.31:0/119353835 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c400f520 con 0x56115aaa5000 2024-04-26T18:12:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.619+0000 7fa8b37fe700 1 -- 172.21.15.31:0/119353835 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8c400fb70 con 0x56115aaa5000 2024-04-26T18:12:40.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.619+0000 7fa8b37fe700 1 -- 172.21.15.31:0/119353835 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c4024be0 con 0x56115aaa5000 2024-04-26T18:12:40.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.620+0000 7fa8da094b80 1 -- 172.21.15.31:0/119353835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 msgr2=0x56115aaa53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:40.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.620+0000 7fa8da094b80 1 --2- 172.21.15.31:0/119353835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115aaa53e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4015000 tx=0x7fa8c4020000 comp rx=0 tx=0).stop 2024-04-26T18:12:40.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.620+0000 7fa8da094b80 1 -- 172.21.15.31:0/119353835 shutdown_connections 2024-04-26T18:12:40.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.620+0000 7fa8da094b80 1 --2- 172.21.15.31:0/119353835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 0x56115aa9fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.620+0000 7fa8da094b80 1 --2- 172.21.15.31:0/119353835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115aaa53e0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.620+0000 7fa8da094b80 1 --2- 172.21.15.31:0/119353835 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5920 0x56115aa9f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.620+0000 7fa8da094b80 1 -- 172.21.15.31:0/119353835 >> 172.21.15.31:0/119353835 conn(0x56115a9cd630 msgr2=0x56115aa99fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:40.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.620+0000 7fa8da094b80 1 -- 172.21.15.31:0/119353835 shutdown_connections 2024-04-26T18:12:40.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.620+0000 7fa8da094b80 1 -- 172.21.15.31:0/119353835 wait complete. 2024-04-26T18:12:40.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.620+0000 7fa8da094b80 1 -- start start 2024-04-26T18:12:40.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8da094b80 1 Processor -- start 2024-04-26T18:12:40.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8da094b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aa9fb90 0x56115aaab510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8da094b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115aaa88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8da094b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aaa5920 0x56115aaa8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8da094b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56115aab06e0 con 0x56115aaa5000 2024-04-26T18:12:40.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8da094b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x56115a9abaf0 con 0x56115aa9fb90 2024-04-26T18:12:40.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8da094b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56115aab0860 con 0x56115aaa5920 2024-04-26T18:12:40.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8da094b80 10 client.0 _ll_register_callbacks cb 0x56115a9899c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:12:40.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8ccd9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115aaa88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8ccd9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115aaa88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36338/0 (socket says 172.21.15.31:36338) 2024-04-26T18:12:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8ccd9d700 1 -- 172.21.15.31:0/798677781 learned_addr learned my addr 172.21.15.31:0/798677781 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:12:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8cdd9f700 1 --2- 172.21.15.31:0/798677781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aaa5920 0x56115aaa8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8cd59e700 1 --2- 172.21.15.31:0/798677781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aa9fb90 0x56115aaab510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8ccd9d700 1 -- 172.21.15.31:0/798677781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aaa5920 msgr2=0x56115aaa8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:12:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8ccd9d700 1 --2- 172.21.15.31:0/798677781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aaa5920 0x56115aaa8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8ccd9d700 1 -- 172.21.15.31:0/798677781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aa9fb90 msgr2=0x56115aaab510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:12:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.621+0000 7fa8ccd9d700 1 --2- 172.21.15.31:0/798677781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aa9fb90 0x56115aaab510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.622+0000 7fa8ccd9d700 1 -- 172.21.15.31:0/798677781 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56115a9af4f0 con 0x56115aaa5000 2024-04-26T18:12:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.622+0000 7fa8ccd9d700 1 --2- 172.21.15.31:0/798677781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115aaa88b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8005d90 tx=0x7fa8b809c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a804055c9fea9bc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.622+0000 7fa8b0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:12:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.622+0000 7fa8b0ff9700 1 -- 172.21.15.31:0/798677781 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b801f040 con 0x56115aaa5000 2024-04-26T18:12:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.622+0000 7fa8b0ff9700 1 -- 172.21.15.31:0/798677781 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8656+0+0 (secure 0 0 0) 0x7fa8b80a01b0 con 0x56115aaa5000 2024-04-26T18:12:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.622+0000 7fa8da094b80 1 -- 172.21.15.31:0/798677781 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8c400fb70 con 0x56115aaa5000 2024-04-26T18:12:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.622+0000 7fa8b0ff9700 1 -- 172.21.15.31:0/798677781 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fa8b80a04e0 con 0x56115aaa5000 2024-04-26T18:12:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.622+0000 7fa8da094b80 10 client.16161 fetch_fsmap learned FSMap version 55 2024-04-26T18:12:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.622+0000 7fa8da094b80 1 -- 172.21.15.31:0/798677781 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x56115aabae50 con 0x56115aaa5000 2024-04-26T18:12:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.622+0000 7fa8b0ff9700 1 -- 172.21.15.31:0/798677781 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8c400fb70 con 0x56115aaa5000 2024-04-26T18:12:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.623+0000 7fa8b0ff9700 1 -- 172.21.15.31:0/798677781 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 55) ==== 8350+0+0 (secure 0 0 0) 0x7fa8b80bc040 con 0x56115aaa5000 2024-04-26T18:12:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.623+0000 7fa8da094b80 10 client.16161 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T18:12:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.623+0000 7fa8da094b80 10 client.16161 resolve_mds: validated gid 25799 aka daemon mds.b 2024-04-26T18:12:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.623+0000 7fa8da094b80 20 client.16161 populate_metadata read hostname 'smithi031' 2024-04-26T18:12:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.623+0000 7fa8da094b80 1 --2- 172.21.15.31:0/798677781 >> [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] conn(0x56115aabd3e0 0x56115aabf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.623+0000 7fa8da094b80 4 client.16161 mds_command: new command op to 25799 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T18:12:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.623+0000 7fa8da094b80 1 -- 172.21.15.31:0/798677781 --> [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56115a9ff280 con 0x56115aabd3e0 2024-04-26T18:12:40.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.623+0000 7fa8cd59e700 1 --2- 172.21.15.31:0/798677781 >> [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] conn(0x56115aabd3e0 0x56115aabf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.623+0000 7fa8b0ff9700 10 client.16161 ms_handle_connect on v2:172.21.15.97:6838/2506986520 2024-04-26T18:12:40.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.623+0000 7fa8cd59e700 1 --2- 172.21.15.31:0/798677781 >> [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] conn(0x56115aabd3e0 0x56115aabf7d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3a32da2cd89141b server_cookie=dd01446f08660845 in_seq=0 out_seq=0 2024-04-26T18:12:40.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.624+0000 7fa8b0ff9700 1 -- 172.21.15.31:0/798677781 <== mds.0 v2:172.21.15.97:6838/2506986520 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56115a9ff280 con 0x56115aabd3e0 2024-04-26T18:12:40.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.624+0000 7fa8b0ff9700 10 client.16161 handle_command_reply: tid=0 2024-04-26T18:12:40.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.624+0000 7fa8da094b80 1 client.16161 shutdown 2024-04-26T18:12:40.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.624+0000 7fa8da094b80 1 -- 172.21.15.31:0/798677781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 msgr2=0x56115aaa88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:40.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.624+0000 7fa8da094b80 1 --2- 172.21.15.31:0/798677781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115aaa88b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8005d90 tx=0x7fa8b809c8a0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.624+0000 7fa8da094b80 1 -- 172.21.15.31:0/798677781 shutdown_connections 2024-04-26T18:12:40.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.624+0000 7fa8da094b80 1 --2- 172.21.15.31:0/798677781 >> [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] conn(0x56115aabd3e0 0x56115aabf7d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.624+0000 7fa8da094b80 1 --2- 172.21.15.31:0/798677781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aa9fb90 0x56115aaab510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.624+0000 7fa8da094b80 1 --2- 172.21.15.31:0/798677781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aaa5920 0x56115aaa8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.624+0000 7fa8da094b80 1 --2- 172.21.15.31:0/798677781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115aaa88b0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.624+0000 7fa8b0ff9700 0 client.16161 ms_handle_reset on v2:172.21.15.97:6838/2506986520 2024-04-26T18:12:40.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.624+0000 7fa8da094b80 1 -- 172.21.15.31:0/798677781 >> 172.21.15.31:0/798677781 conn(0x56115a9cd630 msgr2=0x56115aa99610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:40.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.625+0000 7fa8da094b80 1 -- 172.21.15.31:0/798677781 shutdown_connections 2024-04-26T18:12:40.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.625+0000 7fa8da094b80 1 -- 172.21.15.31:0/798677781 wait complete. 2024-04-26T18:12:40.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.625+0000 7fa8da094b80 20 client.16161 trim_cache size 0 max 16384 2024-04-26T18:12:40.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.644+0000 7fa8da094b80 1 Processor -- start 2024-04-26T18:12:40.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.644+0000 7fa8da094b80 1 -- start start 2024-04-26T18:12:40.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8da094b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 0x56115aa9f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8da094b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5000 0x56115aaa6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8da094b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5920 0x56115aab95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8da094b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81ae380 con 0x56115aaa5920 2024-04-26T18:12:40.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8da094b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81abc10 con 0x56115aaa5000 2024-04-26T18:12:40.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8da094b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81a9a60 con 0x56115aa9fb90 2024-04-26T18:12:40.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8cdd9f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5920 0x56115aab95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8cdd9f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5920 0x56115aab95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36348/0 (socket says 172.21.15.31:36348) 2024-04-26T18:12:40.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8cdd9f700 1 -- 172.21.15.31:0/1140976410 learned_addr learned my addr 172.21.15.31:0/1140976410 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:12:40.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8cd59e700 1 --2- 172.21.15.31:0/1140976410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 0x56115aa9f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8ccd9d700 1 --2- 172.21.15.31:0/1140976410 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5000 0x56115aaa6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8cdd9f700 1 -- 172.21.15.31:0/1140976410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 msgr2=0x56115aa9f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:12:40.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8cdd9f700 1 --2- 172.21.15.31:0/1140976410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 0x56115aa9f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8cdd9f700 1 -- 172.21.15.31:0/1140976410 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5000 msgr2=0x56115aaa6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:12:40.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8cdd9f700 1 --2- 172.21.15.31:0/1140976410 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5000 0x56115aaa6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8cdd9f700 1 -- 172.21.15.31:0/1140976410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56115aa9aa70 con 0x56115aaa5920 2024-04-26T18:12:40.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8ccd9d700 1 --2- 172.21.15.31:0/1140976410 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5000 0x56115aaa6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:12:40.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8cdd9f700 1 --2- 172.21.15.31:0/1140976410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5920 0x56115aab95e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009fb0 tx=0x7fa8c0009100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4045fa69635f65e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:40.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.645+0000 7fa8b2ffd700 1 -- 172.21.15.31:0/1140976410 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c001b030 con 0x56115aaa5920 2024-04-26T18:12:40.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.646+0000 7fa8b2ffd700 1 -- 172.21.15.31:0/1140976410 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8c00178f0 con 0x56115aaa5920 2024-04-26T18:12:40.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.646+0000 7fa8b2ffd700 1 -- 172.21.15.31:0/1140976410 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c001f8f0 con 0x56115aaa5920 2024-04-26T18:12:40.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.646+0000 7fa8da094b80 1 -- 172.21.15.31:0/1140976410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5920 msgr2=0x56115aab95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:40.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.646+0000 7fa8da094b80 1 --2- 172.21.15.31:0/1140976410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5920 0x56115aab95e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009fb0 tx=0x7fa8c0009100 comp rx=0 tx=0).stop 2024-04-26T18:12:40.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.646+0000 7fa8da094b80 1 -- 172.21.15.31:0/1140976410 shutdown_connections 2024-04-26T18:12:40.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.646+0000 7fa8da094b80 1 --2- 172.21.15.31:0/1140976410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5920 0x56115aab95e0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.646+0000 7fa8da094b80 1 --2- 172.21.15.31:0/1140976410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 0x56115aa9f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.646+0000 7fa8da094b80 1 --2- 172.21.15.31:0/1140976410 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5000 0x56115aaa6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.646+0000 7fa8da094b80 1 -- 172.21.15.31:0/1140976410 >> 172.21.15.31:0/1140976410 conn(0x56115a9cd630 msgr2=0x56115aaa9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:40.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.646+0000 7fa8da094b80 1 -- 172.21.15.31:0/1140976410 shutdown_connections 2024-04-26T18:12:40.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.646+0000 7fa8da094b80 1 -- 172.21.15.31:0/1140976410 wait complete. 2024-04-26T18:12:40.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.647+0000 7fa8da094b80 1 -- start start 2024-04-26T18:12:40.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.647+0000 7fa8da094b80 1 Processor -- start 2024-04-26T18:12:40.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.647+0000 7fa8da094b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 0x56115ab1cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.647+0000 7fa8da094b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115ab1d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.647+0000 7fa8da094b80 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5920 0x56115aab2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.647+0000 7fa8da094b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81aa0d0 con 0x56115aaa5000 2024-04-26T18:12:40.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.647+0000 7fa8da094b80 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81ab540 con 0x56115aaa5920 2024-04-26T18:12:40.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.647+0000 7fa8da094b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81aa7a0 con 0x56115aa9fb90 2024-04-26T18:12:40.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.647+0000 7fa8da094b80 10 client.0 _ll_register_callbacks cb 0x56115a9899c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T18:12:40.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.647+0000 7fa8ccd9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115ab1d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.647+0000 7fa8ccd9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115ab1d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36352/0 (socket says 172.21.15.31:36352) 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.647+0000 7fa8ccd9d700 1 -- 172.21.15.31:0/1348976999 learned_addr learned my addr 172.21.15.31:0/1348976999 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.647+0000 7fa8cdd9f700 1 --2- 172.21.15.31:0/1348976999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5920 0x56115aab2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.647+0000 7fa8cd59e700 1 --2- 172.21.15.31:0/1348976999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 0x56115ab1cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8ccd9d700 1 -- 172.21.15.31:0/1348976999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 msgr2=0x56115ab1cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8ccd9d700 1 --2- 172.21.15.31:0/1348976999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 0x56115ab1cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8ccd9d700 1 -- 172.21.15.31:0/1348976999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5920 msgr2=0x56115aab2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8ccd9d700 1 --2- 172.21.15.31:0/1348976999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5920 0x56115aab2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8ccd9d700 1 -- 172.21.15.31:0/1348976999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56115aab3700 con 0x56115aaa5000 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8cdd9f700 1 --2- 172.21.15.31:0/1348976999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5920 0x56115aab2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8ccd9d700 1 --2- 172.21.15.31:0/1348976999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115ab1d1b0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8098b00 tx=0x7fa8b800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848f9227c94cc96f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8b0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8b0ff9700 1 -- 172.21.15.31:0/1348976999 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b801f040 con 0x56115aaa5000 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8b0ff9700 1 -- 172.21.15.31:0/1348976999 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8656+0+0 (secure 0 0 0) 0x7fa8b809c390 con 0x56115aaa5000 2024-04-26T18:12:40.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8da094b80 1 -- 172.21.15.31:0/1348976999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8c00178f0 con 0x56115aaa5000 2024-04-26T18:12:40.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8b0ff9700 1 -- 172.21.15.31:0/1348976999 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fa8b80a4da0 con 0x56115aaa5000 2024-04-26T18:12:40.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8da094b80 10 client.16167 fetch_fsmap learned FSMap version 55 2024-04-26T18:12:40.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8da094b80 1 -- 172.21.15.31:0/1348976999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x56115ab25450 con 0x56115aaa5000 2024-04-26T18:12:40.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8b0ff9700 1 -- 172.21.15.31:0/1348976999 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8b809c130 con 0x56115aaa5000 2024-04-26T18:12:40.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8b0ff9700 1 -- 172.21.15.31:0/1348976999 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 55) ==== 8350+0+0 (secure 0 0 0) 0x7fa8b8080070 con 0x56115aaa5000 2024-04-26T18:12:40.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8da094b80 10 client.16167 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T18:12:40.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8da094b80 10 client.16167 resolve_mds: validated gid 25799 aka daemon mds.b 2024-04-26T18:12:40.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8da094b80 20 client.16167 populate_metadata read hostname 'smithi031' 2024-04-26T18:12:40.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.648+0000 7fa8da094b80 1 --2- 172.21.15.31:0/1348976999 >> [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] conn(0x56115ab25b40 0x56115ab27f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:40.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.649+0000 7fa8da094b80 4 client.16167 mds_command: new command op to 25799 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-26T18:12:40.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.649+0000 7fa8da094b80 1 -- 172.21.15.31:0/1348976999 --> [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x7fa8c00178f0 con 0x56115ab25b40 2024-04-26T18:12:40.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.649+0000 7fa8cd59e700 1 --2- 172.21.15.31:0/1348976999 >> [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] conn(0x56115ab25b40 0x56115ab27f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:40.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.649+0000 7fa8cd59e700 1 --2- 172.21.15.31:0/1348976999 >> [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] conn(0x56115ab25b40 0x56115ab27f60 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67179c54e1d41fea server_cookie=c5fb0be631ab7e0d in_seq=0 out_seq=0 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.649+0000 7fa8b0ff9700 10 client.16167 ms_handle_connect on v2:172.21.15.97:6838/2506986520 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.649+0000 7fa8b0ff9700 1 -- 172.21.15.31:0/1348976999 <== mds.0 v2:172.21.15.97:6838/2506986520 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x7fa8c00178f0 con 0x56115ab25b40 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.649+0000 7fa8b0ff9700 10 client.16167 handle_command_reply: tid=0 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8da094b80 1 client.16167 shutdown 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8da094b80 1 -- 172.21.15.31:0/1348976999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 msgr2=0x56115ab1d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8da094b80 1 --2- 172.21.15.31:0/1348976999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115ab1d1b0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8098b00 tx=0x7fa8b800b480 comp rx=0 tx=0).stop 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8da094b80 1 -- 172.21.15.31:0/1348976999 shutdown_connections 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8da094b80 1 --2- 172.21.15.31:0/1348976999 >> [v2:172.21.15.97:6838/2506986520,v1:172.21.15.97:6839/2506986520] conn(0x56115ab25b40 0x56115ab27f60 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8da094b80 1 --2- 172.21.15.31:0/1348976999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56115aaa5920 0x56115aab2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8da094b80 1 --2- 172.21.15.31:0/1348976999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56115aa9fb90 0x56115ab1cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8b0ff9700 0 client.16167 ms_handle_reset on v2:172.21.15.97:6838/2506986520 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8da094b80 1 --2- 172.21.15.31:0/1348976999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56115aaa5000 0x56115ab1d1b0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8da094b80 1 -- 172.21.15.31:0/1348976999 >> 172.21.15.31:0/1348976999 conn(0x56115a9cd630 msgr2=0x56115aa99ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8da094b80 1 -- 172.21.15.31:0/1348976999 shutdown_connections 2024-04-26T18:12:40.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8da094b80 1 -- 172.21.15.31:0/1348976999 wait complete. 2024-04-26T18:12:40.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8da094b80 20 client.16167 trim_cache size 0 max 16384 2024-04-26T18:12:40.655 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T18:12:40.655 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 55, 2024-04-26T18:12:40.655 INFO:teuthology.orchestra.run.smithi031.stdout: "leader": 25799, 2024-04-26T18:12:40.655 INFO:teuthology.orchestra.run.smithi031.stdout: "set_version": 22, 2024-04-26T18:12:40.655 INFO:teuthology.orchestra.run.smithi031.stdout: "sets": {} 2024-04-26T18:12:40.655 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T18:12:40.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8ce5a0700 1 -- 172.21.15.31:0/2465679336 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa8b40a3db0 msgr2=0x7fa8b40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:40.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8ce5a0700 1 --2- 172.21.15.31:0/2465679336 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa8b40a3db0 0x7fa8b40a6270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa8c819aa80 tx=0x7fa8c4005fd0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8ce5a0700 1 -- 172.21.15.31:0/2465679336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8c8104ea0 msgr2=0x7fa8c819fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:40.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.650+0000 7fa8ce5a0700 1 --2- 172.21.15.31:0/2465679336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8c8104ea0 0x7fa8c819fcc0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800d720 tx=0x7fa8b800da30 comp rx=0 tx=0).stop 2024-04-26T18:12:40.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.651+0000 7fa8ce5a0700 1 -- 172.21.15.31:0/2465679336 shutdown_connections 2024-04-26T18:12:40.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.651+0000 7fa8ce5a0700 1 --2- 172.21.15.31:0/2465679336 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fa8b40a3db0 0x7fa8b40a6270 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.651+0000 7fa8ce5a0700 1 --2- 172.21.15.31:0/2465679336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa8c810d9b0 0x7fa8c8199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.651+0000 7fa8ce5a0700 1 --2- 172.21.15.31:0/2465679336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8c8104580 0x7fa8c819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.651+0000 7fa8ce5a0700 1 --2- 172.21.15.31:0/2465679336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8c8104ea0 0x7fa8c819fcc0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:40.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.651+0000 7fa8ce5a0700 1 -- 172.21.15.31:0/2465679336 >> 172.21.15.31:0/2465679336 conn(0x7fa8c80fddd0 msgr2=0x7fa8c80feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:40.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.651+0000 7fa8ce5a0700 1 -- 172.21.15.31:0/2465679336 shutdown_connections 2024-04-26T18:12:40.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:40.651+0000 7fa8ce5a0700 1 -- 172.21.15.31:0/2465679336 wait complete. 2024-04-26T18:12:41.418 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:41 smithi031 ceph-mon[29164]: pgmap v508: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 24 MiB/s wr, 1.58k op/s 2024-04-26T18:12:41.450 ERROR:teuthology.run_tasks:Manager failed: quiescer Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/qa/tasks/quiescer.py", line 419, in task stop_all_quiescers(ctx.ceph[cluster_name].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph-c_bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1/qa/tasks/quiescer.py", line 378, in stop_all_quiescers raise RuntimeError(f"error during quiesce thrashing: {thrasher.exception}") RuntimeError: error during quiesce thrashing: Error parsing quiesce response: Expecting value: line 1 column 1 (char 0) 2024-04-26T18:12:41.452 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-26T18:12:41.465 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi031.front.sepia.ceph.com 2024-04-26T18:12:41.475 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-26T18:12:41.500 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-26T18:12:41.500 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T18:12:41.500 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-26T18:12:41.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-26T18:12:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:41 smithi186 ceph-mon[32797]: pgmap v508: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 24 MiB/s wr, 1.58k op/s 2024-04-26T18:12:41.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-26T18:12:41.619 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-26T18:12:41.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:41 smithi097 ceph-mon[32419]: pgmap v508: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 24 MiB/s wr, 1.58k op/s 2024-04-26T18:12:41.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.767+0000 7f6dc4486640 1 -- 172.21.15.31:0/2355929607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc103be0 msgr2=0x7f6dbc103fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.767+0000 7f6dc4486640 1 --2- 172.21.15.31:0/2355929607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc103be0 0x7f6dbc103fc0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6db8009900 tx=0x7f6db802a480 comp rx=0 tx=0).stop 2024-04-26T18:12:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.767+0000 7f6dc4486640 1 -- 172.21.15.31:0/2355929607 shutdown_connections 2024-04-26T18:12:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.767+0000 7f6dc4486640 1 --2- 172.21.15.31:0/2355929607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6dbc10cfb0 0x7f6dbc10f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.767+0000 7f6dc4486640 1 --2- 172.21.15.31:0/2355929607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dbc104500 0x7f6dbc10c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.767+0000 7f6dc4486640 1 --2- 172.21.15.31:0/2355929607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc103be0 0x7f6dbc103fc0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.767+0000 7f6dc4486640 1 -- 172.21.15.31:0/2355929607 >> 172.21.15.31:0/2355929607 conn(0x7f6dbc077bb0 msgr2=0x7f6dbc077fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.767+0000 7f6dc4486640 1 -- 172.21.15.31:0/2355929607 shutdown_connections 2024-04-26T18:12:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.767+0000 7f6dc4486640 1 -- 172.21.15.31:0/2355929607 wait complete. 2024-04-26T18:12:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc4486640 1 Processor -- start 2024-04-26T18:12:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc4486640 1 -- start start 2024-04-26T18:12:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc4486640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6dbc103be0 0x7f6dbc199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc4486640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dbc104500 0x7f6dbc1995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc4486640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10cfb0 0x7f6dbc19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc4486640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc112bf0 con 0x7f6dbc10cfb0 2024-04-26T18:12:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc4486640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc112a70 con 0x7f6dbc103be0 2024-04-26T18:12:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc4486640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc112d70 con 0x7f6dbc104500 2024-04-26T18:12:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc2a23640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10cfb0 0x7f6dbc19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc2222640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6dbc103be0 0x7f6dbc199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc2222640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6dbc103be0 0x7f6dbc199070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.31:42896/0 (socket says 172.21.15.31:42896) 2024-04-26T18:12:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc2a23640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10cfb0 0x7f6dbc19d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36360/0 (socket says 172.21.15.31:36360) 2024-04-26T18:12:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc2222640 1 -- 172.21.15.31:0/547269811 learned_addr learned my addr 172.21.15.31:0/547269811 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:12:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc1a21640 1 --2- 172.21.15.31:0/547269811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dbc104500 0x7f6dbc1995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc2a23640 1 -- 172.21.15.31:0/547269811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dbc104500 msgr2=0x7f6dbc1995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc2a23640 1 --2- 172.21.15.31:0/547269811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dbc104500 0x7f6dbc1995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc2a23640 1 -- 172.21.15.31:0/547269811 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6dbc103be0 msgr2=0x7f6dbc199070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc2a23640 1 --2- 172.21.15.31:0/547269811 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6dbc103be0 0x7f6dbc199070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc2a23640 1 -- 172.21.15.31:0/547269811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dbc19e060 con 0x7f6dbc10cfb0 2024-04-26T18:12:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.768+0000 7f6dc2222640 1 --2- 172.21.15.31:0/547269811 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6dbc103be0 0x7f6dbc199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:12:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.769+0000 7f6dc2a23640 1 --2- 172.21.15.31:0/547269811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10cfb0 0x7f6dbc19d930 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6db4005fd0 tx=0x7f6db400edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.769+0000 7f6db37fe640 1 -- 172.21.15.31:0/547269811 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db4004d20 con 0x7f6dbc10cfb0 2024-04-26T18:12:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.769+0000 7f6db37fe640 1 -- 172.21.15.31:0/547269811 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6db4005020 con 0x7f6dbc10cfb0 2024-04-26T18:12:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.769+0000 7f6dc4486640 1 -- 172.21.15.31:0/547269811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dbc19e350 con 0x7f6dbc10cfb0 2024-04-26T18:12:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.769+0000 7f6db37fe640 1 -- 172.21.15.31:0/547269811 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db40116b0 con 0x7f6dbc10cfb0 2024-04-26T18:12:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.769+0000 7f6dc4486640 1 -- 172.21.15.31:0/547269811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dbc1a5be0 con 0x7f6dbc10cfb0 2024-04-26T18:12:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.770+0000 7f6dc4486640 1 -- 172.21.15.31:0/547269811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dbc0506c0 con 0x7f6dbc10cfb0 2024-04-26T18:12:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.771+0000 7f6db37fe640 1 -- 172.21.15.31:0/547269811 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6db40054d0 con 0x7f6dbc10cfb0 2024-04-26T18:12:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.772+0000 7f6db37fe640 1 --2- 172.21.15.31:0/547269811 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f6da80a3dc0 0x7f6da80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.772+0000 7f6dc2222640 1 --2- 172.21.15.31:0/547269811 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f6da80a3dc0 0x7f6da80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.772+0000 7f6db37fe640 1 -- 172.21.15.31:0/547269811 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8656+0+0 (secure 0 0 0) 0x7f6db40d0ca0 con 0x7f6dbc10cfb0 2024-04-26T18:12:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.772+0000 7f6dc2222640 1 --2- 172.21.15.31:0/547269811 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f6da80a3dc0 0x7f6da80a6280 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6db8031440 tx=0x7f6db80058d0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.775+0000 7f6db37fe640 1 -- 172.21.15.31:0/547269811 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6db4016030 con 0x7f6dbc10cfb0 2024-04-26T18:12:41.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.986+0000 7f6dc4486640 1 -- 172.21.15.31:0/547269811 --> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f6dbc0fedb0 con 0x7f6da80a3dc0 2024-04-26T18:12:41.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.994+0000 7f6db37fe640 1 -- 172.21.15.31:0/547269811 <== mgr.24826 v2:172.21.15.31:6800/3595544994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f6dbc0fedb0 con 0x7f6da80a3dc0 2024-04-26T18:12:41.993 INFO:teuthology.orchestra.run.smithi031.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-26T00:00:00", "created": "2024-04-26T17:56:30", "first": "2024-04-26T17:57:00", "last": "2024-04-26T18:12:00", "last_pruned": "2024-04-26T18:12:00", "created_count": 16, "pruned_count": 9, "active": true} 2024-04-26T18:12:41.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.996+0000 7f6dc4486640 1 -- 172.21.15.31:0/547269811 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f6da80a3dc0 msgr2=0x7f6da80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:41.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.996+0000 7f6dc4486640 1 --2- 172.21.15.31:0/547269811 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f6da80a3dc0 0x7f6da80a6280 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6db8031440 tx=0x7f6db80058d0 comp rx=0 tx=0).stop 2024-04-26T18:12:41.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.996+0000 7f6dc4486640 1 -- 172.21.15.31:0/547269811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10cfb0 msgr2=0x7f6dbc19d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:41.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.996+0000 7f6dc4486640 1 --2- 172.21.15.31:0/547269811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10cfb0 0x7f6dbc19d930 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6db4005fd0 tx=0x7f6db400edd0 comp rx=0 tx=0).stop 2024-04-26T18:12:41.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.996+0000 7f6dc4486640 1 -- 172.21.15.31:0/547269811 shutdown_connections 2024-04-26T18:12:41.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.996+0000 7f6dc4486640 1 --2- 172.21.15.31:0/547269811 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f6da80a3dc0 0x7f6da80a6280 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:41.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.996+0000 7f6dc4486640 1 --2- 172.21.15.31:0/547269811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10cfb0 0x7f6dbc19d930 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:41.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.996+0000 7f6dc4486640 1 --2- 172.21.15.31:0/547269811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dbc104500 0x7f6dbc1995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:41.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.996+0000 7f6dc4486640 1 --2- 172.21.15.31:0/547269811 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6dbc103be0 0x7f6dbc199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:41.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.996+0000 7f6dc4486640 1 -- 172.21.15.31:0/547269811 >> 172.21.15.31:0/547269811 conn(0x7f6dbc077bb0 msgr2=0x7f6dbc0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:41.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.996+0000 7f6dc4486640 1 -- 172.21.15.31:0/547269811 shutdown_connections 2024-04-26T18:12:41.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:41.996+0000 7f6dc4486640 1 -- 172.21.15.31:0/547269811 wait complete. 2024-04-26T18:12:42.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-26T18:12:42.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.114+0000 7f1c95c08640 1 -- 172.21.15.31:0/236589086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c90101860 msgr2=0x7f1c90101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.114+0000 7f1c95c08640 1 --2- 172.21.15.31:0/236589086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c90101860 0x7f1c90101ce0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009930 tx=0x7f1c8402a420 comp rx=0 tx=0).stop 2024-04-26T18:12:42.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.114+0000 7f1c95c08640 1 -- 172.21.15.31:0/236589086 shutdown_connections 2024-04-26T18:12:42.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.114+0000 7f1c95c08640 1 --2- 172.21.15.31:0/236589086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c90102220 0x7f1c9010e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.114+0000 7f1c95c08640 1 --2- 172.21.15.31:0/236589086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c90101860 0x7f1c90101ce0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.114+0000 7f1c95c08640 1 --2- 172.21.15.31:0/236589086 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1c90107820 0x7f1c90107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.114+0000 7f1c95c08640 1 -- 172.21.15.31:0/236589086 >> 172.21.15.31:0/236589086 conn(0x7f1c900789b0 msgr2=0x7f1c90078dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:42.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.114+0000 7f1c95c08640 1 -- 172.21.15.31:0/236589086 shutdown_connections 2024-04-26T18:12:42.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.114+0000 7f1c95c08640 1 -- 172.21.15.31:0/236589086 wait complete. 2024-04-26T18:12:42.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.114+0000 7f1c95c08640 1 Processor -- start 2024-04-26T18:12:42.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c95c08640 1 -- start start 2024-04-26T18:12:42.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c95c08640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c90101860 0x7f1c900fe7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:42.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c95c08640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1c90102220 0x7f1c900fed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:42.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c95c08640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c90107820 0x7f1c900ff270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:42.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c95c08640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1c90111ae0 con 0x7f1c90107820 2024-04-26T18:12:42.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c95c08640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f1c90111960 con 0x7f1c90102220 2024-04-26T18:12:42.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c95c08640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1c90111c60 con 0x7f1c90101860 2024-04-26T18:12:42.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c8ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c90107820 0x7f1c900ff270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:42.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c8ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c90107820 0x7f1c900ff270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36380/0 (socket says 172.21.15.31:36380) 2024-04-26T18:12:42.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c8ffff640 1 -- 172.21.15.31:0/2241216549 learned_addr learned my addr 172.21.15.31:0/2241216549 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:12:42.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c8effd640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1c90102220 0x7f1c900fed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:42.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c8f7fe640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c90101860 0x7f1c900fe7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:42.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c8ffff640 1 -- 172.21.15.31:0/2241216549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c90101860 msgr2=0x7f1c900fe7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c8ffff640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c90101860 0x7f1c900fe7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c8ffff640 1 -- 172.21.15.31:0/2241216549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1c90102220 msgr2=0x7f1c900fed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c8ffff640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1c90102220 0x7f1c900fed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c8ffff640 1 -- 172.21.15.31:0/2241216549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c901a75f0 con 0x7f1c90107820 2024-04-26T18:12:42.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c8f7fe640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c90101860 0x7f1c900fe7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:12:42.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.115+0000 7f1c8ffff640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c90107820 0x7f1c900ff270 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1c8800eaa0 tx=0x7f1c8800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:42.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.116+0000 7f1c8cff9640 1 -- 172.21.15.31:0/2241216549 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8800cdb0 con 0x7f1c90107820 2024-04-26T18:12:42.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.116+0000 7f1c8cff9640 1 -- 172.21.15.31:0/2241216549 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c88005020 con 0x7f1c90107820 2024-04-26T18:12:42.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.116+0000 7f1c95c08640 1 -- 172.21.15.31:0/2241216549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c901a78e0 con 0x7f1c90107820 2024-04-26T18:12:42.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.116+0000 7f1c8cff9640 1 -- 172.21.15.31:0/2241216549 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c880116d0 con 0x7f1c90107820 2024-04-26T18:12:42.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.116+0000 7f1c95c08640 1 -- 172.21.15.31:0/2241216549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c901a7e50 con 0x7f1c90107820 2024-04-26T18:12:42.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.117+0000 7f1c95c08640 1 -- 172.21.15.31:0/2241216549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c900506c0 con 0x7f1c90107820 2024-04-26T18:12:42.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.119+0000 7f1c8cff9640 1 -- 172.21.15.31:0/2241216549 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1c880051d0 con 0x7f1c90107820 2024-04-26T18:12:42.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.119+0000 7f1c8cff9640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1c780a3e40 0x7f1c780a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:42.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.119+0000 7f1c8cff9640 1 -- 172.21.15.31:0/2241216549 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8656+0+0 (secure 0 0 0) 0x7f1c880d0d70 con 0x7f1c90107820 2024-04-26T18:12:42.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.119+0000 7f1c8f7fe640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1c780a3e40 0x7f1c780a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:42.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.120+0000 7f1c8f7fe640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1c780a3e40 0x7f1c780a6300 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1c80009ba0 tx=0x7f1c80005f90 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:42.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.123+0000 7f1c8cff9640 1 -- 172.21.15.31:0/2241216549 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c88016030 con 0x7f1c90107820 2024-04-26T18:12:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.335+0000 7f1c95c08640 1 -- 172.21.15.31:0/2241216549 --> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f1c90062d70 con 0x7f1c780a3e40 2024-04-26T18:12:42.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.342+0000 7f1c8cff9640 1 -- 172.21.15.31:0/2241216549 <== mgr.24826 v2:172.21.15.31:6800/3595544994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f1c90062d70 con 0x7f1c780a3e40 2024-04-26T18:12:42.341 INFO:teuthology.orchestra.run.smithi031.stdout:/ 1m 6m3h 2024-04-26T18:12:42.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.344+0000 7f1c95c08640 1 -- 172.21.15.31:0/2241216549 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1c780a3e40 msgr2=0x7f1c780a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.344+0000 7f1c95c08640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1c780a3e40 0x7f1c780a6300 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1c80009ba0 tx=0x7f1c80005f90 comp rx=0 tx=0).stop 2024-04-26T18:12:42.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.344+0000 7f1c95c08640 1 -- 172.21.15.31:0/2241216549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c90107820 msgr2=0x7f1c900ff270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.344+0000 7f1c95c08640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c90107820 0x7f1c900ff270 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1c8800eaa0 tx=0x7f1c8800ef60 comp rx=0 tx=0).stop 2024-04-26T18:12:42.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.344+0000 7f1c95c08640 1 -- 172.21.15.31:0/2241216549 shutdown_connections 2024-04-26T18:12:42.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.344+0000 7f1c95c08640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1c780a3e40 0x7f1c780a6300 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.344+0000 7f1c95c08640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c90107820 0x7f1c900ff270 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.344+0000 7f1c95c08640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1c90102220 0x7f1c900fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.344+0000 7f1c95c08640 1 --2- 172.21.15.31:0/2241216549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c90101860 0x7f1c900fe7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.344+0000 7f1c95c08640 1 -- 172.21.15.31:0/2241216549 >> 172.21.15.31:0/2241216549 conn(0x7f1c900789b0 msgr2=0x7f1c9006a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:42.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.344+0000 7f1c95c08640 1 -- 172.21.15.31:0/2241216549 shutdown_connections 2024-04-26T18:12:42.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.344+0000 7f1c95c08640 1 -- 172.21.15.31:0/2241216549 wait complete. 2024-04-26T18:12:42.355 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-26T18:12:42.366 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T18:12:42.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.469+0000 7fc210fe4640 1 -- 172.21.15.31:0/210621302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc20c101880 msgr2=0x7fc20c101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.469+0000 7fc210fe4640 1 --2- 172.21.15.31:0/210621302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc20c101880 0x7fc20c101d00 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4009930 tx=0x7fc1f402a420 comp rx=0 tx=0).stop 2024-04-26T18:12:42.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.469+0000 7fc210fe4640 1 -- 172.21.15.31:0/210621302 shutdown_connections 2024-04-26T18:12:42.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.469+0000 7fc210fe4640 1 --2- 172.21.15.31:0/210621302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c102240 0x7fc20c10e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.469+0000 7fc210fe4640 1 --2- 172.21.15.31:0/210621302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc20c101880 0x7fc20c101d00 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.469+0000 7fc210fe4640 1 --2- 172.21.15.31:0/210621302 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc20c107840 0x7fc20c107c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.469+0000 7fc210fe4640 1 -- 172.21.15.31:0/210621302 >> 172.21.15.31:0/210621302 conn(0x7fc20c078990 msgr2=0x7fc20c078da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:42.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.469+0000 7fc210fe4640 1 -- 172.21.15.31:0/210621302 shutdown_connections 2024-04-26T18:12:42.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.469+0000 7fc210fe4640 1 -- 172.21.15.31:0/210621302 wait complete. 2024-04-26T18:12:42.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc210fe4640 1 Processor -- start 2024-04-26T18:12:42.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc210fe4640 1 -- start start 2024-04-26T18:12:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc210fe4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c101880 0x7fc20c19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc210fe4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc20c102240 0x7fc20c19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc210fe4640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc20c107840 0x7fc20c1a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc210fe4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc20c111b00 con 0x7fc20c102240 2024-04-26T18:12:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc210fe4640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fc20c111980 con 0x7fc20c107840 2024-04-26T18:12:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc210fe4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc20c111c80 con 0x7fc20c101880 2024-04-26T18:12:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc203fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc20c102240 0x7fc20c19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc203fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc20c102240 0x7fc20c19a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36398/0 (socket says 172.21.15.31:36398) 2024-04-26T18:12:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc203fff640 1 -- 172.21.15.31:0/2154445958 learned_addr learned my addr 172.21.15.31:0/2154445958 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:12:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc20ad9d640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc20c107840 0x7fc20c1a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc20a59c640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c101880 0x7fc20c19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc203fff640 1 -- 172.21.15.31:0/2154445958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c101880 msgr2=0x7fc20c19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc203fff640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c101880 0x7fc20c19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc203fff640 1 -- 172.21.15.31:0/2154445958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc20c107840 msgr2=0x7fc20c1a5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc203fff640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc20c107840 0x7fc20c1a5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc203fff640 1 -- 172.21.15.31:0/2154445958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc20c1a55c0 con 0x7fc20c102240 2024-04-26T18:12:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc20a59c640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c101880 0x7fc20c19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:12:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.470+0000 7fc203fff640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc20c102240 0x7fc20c19a540 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4005940 tx=0x7fc1f40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.471+0000 7fc2037fe640 1 -- 172.21.15.31:0/2154445958 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f403f030 con 0x7fc20c102240 2024-04-26T18:12:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.471+0000 7fc210fe4640 1 -- 172.21.15.31:0/2154445958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc20c1a5790 con 0x7fc20c102240 2024-04-26T18:12:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.471+0000 7fc2037fe640 1 -- 172.21.15.31:0/2154445958 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1f4003cb0 con 0x7fc20c102240 2024-04-26T18:12:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.471+0000 7fc2037fe640 1 -- 172.21.15.31:0/2154445958 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4035070 con 0x7fc20c102240 2024-04-26T18:12:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.471+0000 7fc210fe4640 1 -- 172.21.15.31:0/2154445958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc20c1a5c00 con 0x7fc20c102240 2024-04-26T18:12:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.472+0000 7fc210fe4640 1 -- 172.21.15.31:0/2154445958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc20c102f70 con 0x7fc20c102240 2024-04-26T18:12:42.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.473+0000 7fc2037fe640 1 -- 172.21.15.31:0/2154445958 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc1f4042030 con 0x7fc20c102240 2024-04-26T18:12:42.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.473+0000 7fc2037fe640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc1f80a3d20 0x7fc1f80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:42.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.474+0000 7fc20a59c640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc1f80a3d20 0x7fc1f80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:42.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.474+0000 7fc2037fe640 1 -- 172.21.15.31:0/2154445958 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8656+0+0 (secure 0 0 0) 0x7fc1f40f1300 con 0x7fc20c102240 2024-04-26T18:12:42.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.474+0000 7fc20a59c640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc1f80a3d20 0x7fc1f80a61e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc005e00 tx=0x7fc1fc005c70 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:42.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.477+0000 7fc2037fe640 1 -- 172.21.15.31:0/2154445958 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc1f40bd920 con 0x7fc20c102240 2024-04-26T18:12:42.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.665+0000 7fc210fe4640 1 -- 172.21.15.31:0/2154445958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc20c109dc0 con 0x7fc20c102240 2024-04-26T18:12:42.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.667+0000 7fc2037fe640 1 -- 172.21.15.31:0/2154445958 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) ==== 74+0+23309 (secure 0 0 0) 0x7fc1f40bd220 con 0x7fc20c102240 2024-04-26T18:12:42.667 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:42.667 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":111,"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","created":"2024-04-26T17:44:48.618250+0000","modified":"2024-04-26T18:12:18.840966+0000","last_up_change":"2024-04-26T17:52:00.798886+0000","last_in_change":"2024-04-26T17:51:40.613593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T17:48:12.307250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T17:52:58.650338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T17:53:02.488144+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":26,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f684ee5-eeee-454a-a00b-e5d6d2901572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6803","nonce":1957398328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6805","nonce":1957398328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6809","nonce":1957398328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6807","nonce":1957398328}]},"public_addr":"172.21.15.31:6803/1957398328","cluster_addr":"172.21.15.31:6805/1957398328","heartbeat_back_addr":"172.21.15.31:6809/1957398328","heartbeat_front_addr":"172.21.15.31:6807/1957398328","state":["exists","up"]},{"osd":1,"uuid":"08be6b26-cb60-41f7-be8b-01a0c93ebafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6801","nonce":4209866427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6803","nonce":4209866427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6807","nonce":4209866427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6805","nonce":4209866427}]},"public_addr":"172.21.15.97:6801/4209866427","cluster_addr":"172.21.15.97:6803/4209866427","heartbeat_back_addr":"172.21.15.97:6807/4209866427","heartbeat_front_addr":"172.21.15.97:6805/4209866427","state":["exists","up"]},{"osd":2,"uuid":"498216b1-a0ac-4923-ade8-ce50fb97d472","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6801","nonce":223535147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6803","nonce":223535147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6807","nonce":223535147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6805","nonce":223535147}]},"public_addr":"172.21.15.186:6801/223535147","cluster_addr":"172.21.15.186:6803/223535147","heartbeat_back_addr":"172.21.15.186:6807/223535147","heartbeat_front_addr":"172.21.15.186:6805/223535147","state":["exists","up"]},{"osd":3,"uuid":"568a352d-a430-4f0f-9a49-6f992e0ceae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6811","nonce":2640112341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6813","nonce":2640112341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6817","nonce":2640112341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6815","nonce":2640112341}]},"public_addr":"172.21.15.31:6811/2640112341","cluster_addr":"172.21.15.31:6813/2640112341","heartbeat_back_addr":"172.21.15.31:6817/2640112341","heartbeat_front_addr":"172.21.15.31:6815/2640112341","state":["exists","up"]},{"osd":4,"uuid":"112ccbf7-24e8-473c-9852-2e7167dd3108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6809","nonce":1359263143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6811","nonce":1359263143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6815","nonce":1359263143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6813","nonce":1359263143}]},"public_addr":"172.21.15.97:6809/1359263143","cluster_addr":"172.21.15.97:6811/1359263143","heartbeat_back_addr":"172.21.15.97:6815/1359263143","heartbeat_front_addr":"172.21.15.97:6813/1359263143","state":["exists","up"]},{"osd":5,"uuid":"eb17759e-1ad8-4bda-9f61-54d2a9781535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6809","nonce":2006134626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6811","nonce":2006134626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6815","nonce":2006134626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6813","nonce":2006134626}]},"public_addr":"172.21.15.186:6809/2006134626","cluster_addr":"172.21.15.186:6811/2006134626","heartbeat_back_addr":"172.21.15.186:6815/2006134626","heartbeat_front_addr":"172.21.15.186:6813/2006134626","state":["exists","up"]},{"osd":6,"uuid":"f1d11e38-40e9-4db4-868f-145f1ffc0d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6819","nonce":2955759395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6821","nonce":2955759395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6825","nonce":2955759395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6823","nonce":2955759395}]},"public_addr":"172.21.15.31:6819/2955759395","cluster_addr":"172.21.15.31:6821/2955759395","heartbeat_back_addr":"172.21.15.31:6825/2955759395","heartbeat_front_addr":"172.21.15.31:6823/2955759395","state":["exists","up"]},{"osd":7,"uuid":"4e95f728-23ff-4bca-9b94-afde2662a5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6817","nonce":3940083075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6819","nonce":3940083075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6823","nonce":3940083075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6821","nonce":3940083075}]},"public_addr":"172.21.15.97:6817/3940083075","cluster_addr":"172.21.15.97:6819/3940083075","heartbeat_back_addr":"172.21.15.97:6823/3940083075","heartbeat_front_addr":"172.21.15.97:6821/3940083075","state":["exists","up"]},{"osd":8,"uuid":"5f91c915-2568-4d20-b645-00dad2efe952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6817","nonce":3193558170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6819","nonce":3193558170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6823","nonce":3193558170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6821","nonce":3193558170}]},"public_addr":"172.21.15.186:6817/3193558170","cluster_addr":"172.21.15.186:6819/3193558170","heartbeat_back_addr":"172.21.15.186:6823/3193558170","heartbeat_front_addr":"172.21.15.186:6821/3193558170","state":["exists","up"]},{"osd":9,"uuid":"12fda87d-830a-461f-a6cb-4699c52dc5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6827","nonce":3660801905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6829","nonce":3660801905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6833","nonce":3660801905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6831","nonce":3660801905}]},"public_addr":"172.21.15.31:6827/3660801905","cluster_addr":"172.21.15.31:6829/3660801905","heartbeat_back_addr":"172.21.15.31:6833/3660801905","heartbeat_front_addr":"172.21.15.31:6831/3660801905","state":["exists","up"]},{"osd":10,"uuid":"92a639b3-918b-485c-bd30-258b4224993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6825","nonce":4097344903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6827","nonce":4097344903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6831","nonce":4097344903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6829","nonce":4097344903}]},"public_addr":"172.21.15.97:6825/4097344903","cluster_addr":"172.21.15.97:6827/4097344903","heartbeat_back_addr":"172.21.15.97:6831/4097344903","heartbeat_front_addr":"172.21.15.97:6829/4097344903","state":["exists","up"]},{"osd":11,"uuid":"842aa599-d21d-4e4a-b872-e9ac20d8c0ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6825","nonce":1063973352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6827","nonce":1063973352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6831","nonce":1063973352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6829","nonce":1063973352}]},"public_addr":"172.21.15.186:6825/1063973352","cluster_addr":"172.21.15.186:6827/1063973352","heartbeat_back_addr":"172.21.15.186:6831/1063973352","heartbeat_front_addr":"172.21.15.186:6829/1063973352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:21.305331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:44.877932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:08.784665+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:34.381762+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:58.999358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:23.620237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:49.961719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:15.363702+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:40.599703+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:07.391645+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:32.800851+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:59.384148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.186:6837/1510454985":"2024-04-27T18:04:55.814793+0000","172.21.15.97:6839/3765855684":"2024-04-27T18:04:17.185934+0000","172.21.15.97:6838/3765855684":"2024-04-27T18:04:17.185934+0000","172.21.15.31:0/3101596726":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/288292084":"2024-04-27T17:45:48.218816+0000","172.21.15.31:6801/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/3148082955":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/810461143":"2024-04-27T17:45:21.129062+0000","172.21.15.186:6836/1510454985":"2024-04-27T18:04:55.814793+0000","172.21.15.31:6801/153341213":"2024-04-27T17:45:48.218816+0000","172.21.15.31:6800/153341213":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/614760140":"2024-04-27T17:56:29.776588+0000","172.21.15.31:0/718966776":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/3429421652":"2024-04-27T17:56:29.776588+0000","172.21.15.31:0/2035551671":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/2095760002":"2024-04-27T17:56:29.776588+0000","172.21.15.31:6800/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/540352687":"2024-04-27T17:56:29.776588+0000","172.21.15.31:6800/2410008001":"2024-04-27T17:56:29.776588+0000","172.21.15.31:6801/2410008001":"2024-04-27T17:56:29.776588+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":13,"length":1},{"begin":26,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T18:12:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.669+0000 7fc210fe4640 1 -- 172.21.15.31:0/2154445958 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc1f80a3d20 msgr2=0x7fc1f80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.669+0000 7fc210fe4640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc1f80a3d20 0x7fc1f80a61e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc005e00 tx=0x7fc1fc005c70 comp rx=0 tx=0).stop 2024-04-26T18:12:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.669+0000 7fc210fe4640 1 -- 172.21.15.31:0/2154445958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc20c102240 msgr2=0x7fc20c19a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.669+0000 7fc210fe4640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc20c102240 0x7fc20c19a540 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4005940 tx=0x7fc1f40042f0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.669+0000 7fc210fe4640 1 -- 172.21.15.31:0/2154445958 shutdown_connections 2024-04-26T18:12:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.669+0000 7fc210fe4640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc20c107840 0x7fc20c1a5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.669+0000 7fc210fe4640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fc1f80a3d20 0x7fc1f80a61e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.669+0000 7fc210fe4640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc20c102240 0x7fc20c19a540 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.669+0000 7fc210fe4640 1 --2- 172.21.15.31:0/2154445958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc20c101880 0x7fc20c19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.669+0000 7fc210fe4640 1 -- 172.21.15.31:0/2154445958 >> 172.21.15.31:0/2154445958 conn(0x7fc20c078990 msgr2=0x7fc20c1044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.669+0000 7fc210fe4640 1 -- 172.21.15.31:0/2154445958 shutdown_connections 2024-04-26T18:12:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.669+0000 7fc210fe4640 1 -- 172.21.15.31:0/2154445958 wait complete. 2024-04-26T18:12:42.681 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T17:48:12.307250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T17:52:58.650338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T17:53:02.488144+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 26, 'snap_epoch': 110, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T18:12:42.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T18:12:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.788+0000 7f2b3d711640 1 -- 172.21.15.31:0/1320511897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b3806b970 msgr2=0x7f2b3810f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.788+0000 7f2b3d711640 1 --2- 172.21.15.31:0/1320511897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b3806b970 0x7f2b3810f7e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2b30009cd0 tx=0x7f2b3002a8e0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.788+0000 7f2b3d711640 1 -- 172.21.15.31:0/1320511897 shutdown_connections 2024-04-26T18:12:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.788+0000 7f2b3d711640 1 --2- 172.21.15.31:0/1320511897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b3806b970 0x7f2b3810f7e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.788+0000 7f2b3d711640 1 --2- 172.21.15.31:0/1320511897 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b3806afb0 0x7f2b3806b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.788+0000 7f2b3d711640 1 --2- 172.21.15.31:0/1320511897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b38108920 0x7f2b38108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.788+0000 7f2b3d711640 1 -- 172.21.15.31:0/1320511897 >> 172.21.15.31:0/1320511897 conn(0x7f2b38078890 msgr2=0x7f2b38078ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.788+0000 7f2b3d711640 1 -- 172.21.15.31:0/1320511897 shutdown_connections 2024-04-26T18:12:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.788+0000 7f2b3d711640 1 -- 172.21.15.31:0/1320511897 wait complete. 2024-04-26T18:12:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b3d711640 1 Processor -- start 2024-04-26T18:12:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b3d711640 1 -- start start 2024-04-26T18:12:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b3d711640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b3806afb0 0x7f2b3819b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b3d711640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b3806b970 0x7f2b3819b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b3d711640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b38108920 0x7f2b3819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b3d711640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2b38112bb0 con 0x7f2b3806b970 2024-04-26T18:12:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b3d711640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f2b38112a30 con 0x7f2b38108920 2024-04-26T18:12:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b3d711640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b38112d30 con 0x7f2b3806afb0 2024-04-26T18:12:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b2ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b3806b970 0x7f2b3819b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b2ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b3806b970 0x7f2b3819b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36418/0 (socket says 172.21.15.31:36418) 2024-04-26T18:12:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b2ffff640 1 -- 172.21.15.31:0/1276918768 learned_addr learned my addr 172.21.15.31:0/1276918768 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:12:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b36ffd640 1 --2- 172.21.15.31:0/1276918768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b3806afb0 0x7f2b3819b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b377fe640 1 --2- 172.21.15.31:0/1276918768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b38108920 0x7f2b3819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b2ffff640 1 -- 172.21.15.31:0/1276918768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b3806afb0 msgr2=0x7f2b3819b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b2ffff640 1 --2- 172.21.15.31:0/1276918768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b3806afb0 0x7f2b3819b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b2ffff640 1 -- 172.21.15.31:0/1276918768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b38108920 msgr2=0x7f2b3819fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b2ffff640 1 --2- 172.21.15.31:0/1276918768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b38108920 0x7f2b3819fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.789+0000 7f2b2ffff640 1 -- 172.21.15.31:0/1276918768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b381a02b0 con 0x7f2b3806b970 2024-04-26T18:12:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.790+0000 7f2b2ffff640 1 --2- 172.21.15.31:0/1276918768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b3806b970 0x7f2b3819b800 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000e9f0 tx=0x7f2b2000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.790+0000 7f2b34ff9640 1 -- 172.21.15.31:0/1276918768 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2000cd80 con 0x7f2b3806b970 2024-04-26T18:12:42.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.790+0000 7f2b3d711640 1 -- 172.21.15.31:0/1276918768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b381a05a0 con 0x7f2b3806b970 2024-04-26T18:12:42.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.790+0000 7f2b34ff9640 1 -- 172.21.15.31:0/1276918768 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b200108e0 con 0x7f2b3806b970 2024-04-26T18:12:42.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.790+0000 7f2b34ff9640 1 -- 172.21.15.31:0/1276918768 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b20018980 con 0x7f2b3806b970 2024-04-26T18:12:42.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.790+0000 7f2b3d711640 1 -- 172.21.15.31:0/1276918768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b381a7e30 con 0x7f2b3806b970 2024-04-26T18:12:42.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.791+0000 7f2b3d711640 1 -- 172.21.15.31:0/1276918768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b38069ba0 con 0x7f2b3806b970 2024-04-26T18:12:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.793+0000 7f2b34ff9640 1 -- 172.21.15.31:0/1276918768 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2b20010430 con 0x7f2b3806b970 2024-04-26T18:12:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.793+0000 7f2b34ff9640 1 --2- 172.21.15.31:0/1276918768 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2b240a3d20 0x7f2b240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.793+0000 7f2b36ffd640 1 --2- 172.21.15.31:0/1276918768 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2b240a3d20 0x7f2b240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.793+0000 7f2b34ff9640 1 -- 172.21.15.31:0/1276918768 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8656+0+0 (secure 0 0 0) 0x7f2b200d1170 con 0x7f2b3806b970 2024-04-26T18:12:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.794+0000 7f2b36ffd640 1 --2- 172.21.15.31:0/1276918768 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2b240a3d20 0x7f2b240a61e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2b28005e00 tx=0x7f2b28005c50 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:42.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.797+0000 7f2b34ff9640 1 -- 172.21.15.31:0/1276918768 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b2009d790 con 0x7f2b3806b970 2024-04-26T18:12:42.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.983+0000 7f2b3d711640 1 -- 172.21.15.31:0/1276918768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2b3802dad0 con 0x7f2b3806b970 2024-04-26T18:12:42.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.983+0000 7f2b34ff9640 1 -- 172.21.15.31:0/1276918768 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) ==== 93+0+10 (secure 0 0 0) 0x7f2b2009d090 con 0x7f2b3806b970 2024-04-26T18:12:42.983 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-26T18:12:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.985+0000 7f2b3d711640 1 -- 172.21.15.31:0/1276918768 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2b240a3d20 msgr2=0x7f2b240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.985+0000 7f2b3d711640 1 --2- 172.21.15.31:0/1276918768 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2b240a3d20 0x7f2b240a61e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2b28005e00 tx=0x7f2b28005c50 comp rx=0 tx=0).stop 2024-04-26T18:12:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.985+0000 7f2b3d711640 1 -- 172.21.15.31:0/1276918768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b3806b970 msgr2=0x7f2b3819b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.985+0000 7f2b3d711640 1 --2- 172.21.15.31:0/1276918768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b3806b970 0x7f2b3819b800 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000e9f0 tx=0x7f2b2000eeb0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.985+0000 7f2b3d711640 1 -- 172.21.15.31:0/1276918768 shutdown_connections 2024-04-26T18:12:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.985+0000 7f2b3d711640 1 --2- 172.21.15.31:0/1276918768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b38108920 0x7f2b3819fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.985+0000 7f2b3d711640 1 --2- 172.21.15.31:0/1276918768 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f2b240a3d20 0x7f2b240a61e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.985+0000 7f2b3d711640 1 --2- 172.21.15.31:0/1276918768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b3806b970 0x7f2b3819b800 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.985+0000 7f2b3d711640 1 --2- 172.21.15.31:0/1276918768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b3806afb0 0x7f2b3819b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.985+0000 7f2b3d711640 1 -- 172.21.15.31:0/1276918768 >> 172.21.15.31:0/1276918768 conn(0x7f2b38078890 msgr2=0x7f2b3810dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.985+0000 7f2b3d711640 1 -- 172.21.15.31:0/1276918768 shutdown_connections 2024-04-26T18:12:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:42.985+0000 7f2b3d711640 1 -- 172.21.15.31:0/1276918768 wait complete. 2024-04-26T18:12:42.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T18:12:43.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.102+0000 7f0062445640 1 -- 172.21.15.31:0/2979036451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005c06b8d0 msgr2=0x7f005c10ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:43.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.102+0000 7f0062445640 1 --2- 172.21.15.31:0/2979036451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005c06b8d0 0x7f005c10ee00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f0050009900 tx=0x7f005002a410 comp rx=0 tx=0).stop 2024-04-26T18:12:43.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.102+0000 7f0062445640 1 -- 172.21.15.31:0/2979036451 shutdown_connections 2024-04-26T18:12:43.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.102+0000 7f0062445640 1 --2- 172.21.15.31:0/2979036451 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f005c10f340 0x7f005c111730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.102+0000 7f0062445640 1 --2- 172.21.15.31:0/2979036451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005c06b8d0 0x7f005c10ee00 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.102+0000 7f0062445640 1 --2- 172.21.15.31:0/2979036451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f005c06afb0 0x7f005c06b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.102+0000 7f0062445640 1 -- 172.21.15.31:0/2979036451 >> 172.21.15.31:0/2979036451 conn(0x7f005c078890 msgr2=0x7f005c078ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:43.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.102+0000 7f0062445640 1 -- 172.21.15.31:0/2979036451 shutdown_connections 2024-04-26T18:12:43.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.102+0000 7f0062445640 1 -- 172.21.15.31:0/2979036451 wait complete. 2024-04-26T18:12:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.103+0000 7f0062445640 1 Processor -- start 2024-04-26T18:12:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.103+0000 7f0062445640 1 -- start start 2024-04-26T18:12:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.103+0000 7f0062445640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005c06afb0 0x7f005c19b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.103+0000 7f0062445640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f005c06b8d0 0x7f005c19b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.103+0000 7f0062445640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f005c10f340 0x7f005c19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.103+0000 7f0062445640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f005c114db0 con 0x7f005c06afb0 2024-04-26T18:12:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.103+0000 7f0062445640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f005c114c30 con 0x7f005c06b8d0 2024-04-26T18:12:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.103+0000 7f0062445640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f005c114f30 con 0x7f005c10f340 2024-04-26T18:12:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.103+0000 7f005bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005c06afb0 0x7f005c19b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.103+0000 7f005bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005c06afb0 0x7f005c19b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36424/0 (socket says 172.21.15.31:36424) 2024-04-26T18:12:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.103+0000 7f005bfff640 1 -- 172.21.15.31:0/3887222010 learned_addr learned my addr 172.21.15.31:0/3887222010 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:12:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.103+0000 7f005b7fe640 1 --2- 172.21.15.31:0/3887222010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f005c06b8d0 0x7f005c19b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.104+0000 7f00609e2640 1 --2- 172.21.15.31:0/3887222010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f005c10f340 0x7f005c19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:43.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.104+0000 7f005bfff640 1 -- 172.21.15.31:0/3887222010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f005c10f340 msgr2=0x7f005c19fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:43.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.104+0000 7f005bfff640 1 --2- 172.21.15.31:0/3887222010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f005c10f340 0x7f005c19fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.104+0000 7f005bfff640 1 -- 172.21.15.31:0/3887222010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f005c06b8d0 msgr2=0x7f005c19b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:43.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.104+0000 7f005bfff640 1 --2- 172.21.15.31:0/3887222010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f005c06b8d0 0x7f005c19b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.104+0000 7f005bfff640 1 -- 172.21.15.31:0/3887222010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005c1a0340 con 0x7f005c06afb0 2024-04-26T18:12:43.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.104+0000 7f005bfff640 1 --2- 172.21.15.31:0/3887222010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005c06afb0 0x7f005c19b350 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f004c00d770 tx=0x7f004c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:43.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.104+0000 7f00597fa640 1 -- 172.21.15.31:0/3887222010 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c017070 con 0x7f005c06afb0 2024-04-26T18:12:43.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.104+0000 7f0062445640 1 -- 172.21.15.31:0/3887222010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f005c1a0630 con 0x7f005c06afb0 2024-04-26T18:12:43.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.104+0000 7f00597fa640 1 -- 172.21.15.31:0/3887222010 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f004c0054d0 con 0x7f005c06afb0 2024-04-26T18:12:43.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.104+0000 7f00597fa640 1 -- 172.21.15.31:0/3887222010 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c004c30 con 0x7f005c06afb0 2024-04-26T18:12:43.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.104+0000 7f0062445640 1 -- 172.21.15.31:0/3887222010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f005c1a7ec0 con 0x7f005c06afb0 2024-04-26T18:12:43.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.105+0000 7f0062445640 1 -- 172.21.15.31:0/3887222010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f005c101e40 con 0x7f005c06afb0 2024-04-26T18:12:43.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.106+0000 7f00597fa640 1 -- 172.21.15.31:0/3887222010 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f004c004dd0 con 0x7f005c06afb0 2024-04-26T18:12:43.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.107+0000 7f00597fa640 1 --2- 172.21.15.31:0/3887222010 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f00440a3d20 0x7f00440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:43.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.107+0000 7f00597fa640 1 -- 172.21.15.31:0/3887222010 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8656+0+0 (secure 0 0 0) 0x7f004c0d0830 con 0x7f005c06afb0 2024-04-26T18:12:43.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.107+0000 7f005b7fe640 1 --2- 172.21.15.31:0/3887222010 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f00440a3d20 0x7f00440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:43.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.108+0000 7f005b7fe640 1 --2- 172.21.15.31:0/3887222010 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f00440a3d20 0x7f00440a61e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0050031440 tx=0x7f00500095f0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:43.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.111+0000 7f00597fa640 1 -- 172.21.15.31:0/3887222010 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f004c09ce10 con 0x7f005c06afb0 2024-04-26T18:12:43.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.298+0000 7f0062445640 1 -- 172.21.15.31:0/3887222010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f005c10a4d0 con 0x7f005c06afb0 2024-04-26T18:12:43.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.299+0000 7f00597fa640 1 -- 172.21.15.31:0/3887222010 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) ==== 104+0+11 (secure 0 0 0) 0x7f004c09c710 con 0x7f005c06afb0 2024-04-26T18:12:43.298 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-26T18:12:43.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.301+0000 7f0062445640 1 -- 172.21.15.31:0/3887222010 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f00440a3d20 msgr2=0x7f00440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.301+0000 7f0062445640 1 --2- 172.21.15.31:0/3887222010 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f00440a3d20 0x7f00440a61e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0050031440 tx=0x7f00500095f0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.301+0000 7f0062445640 1 -- 172.21.15.31:0/3887222010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005c06afb0 msgr2=0x7f005c19b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.301+0000 7f0062445640 1 --2- 172.21.15.31:0/3887222010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005c06afb0 0x7f005c19b350 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f004c00d770 tx=0x7f004c00dc30 comp rx=0 tx=0).stop 2024-04-26T18:12:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.301+0000 7f0062445640 1 -- 172.21.15.31:0/3887222010 shutdown_connections 2024-04-26T18:12:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.301+0000 7f0062445640 1 --2- 172.21.15.31:0/3887222010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f005c10f340 0x7f005c19fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.301+0000 7f0062445640 1 --2- 172.21.15.31:0/3887222010 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f005c06b8d0 0x7f005c19b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.301+0000 7f0062445640 1 --2- 172.21.15.31:0/3887222010 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f00440a3d20 0x7f00440a61e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.301+0000 7f0062445640 1 --2- 172.21.15.31:0/3887222010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005c06afb0 0x7f005c19b350 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.301+0000 7f0062445640 1 -- 172.21.15.31:0/3887222010 >> 172.21.15.31:0/3887222010 conn(0x7f005c078890 msgr2=0x7f005c110890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.301+0000 7f0062445640 1 -- 172.21.15.31:0/3887222010 shutdown_connections 2024-04-26T18:12:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.301+0000 7f0062445640 1 -- 172.21.15.31:0/3887222010 wait complete. 2024-04-26T18:12:43.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:43 smithi031 ceph-mon[29164]: from='client.16173 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T18:12:43.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:43 smithi031 ceph-mon[29164]: pgmap v509: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 23 MiB/s wr, 1.52k op/s 2024-04-26T18:12:43.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:43 smithi031 ceph-mon[29164]: from='client.16179 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T18:12:43.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:43 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2154445958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T18:12:43.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:43 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1276918768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T18:12:43.312 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T18:12:43.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.418+0000 7fda6d933640 1 -- 172.21.15.31:0/640629800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda681044b0 msgr2=0x7fda6810c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.418+0000 7fda6d933640 1 --2- 172.21.15.31:0/640629800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda681044b0 0x7fda6810c990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fda5c009900 tx=0x7fda5c02a480 comp rx=0 tx=0).stop 2024-04-26T18:12:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.418+0000 7fda6d933640 1 -- 172.21.15.31:0/640629800 shutdown_connections 2024-04-26T18:12:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.418+0000 7fda6d933640 1 --2- 172.21.15.31:0/640629800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda6810cf60 0x7fda6810f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.418+0000 7fda6d933640 1 --2- 172.21.15.31:0/640629800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda681044b0 0x7fda6810c990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.418+0000 7fda6d933640 1 --2- 172.21.15.31:0/640629800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda68103b90 0x7fda68103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.418+0000 7fda6d933640 1 -- 172.21.15.31:0/640629800 >> 172.21.15.31:0/640629800 conn(0x7fda68077bb0 msgr2=0x7fda68077fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.418+0000 7fda6d933640 1 -- 172.21.15.31:0/640629800 shutdown_connections 2024-04-26T18:12:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.418+0000 7fda6d933640 1 -- 172.21.15.31:0/640629800 wait complete. 2024-04-26T18:12:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.419+0000 7fda6d933640 1 Processor -- start 2024-04-26T18:12:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.419+0000 7fda6d933640 1 -- start start 2024-04-26T18:12:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.419+0000 7fda6d933640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda68103b90 0x7fda68199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.419+0000 7fda6d933640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda681044b0 0x7fda68199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.419+0000 7fda6d933640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda6810cf60 0x7fda6819dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.419+0000 7fda6d933640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fda68112c00 con 0x7fda6810cf60 2024-04-26T18:12:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.419+0000 7fda6d933640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fda68112a80 con 0x7fda681044b0 2024-04-26T18:12:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.419+0000 7fda6d933640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda68112d80 con 0x7fda68103b90 2024-04-26T18:12:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.419+0000 7fda677fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda6810cf60 0x7fda6819dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.419+0000 7fda677fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda6810cf60 0x7fda6819dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36438/0 (socket says 172.21.15.31:36438) 2024-04-26T18:12:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.419+0000 7fda677fe640 1 -- 172.21.15.31:0/3465760541 learned_addr learned my addr 172.21.15.31:0/3465760541 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:12:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.419+0000 7fda66ffd640 1 --2- 172.21.15.31:0/3465760541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda68103b90 0x7fda68199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.419+0000 7fda667fc640 1 --2- 172.21.15.31:0/3465760541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda681044b0 0x7fda68199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.420+0000 7fda677fe640 1 -- 172.21.15.31:0/3465760541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda68103b90 msgr2=0x7fda68199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.420+0000 7fda677fe640 1 --2- 172.21.15.31:0/3465760541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda68103b90 0x7fda68199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.420+0000 7fda677fe640 1 -- 172.21.15.31:0/3465760541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda681044b0 msgr2=0x7fda68199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.420+0000 7fda677fe640 1 --2- 172.21.15.31:0/3465760541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda681044b0 0x7fda68199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.420+0000 7fda677fe640 1 -- 172.21.15.31:0/3465760541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda6819e1f0 con 0x7fda6810cf60 2024-04-26T18:12:43.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.420+0000 7fda677fe640 1 --2- 172.21.15.31:0/3465760541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda6810cf60 0x7fda6819dac0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fda6000cab0 tx=0x7fda6000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:43.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.420+0000 7fda6c931640 1 -- 172.21.15.31:0/3465760541 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6000ad60 con 0x7fda6810cf60 2024-04-26T18:12:43.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.420+0000 7fda6d933640 1 -- 172.21.15.31:0/3465760541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda6819e4e0 con 0x7fda6810cf60 2024-04-26T18:12:43.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.420+0000 7fda6c931640 1 -- 172.21.15.31:0/3465760541 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda6000f900 con 0x7fda6810cf60 2024-04-26T18:12:43.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.420+0000 7fda6c931640 1 -- 172.21.15.31:0/3465760541 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60018980 con 0x7fda6810cf60 2024-04-26T18:12:43.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.420+0000 7fda6d933640 1 -- 172.21.15.31:0/3465760541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda681a5d70 con 0x7fda6810cf60 2024-04-26T18:12:43.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.421+0000 7fda6d933640 1 -- 172.21.15.31:0/3465760541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda6806b750 con 0x7fda6810cf60 2024-04-26T18:12:43.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.422+0000 7fda6c931640 1 -- 172.21.15.31:0/3465760541 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fda6000f450 con 0x7fda6810cf60 2024-04-26T18:12:43.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.423+0000 7fda6c931640 1 --2- 172.21.15.31:0/3465760541 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fda500a3d20 0x7fda500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:43.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.423+0000 7fda66ffd640 1 --2- 172.21.15.31:0/3465760541 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fda500a3d20 0x7fda500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:43.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.423+0000 7fda6c931640 1 -- 172.21.15.31:0/3465760541 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8656+0+0 (secure 0 0 0) 0x7fda600d0b30 con 0x7fda6810cf60 2024-04-26T18:12:43.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.424+0000 7fda66ffd640 1 --2- 172.21.15.31:0/3465760541 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fda500a3d20 0x7fda500a61e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fda58009990 tx=0x7fda58009110 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:43.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.428+0000 7fda6c931640 1 -- 172.21.15.31:0/3465760541 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda60014030 con 0x7fda6810cf60 2024-04-26T18:12:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:43 smithi186 ceph-mon[32797]: from='client.16173 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T18:12:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:43 smithi186 ceph-mon[32797]: pgmap v509: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 23 MiB/s wr, 1.52k op/s 2024-04-26T18:12:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:43 smithi186 ceph-mon[32797]: from='client.16179 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T18:12:43.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:43 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2154445958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T18:12:43.518 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:43 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1276918768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T18:12:43.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.614+0000 7fda6d933640 1 -- 172.21.15.31:0/3465760541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fda6819a550 con 0x7fda6810cf60 2024-04-26T18:12:43.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.614+0000 7fda6c931640 1 -- 172.21.15.31:0/3465760541 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) ==== 100+0+11 (secure 0 0 0) 0x7fda6009d150 con 0x7fda6810cf60 2024-04-26T18:12:43.614 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-26T18:12:43.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.616+0000 7fda6d933640 1 -- 172.21.15.31:0/3465760541 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fda500a3d20 msgr2=0x7fda500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:43.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.616+0000 7fda6d933640 1 --2- 172.21.15.31:0/3465760541 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fda500a3d20 0x7fda500a61e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fda58009990 tx=0x7fda58009110 comp rx=0 tx=0).stop 2024-04-26T18:12:43.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.616+0000 7fda6d933640 1 -- 172.21.15.31:0/3465760541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda6810cf60 msgr2=0x7fda6819dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:43.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.616+0000 7fda6d933640 1 --2- 172.21.15.31:0/3465760541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda6810cf60 0x7fda6819dac0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fda6000cab0 tx=0x7fda6000ceb0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.616+0000 7fda6d933640 1 -- 172.21.15.31:0/3465760541 shutdown_connections 2024-04-26T18:12:43.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.616+0000 7fda6d933640 1 --2- 172.21.15.31:0/3465760541 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fda500a3d20 0x7fda500a61e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.616+0000 7fda6d933640 1 --2- 172.21.15.31:0/3465760541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda6810cf60 0x7fda6819dac0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.616+0000 7fda6d933640 1 --2- 172.21.15.31:0/3465760541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda681044b0 0x7fda68199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.616+0000 7fda6d933640 1 --2- 172.21.15.31:0/3465760541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda68103b90 0x7fda68199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.616+0000 7fda6d933640 1 -- 172.21.15.31:0/3465760541 >> 172.21.15.31:0/3465760541 conn(0x7fda68077bb0 msgr2=0x7fda680fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:43.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.616+0000 7fda6d933640 1 -- 172.21.15.31:0/3465760541 shutdown_connections 2024-04-26T18:12:43.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.616+0000 7fda6d933640 1 -- 172.21.15.31:0/3465760541 wait complete. 2024-04-26T18:12:43.627 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-26T18:12:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:43 smithi097 ceph-mon[32419]: from='client.16173 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T18:12:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:43 smithi097 ceph-mon[32419]: pgmap v509: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 23 MiB/s wr, 1.52k op/s 2024-04-26T18:12:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:43 smithi097 ceph-mon[32419]: from='client.16179 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T18:12:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:43 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2154445958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T18:12:43.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:43 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1276918768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T18:12:43.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.731+0000 7f3174c25640 1 -- 172.21.15.31:0/1295438013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3170107820 msgr2=0x7f3170107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:43.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.731+0000 7f3174c25640 1 --2- 172.21.15.31:0/1295438013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3170107820 0x7f3170107c00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3160009930 tx=0x7f316002a420 comp rx=0 tx=0).stop 2024-04-26T18:12:43.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.731+0000 7f3174c25640 1 -- 172.21.15.31:0/1295438013 shutdown_connections 2024-04-26T18:12:43.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.731+0000 7f3174c25640 1 --2- 172.21.15.31:0/1295438013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31701021d0 0x7f317010e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.731+0000 7f3174c25640 1 --2- 172.21.15.31:0/1295438013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170101810 0x7f3170101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.731+0000 7f3174c25640 1 --2- 172.21.15.31:0/1295438013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3170107820 0x7f3170107c00 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.731+0000 7f3174c25640 1 -- 172.21.15.31:0/1295438013 >> 172.21.15.31:0/1295438013 conn(0x7f3170078930 msgr2=0x7f3170078d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:43.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.731+0000 7f3174c25640 1 -- 172.21.15.31:0/1295438013 shutdown_connections 2024-04-26T18:12:43.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.731+0000 7f3174c25640 1 -- 172.21.15.31:0/1295438013 wait complete. 2024-04-26T18:12:43.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f3174c25640 1 Processor -- start 2024-04-26T18:12:43.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f3174c25640 1 -- start start 2024-04-26T18:12:43.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f3174c25640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3170101810 0x7f3170199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:43.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f3174c25640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31701021d0 0x7f3170199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:43.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f3174c25640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170107820 0x7f317019d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:43.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f3174c25640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3170111aa0 con 0x7f31701021d0 2024-04-26T18:12:43.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f3174c25640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3170111920 con 0x7f3170107820 2024-04-26T18:12:43.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f3174c25640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3170111c20 con 0x7f3170101810 2024-04-26T18:12:43.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f316dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31701021d0 0x7f3170199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:43.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f316e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3170101810 0x7f3170199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:43.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f316dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31701021d0 0x7f3170199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36456/0 (socket says 172.21.15.31:36456) 2024-04-26T18:12:43.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f316dd9b640 1 -- 172.21.15.31:0/1375474921 learned_addr learned my addr 172.21.15.31:0/1375474921 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:12:43.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f316ed9d640 1 --2- 172.21.15.31:0/1375474921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170107820 0x7f317019d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:43.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f316dd9b640 1 -- 172.21.15.31:0/1375474921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3170101810 msgr2=0x7f3170199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:43.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f316dd9b640 1 --2- 172.21.15.31:0/1375474921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3170101810 0x7f3170199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f316dd9b640 1 -- 172.21.15.31:0/1375474921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170107820 msgr2=0x7f317019d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:43.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f316dd9b640 1 --2- 172.21.15.31:0/1375474921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170107820 0x7f317019d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:43.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.732+0000 7f316dd9b640 1 -- 172.21.15.31:0/1375474921 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317019e120 con 0x7f31701021d0 2024-04-26T18:12:43.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.733+0000 7f316dd9b640 1 --2- 172.21.15.31:0/1375474921 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31701021d0 0x7f3170199670 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f316400c890 tx=0x7f316400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:43.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.733+0000 7f315f7fe640 1 -- 172.21.15.31:0/1375474921 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3164004220 con 0x7f31701021d0 2024-04-26T18:12:43.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.733+0000 7f3174c25640 1 -- 172.21.15.31:0/1375474921 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f317019e410 con 0x7f31701021d0 2024-04-26T18:12:43.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.733+0000 7f315f7fe640 1 -- 172.21.15.31:0/1375474921 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31640043c0 con 0x7f31701021d0 2024-04-26T18:12:43.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.733+0000 7f315f7fe640 1 -- 172.21.15.31:0/1375474921 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3164004a00 con 0x7f31701021d0 2024-04-26T18:12:43.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.733+0000 7f3174c25640 1 -- 172.21.15.31:0/1375474921 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31701a5ca0 con 0x7f31701021d0 2024-04-26T18:12:43.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.734+0000 7f3174c25640 1 -- 172.21.15.31:0/1375474921 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3170069750 con 0x7f31701021d0 2024-04-26T18:12:43.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.735+0000 7f315f7fe640 1 -- 172.21.15.31:0/1375474921 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f31640079b0 con 0x7f31701021d0 2024-04-26T18:12:43.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.735+0000 7f315f7fe640 1 --2- 172.21.15.31:0/1375474921 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f31580a3d20 0x7f31580a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:12:43.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.736+0000 7f316e59c640 1 --2- 172.21.15.31:0/1375474921 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f31580a3d20 0x7f31580a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:12:43.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.736+0000 7f315f7fe640 1 -- 172.21.15.31:0/1375474921 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8656+0+0 (secure 0 0 0) 0x7f31640d0560 con 0x7f31701021d0 2024-04-26T18:12:43.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.736+0000 7f316e59c640 1 --2- 172.21.15.31:0/1375474921 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f31580a3d20 0x7f31580a61e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3160009900 tx=0x7f3160000bc0 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:12:43.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.740+0000 7f315f7fe640 1 -- 172.21.15.31:0/1375474921 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3164014030 con 0x7f31701021d0 2024-04-26T18:12:43.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.980+0000 7f3174c25640 1 -- 172.21.15.31:0/1375474921 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f317010adf0 con 0x7f31701021d0 2024-04-26T18:12:43.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.988+0000 7f315f7fe640 1 -- 172.21.15.31:0/1375474921 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746063 (secure 0 0 0) 0x7f316409c4f0 con 0x7f31701021d0 2024-04-26T18:12:43.999 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:12:43.999 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T18:12:43.999 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 26, 2024-04-26T18:12:43.999 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:43.999 INFO:teuthology.orchestra.run.smithi031.stdout: "active_gid": 24826, 2024-04-26T18:12:43.999 INFO:teuthology.orchestra.run.smithi031.stdout: "active_name": "x", 2024-04-26T18:12:43.999 INFO:teuthology.orchestra.run.smithi031.stdout: "active_addrs": { 2024-04-26T18:12:44.000 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-26T18:12:44.000 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.000 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-26T18:12:44.000 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6800", 2024-04-26T18:12:44.000 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3595544994 2024-04-26T18:12:44.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.000 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.000 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-04-26T18:12:44.000 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6801", 2024-04-26T18:12:44.000 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3595544994 2024-04-26T18:12:44.000 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.000 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-26T18:12:44.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.001 INFO:teuthology.orchestra.run.smithi031.stdout: "active_addr": "172.21.15.31:6801/3595544994", 2024-04-26T18:12:44.001 INFO:teuthology.orchestra.run.smithi031.stdout: "active_change": "2024-04-26T17:56:29.776787+0000", 2024-04-26T18:12:44.001 INFO:teuthology.orchestra.run.smithi031.stdout: "active_mgr_features": 4540701547738038271, 2024-04-26T18:12:44.001 INFO:teuthology.orchestra.run.smithi031.stdout: "available": true, 2024-04-26T18:12:44.001 INFO:teuthology.orchestra.run.smithi031.stdout: "standbys": [ 2024-04-26T18:12:44.001 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.001 INFO:teuthology.orchestra.run.smithi031.stdout: "gid": 24820, 2024-04-26T18:12:44.001 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "z", 2024-04-26T18:12:44.001 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_features": 4540701547738038271, 2024-04-26T18:12:44.001 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2024-04-26T18:12:44.001 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.001 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2024-04-26T18:12:44.001 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.001 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.002 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.002 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:44.002 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:44.002 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.002 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.002 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.002 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:44.002 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.002 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.002 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.002 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T18:12:44.002 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2024-04-26T18:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2024-04-26T18:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2024-04-26T18:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2024-04-26T18:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2024-04-26T18:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2024-04-26T18:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2024-04-26T18:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.011 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2024-04-26T18:12:44.011 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2024-04-26T18:12:44.011 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.011 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.011 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.011 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.011 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.011 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.011 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.011 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2024-04-26T18:12:44.011 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.011 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.012 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.012 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2024-04-26T18:12:44.012 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2024-04-26T18:12:44.012 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.012 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.012 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.012 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.012 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.012 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.012 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.012 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2024-04-26T18:12:44.012 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.012 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.013 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.013 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2024-04-26T18:12:44.013 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2024-04-26T18:12:44.013 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.013 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.013 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.013 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2024-04-26T18:12:44.013 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.013 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.013 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.013 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2024-04-26T18:12:44.013 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.014 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.014 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.014 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2024-04-26T18:12:44.014 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2024-04-26T18:12:44.014 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.014 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.014 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.014 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.014 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.014 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.014 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.015 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2024-04-26T18:12:44.015 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.015 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.015 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.015 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2024-04-26T18:12:44.015 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2024-04-26T18:12:44.015 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.015 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.015 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.015 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.015 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.016 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.016 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.016 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T18:12:44.016 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.016 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.016 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.016 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2024-04-26T18:12:44.016 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2024-04-26T18:12:44.016 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.016 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.016 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.016 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.017 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.017 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.017 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.017 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2024-04-26T18:12:44.017 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.017 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.017 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.018 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.018 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.018 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.018 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2024-04-26T18:12:44.018 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.019 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.019 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.019 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2024-04-26T18:12:44.019 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2024-04-26T18:12:44.019 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.019 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.020 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.020 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.020 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.020 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.020 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.020 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T18:12:44.021 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.021 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.021 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.021 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2024-04-26T18:12:44.021 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2024-04-26T18:12:44.022 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.022 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.022 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.022 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2024-04-26T18:12:44.022 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.022 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.023 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.023 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T18:12:44.023 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T18:12:44.023 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.023 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.024 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2024-04-26T18:12:44.024 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2024-04-26T18:12:44.024 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:44.024 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.024 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.024 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:44.024 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-26T18:12:44.024 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-26T18:12:44.024 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.024 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T18:12:44.024 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T18:12:44.024 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.025 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.025 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2024-04-26T18:12:44.025 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2024-04-26T18:12:44.025 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:44.025 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.025 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.025 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2024-04-26T18:12:44.025 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-26T18:12:44.025 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2024-04-26T18:12:44.025 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.025 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T18:12:44.026 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.026 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.026 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.026 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2024-04-26T18:12:44.026 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2024-04-26T18:12:44.026 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.026 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.026 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.026 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T18:12:44.027 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.027 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.027 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.027 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T18:12:44.027 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T18:12:44.027 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.027 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.027 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2024-04-26T18:12:44.027 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2024-04-26T18:12:44.027 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.027 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.027 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.027 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2024-04-26T18:12:44.028 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2024-04-26T18:12:44.028 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2024-04-26T18:12:44.028 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.028 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2024-04-26T18:12:44.028 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T18:12:44.028 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.028 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.028 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2024-04-26T18:12:44.028 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2024-04-26T18:12:44.028 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.029 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.029 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.029 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2024-04-26T18:12:44.029 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.029 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.029 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.029 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T18:12:44.029 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T18:12:44.029 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.029 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.030 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2024-04-26T18:12:44.030 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2024-04-26T18:12:44.030 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:44.030 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.030 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.030 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:44.030 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-26T18:12:44.030 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-26T18:12:44.030 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.030 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T18:12:44.031 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T18:12:44.031 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.031 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.031 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.031 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.031 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.031 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.031 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.031 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.031 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.031 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.031 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.032 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.032 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.032 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.032 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.032 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.032 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.032 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.032 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.032 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.032 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.033 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.033 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.033 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.033 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.033 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.033 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.033 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.033 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.033 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.033 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.033 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.034 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.034 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.034 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.034 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.034 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.034 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.034 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.034 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.034 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.034 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.034 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.034 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.035 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.035 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.035 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.035 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.035 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.035 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.035 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.035 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.035 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.035 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.035 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.036 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.036 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.036 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.036 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.036 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.036 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.036 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.036 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.036 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.036 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.036 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.037 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.037 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.037 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.037 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2024-04-26T18:12:44.037 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2024-04-26T18:12:44.037 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.037 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.037 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.037 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:44.037 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.037 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.037 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.038 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T18:12:44.038 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.038 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.038 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.038 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-26T18:12:44.038 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-26T18:12:44.038 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.038 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.038 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.038 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2024-04-26T18:12:44.039 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.039 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.039 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.039 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2024-04-26T18:12:44.039 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2024-04-26T18:12:44.039 INFO:teuthology.orchestra.run.smithi031.stdout: "read", 2024-04-26T18:12:44.039 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap", 2024-04-26T18:12:44.039 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap-read" 2024-04-26T18:12:44.039 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.039 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2024-04-26T18:12:44.040 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.040 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.040 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.040 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2024-04-26T18:12:44.040 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2024-04-26T18:12:44.041 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.041 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.041 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.041 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.041 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.041 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.042 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.042 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T18:12:44.042 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.042 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.042 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.043 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:44.043 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:44.043 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.043 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.043 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.043 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:44.043 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.044 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.044 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.044 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T18:12:44.044 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.044 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.044 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.044 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2024-04-26T18:12:44.044 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2024-04-26T18:12:44.044 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.045 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.045 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.045 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-26T18:12:44.045 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-26T18:12:44.045 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.045 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.045 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T18:12:44.046 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T18:12:44.046 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.046 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.046 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2024-04-26T18:12:44.046 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2024-04-26T18:12:44.046 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:44.046 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.046 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.046 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.046 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.046 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.047 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.047 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T18:12:44.047 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.047 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.047 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.047 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.047 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.047 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2024-04-26T18:12:44.047 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.047 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.047 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.048 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2024-04-26T18:12:44.048 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2024-04-26T18:12:44.048 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.048 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.048 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.048 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-26T18:12:44.048 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.048 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.048 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.048 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T18:12:44.049 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.049 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.049 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.049 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2024-04-26T18:12:44.049 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2024-04-26T18:12:44.049 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.049 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.049 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.049 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2024-04-26T18:12:44.049 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.050 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.050 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.050 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T18:12:44.050 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.050 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.050 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.050 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2024-04-26T18:12:44.050 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2024-04-26T18:12:44.050 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.050 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.050 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.050 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2024-04-26T18:12:44.051 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.051 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.051 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.051 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T18:12:44.051 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.051 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.051 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.051 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2024-04-26T18:12:44.051 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2024-04-26T18:12:44.052 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.052 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.052 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.052 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.052 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.052 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.052 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.052 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T18:12:44.052 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T18:12:44.052 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.052 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.053 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2024-04-26T18:12:44.053 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2024-04-26T18:12:44.053 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.053 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.053 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.053 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:44.053 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.053 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.053 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.053 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T18:12:44.053 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.053 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.053 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.054 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2024-04-26T18:12:44.054 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T18:12:44.054 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.054 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.054 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.054 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2024-04-26T18:12:44.054 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.054 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.054 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.054 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T18:12:44.055 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.055 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.055 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.055 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_log_destination": { 2024-04-26T18:12:44.055 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm_log_destination", 2024-04-26T18:12:44.055 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: "file", 2024-04-26T18:12:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: "file,syslog", 2024-04-26T18:12:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: "syslog" 2024-04-26T18:12:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T18:12:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.057 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2024-04-26T18:12:44.057 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2024-04-26T18:12:44.057 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.057 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.057 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.057 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.058 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.058 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.058 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.058 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T18:12:44.058 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.058 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.059 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.059 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2024-04-26T18:12:44.059 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2024-04-26T18:12:44.059 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.059 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.059 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.059 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.059 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.059 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.059 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.059 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T18:12:44.059 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.060 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.060 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.060 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2024-04-26T18:12:44.060 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2024-04-26T18:12:44.060 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.060 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.060 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.060 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.060 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.060 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.060 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.061 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T18:12:44.061 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.061 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.061 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.061 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2024-04-26T18:12:44.061 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2024-04-26T18:12:44.061 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.061 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.061 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.061 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T18:12:44.062 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.062 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.062 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.062 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-26T18:12:44.062 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.062 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.062 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.062 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2024-04-26T18:12:44.062 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2024-04-26T18:12:44.062 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.062 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.063 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.063 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T18:12:44.063 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.063 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.063 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.063 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2024-04-26T18:12:44.063 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.063 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.063 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.063 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2024-04-26T18:12:44.063 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2024-04-26T18:12:44.063 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.063 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.064 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.064 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T18:12:44.064 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.064 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.064 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.064 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2024-04-26T18:12:44.064 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.064 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.064 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.065 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2024-04-26T18:12:44.065 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2024-04-26T18:12:44.065 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.065 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.065 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.065 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T18:12:44.065 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.065 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.065 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.065 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-26T18:12:44.066 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.066 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.066 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.066 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2024-04-26T18:12:44.066 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2024-04-26T18:12:44.066 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.066 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.066 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.066 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T18:12:44.066 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.067 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.067 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.067 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2024-04-26T18:12:44.067 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.067 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.067 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.067 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2024-04-26T18:12:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2024-04-26T18:12:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T18:12:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2024-04-26T18:12:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2024-04-26T18:12:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2024-04-26T18:12:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T18:12:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2024-04-26T18:12:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2024-04-26T18:12:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2024-04-26T18:12:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T18:12:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2024-04-26T18:12:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2024-04-26T18:12:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2024-04-26T18:12:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T18:12:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2024-04-26T18:12:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2024-04-26T18:12:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2024-04-26T18:12:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T18:12:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2024-04-26T18:12:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2024-04-26T18:12:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2024-04-26T18:12:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T18:12:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-26T18:12:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nvmeof": { 2024-04-26T18:12:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nvmeof", 2024-04-26T18:12:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T18:12:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nvme-of container image", 2024-04-26T18:12:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2024-04-26T18:12:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2024-04-26T18:12:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T18:12:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-26T18:12:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2024-04-26T18:12:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2024-04-26T18:12:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T18:12:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2024-04-26T18:12:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_samba": { 2024-04-26T18:12:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_samba", 2024-04-26T18:12:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T18:12:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Samba/SMB container image", 2024-04-26T18:12:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2024-04-26T18:12:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2024-04-26T18:12:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T18:12:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2024-04-26T18:12:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2024-04-26T18:12:44.081 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2024-04-26T18:12:44.081 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.081 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.081 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.081 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.082 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.082 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.082 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.082 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T18:12:44.082 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.082 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.082 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.082 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2024-04-26T18:12:44.082 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2024-04-26T18:12:44.082 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.082 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.082 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.083 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:44.083 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.085 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.085 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.085 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T18:12:44.085 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.086 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.086 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.086 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2024-04-26T18:12:44.086 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T18:12:44.086 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.086 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.086 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.086 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2024-04-26T18:12:44.086 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.087 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.087 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.087 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T18:12:44.087 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.087 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.087 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.087 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2024-04-26T18:12:44.087 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2024-04-26T18:12:44.087 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.087 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.088 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.088 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2024-04-26T18:12:44.088 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.088 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.088 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.088 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T18:12:44.088 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.088 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.088 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.088 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2024-04-26T18:12:44.088 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2024-04-26T18:12:44.088 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.089 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.089 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.089 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2024-04-26T18:12:44.089 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.089 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.089 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.089 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2024-04-26T18:12:44.089 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.089 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.089 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.089 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2024-04-26T18:12:44.089 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2024-04-26T18:12:44.090 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.090 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.090 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.090 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.090 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.090 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.090 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.090 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T18:12:44.090 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.090 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.091 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.091 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2024-04-26T18:12:44.091 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2024-04-26T18:12:44.091 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.091 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.091 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.091 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:44.091 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.091 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.091 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.092 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2024-04-26T18:12:44.092 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.092 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.092 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.092 INFO:teuthology.orchestra.run.smithi031.stdout: "grafana_dashboards_path": { 2024-04-26T18:12:44.092 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "grafana_dashboards_path", 2024-04-26T18:12:44.092 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.092 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.092 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.092 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T18:12:44.093 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.093 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.093 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.093 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T18:12:44.093 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.093 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.093 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.093 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2024-04-26T18:12:44.093 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2024-04-26T18:12:44.093 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.093 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.093 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.093 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:44.094 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.094 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.094 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.094 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2024-04-26T18:12:44.094 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.094 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.094 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.094 INFO:teuthology.orchestra.run.smithi031.stdout: "hw_monitoring": { 2024-04-26T18:12:44.094 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hw_monitoring", 2024-04-26T18:12:44.095 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.095 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.095 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.095 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.095 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.095 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.095 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.095 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T18:12:44.095 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.095 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.095 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.096 INFO:teuthology.orchestra.run.smithi031.stdout: "inventory_list_all": { 2024-04-26T18:12:44.096 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "inventory_list_all", 2024-04-26T18:12:44.096 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.096 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.096 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.096 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.096 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.096 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.096 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.096 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T18:12:44.096 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.096 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.096 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.097 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.097 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.097 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.097 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.097 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.097 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.097 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.097 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.097 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.097 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.097 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.097 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.098 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.098 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.098 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.098 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.098 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.098 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.098 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.099 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.099 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2024-04-26T18:12:44.099 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2024-04-26T18:12:44.099 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.099 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.099 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.100 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.100 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.100 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.100 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.100 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T18:12:44.101 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.101 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.101 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.101 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.101 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.102 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.102 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.102 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.102 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.102 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.102 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.102 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.102 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T18:12:44.102 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.103 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.103 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.103 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.103 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.103 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.103 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.104 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.104 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.104 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.104 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.104 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.105 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.105 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.105 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.105 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.105 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.105 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.106 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.106 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.106 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.106 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.106 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.106 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.106 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.107 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.107 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.107 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.107 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.107 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.107 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.107 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.107 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.108 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.108 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.108 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.108 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T18:12:44.108 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T18:12:44.108 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.108 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.108 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.109 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.109 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.109 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.109 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.109 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T18:12:44.109 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.109 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.109 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.110 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T18:12:44.110 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T18:12:44.110 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.110 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.110 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.110 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2024-04-26T18:12:44.110 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.111 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.111 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.111 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T18:12:44.111 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.111 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.111 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.111 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2024-04-26T18:12:44.111 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2024-04-26T18:12:44.112 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.112 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.112 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.112 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.112 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.112 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.112 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.112 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2024-04-26T18:12:44.112 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.113 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.113 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.113 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2024-04-26T18:12:44.113 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2024-04-26T18:12:44.113 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.113 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.113 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.114 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.114 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.114 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.114 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.114 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T18:12:44.114 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.114 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.114 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.115 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2024-04-26T18:12:44.115 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2024-04-26T18:12:44.115 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.115 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.115 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.115 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.115 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.115 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.116 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.116 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2024-04-26T18:12:44.116 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.116 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.116 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-26T18:12:44.116 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-26T18:12:44.116 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.117 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.117 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.117 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2024-04-26T18:12:44.117 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.117 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.117 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.117 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2024-04-26T18:12:44.117 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2024-04-26T18:12:44.118 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.118 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T18:12:44.118 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.118 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.118 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.118 INFO:teuthology.orchestra.run.smithi031.stdout: "oob_default_addr": { 2024-04-26T18:12:44.118 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "oob_default_addr", 2024-04-26T18:12:44.118 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.118 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.118 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.119 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "169.254.1.1", 2024-04-26T18:12:44.119 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.119 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.119 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.119 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T18:12:44.119 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.119 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.119 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.119 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2024-04-26T18:12:44.119 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2024-04-26T18:12:44.119 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.119 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.119 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.120 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T18:12:44.120 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.120 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.120 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.120 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T18:12:44.120 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.120 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.120 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.120 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2024-04-26T18:12:44.120 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2024-04-26T18:12:44.120 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.120 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.121 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.121 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.121 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.121 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.121 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.121 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T18:12:44.121 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.121 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.121 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.121 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2024-04-26T18:12:44.121 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2024-04-26T18:12:44.121 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.121 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.122 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.122 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.122 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.122 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.122 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.122 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T18:12:44.122 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.122 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.122 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.122 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2024-04-26T18:12:44.122 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2024-04-26T18:12:44.122 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.122 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.123 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.123 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.123 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.123 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.123 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.123 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T18:12:44.123 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.123 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.123 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.123 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2024-04-26T18:12:44.123 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2024-04-26T18:12:44.123 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.123 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.124 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.124 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.124 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.124 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.124 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.124 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T18:12:44.124 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.124 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.124 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.124 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2024-04-26T18:12:44.124 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2024-04-26T18:12:44.124 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.124 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T18:12:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2024-04-26T18:12:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2024-04-26T18:12:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.125 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.126 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.126 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2024-04-26T18:12:44.126 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.126 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.126 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.126 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2024-04-26T18:12:44.126 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.126 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.126 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.126 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2024-04-26T18:12:44.126 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2024-04-26T18:12:44.126 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.127 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.128 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.128 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.128 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.128 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.128 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.128 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T18:12:44.128 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.128 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.128 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.128 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2024-04-26T18:12:44.128 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2024-04-26T18:12:44.128 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.128 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.129 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.129 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.129 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.129 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.129 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.129 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T18:12:44.129 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.129 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.129 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.129 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2024-04-26T18:12:44.130 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2024-04-26T18:12:44.130 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.130 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.130 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.130 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.130 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.130 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.130 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.130 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T18:12:44.130 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.130 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.130 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.130 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2024-04-26T18:12:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2024-04-26T18:12:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T18:12:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.131 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.133 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2024-04-26T18:12:44.133 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.133 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.133 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.133 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.133 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.133 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.133 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.133 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.133 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.133 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.133 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.134 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.134 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.134 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.134 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.134 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.134 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.134 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.134 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.134 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.134 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.134 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.135 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.135 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.135 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.135 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.135 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.135 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.135 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.135 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.135 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.135 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.135 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.135 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.135 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.136 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.136 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.136 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.136 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.136 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.136 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.136 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.136 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.136 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.136 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.136 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.136 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.136 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.137 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.138 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.138 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.138 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.138 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.138 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.138 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.138 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.138 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.138 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2024-04-26T18:12:44.138 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2024-04-26T18:12:44.138 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.138 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.139 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.139 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2024-04-26T18:12:44.139 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.139 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.139 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.139 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T18:12:44.139 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.139 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.139 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.139 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2024-04-26T18:12:44.139 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2024-04-26T18:12:44.139 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.139 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.140 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.140 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2024-04-26T18:12:44.140 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.140 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.140 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.140 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T18:12:44.140 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.140 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.140 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.140 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.140 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2024-04-26T18:12:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T18:12:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T18:12:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.141 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.142 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.142 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.142 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.142 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T18:12:44.142 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T18:12:44.142 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.142 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.142 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.142 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.142 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.143 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.143 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.143 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.143 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.143 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.143 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.143 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T18:12:44.143 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T18:12:44.143 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.143 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.143 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.143 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.143 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.144 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.144 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.144 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.144 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.144 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.144 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.144 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2024-04-26T18:12:44.144 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T18:12:44.144 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.144 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.144 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.144 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.144 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.145 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.145 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.145 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.145 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.145 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.145 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.145 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T18:12:44.145 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T18:12:44.145 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.145 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.145 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.145 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.146 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.146 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.146 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.146 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.146 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.146 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.146 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.146 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T18:12:44.146 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T18:12:44.146 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.146 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.146 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.146 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.147 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.147 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.147 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.147 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.147 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.147 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.147 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.147 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T18:12:44.147 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T18:12:44.147 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.147 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.147 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.148 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.148 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.148 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.148 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.148 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.148 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.148 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.148 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.148 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T18:12:44.148 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T18:12:44.148 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.148 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.149 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.149 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.149 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.149 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.149 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.149 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.149 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.149 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.149 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.149 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T18:12:44.149 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T18:12:44.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.149 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T18:12:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T18:12:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.151 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.151 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.151 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.151 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.151 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.151 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.151 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.151 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.151 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.151 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.151 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T18:12:44.151 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T18:12:44.151 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.152 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.152 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.152 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.152 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.152 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.152 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.152 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.152 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.152 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.152 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.152 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T18:12:44.152 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T18:12:44.153 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.153 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.153 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.153 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.153 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.153 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.153 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.153 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.153 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.153 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.153 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.153 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T18:12:44.154 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T18:12:44.154 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.154 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.154 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.154 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.154 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.154 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.154 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.154 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.154 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.154 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.154 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.154 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T18:12:44.155 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T18:12:44.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.155 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.155 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.155 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.155 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.155 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.155 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.155 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.155 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.155 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.155 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.156 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T18:12:44.156 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T18:12:44.156 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.156 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.156 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.156 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-26T18:12:44.156 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.156 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.156 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.156 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.156 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.156 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.156 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.157 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T18:12:44.157 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T18:12:44.157 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.157 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.157 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.157 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.157 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.157 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.157 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.157 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.157 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.157 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.157 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.158 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2024-04-26T18:12:44.158 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2024-04-26T18:12:44.158 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.158 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.158 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.158 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.158 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.158 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.158 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.158 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.158 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.158 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.158 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.159 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T18:12:44.159 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T18:12:44.159 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.159 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.159 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.159 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-26T18:12:44.159 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.159 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.159 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.159 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.159 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.159 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.160 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.160 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T18:12:44.160 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T18:12:44.160 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.160 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.160 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.160 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.160 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.160 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.160 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.160 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.160 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.161 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.161 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.161 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T18:12:44.161 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T18:12:44.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.161 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.161 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.161 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.161 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.161 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.161 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.162 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.162 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.162 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.162 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.162 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T18:12:44.162 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T18:12:44.162 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.162 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.162 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.162 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.162 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.162 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.162 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.163 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.163 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.163 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.163 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.163 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T18:12:44.163 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T18:12:44.163 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.163 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.163 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.163 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.163 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.163 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.164 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.164 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.164 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.164 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.164 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.164 INFO:teuthology.orchestra.run.smithi031.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T18:12:44.164 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T18:12:44.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.164 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.164 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.164 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "[]", 2024-04-26T18:12:44.164 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.164 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.165 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.165 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.165 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.165 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.165 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.165 INFO:teuthology.orchestra.run.smithi031.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T18:12:44.165 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T18:12:44.165 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.165 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.165 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.165 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "{}", 2024-04-26T18:12:44.165 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.166 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.166 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.166 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.166 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.166 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.166 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.166 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T18:12:44.166 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T18:12:44.166 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.166 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.166 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.166 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.167 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.167 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.167 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.167 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.167 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.167 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.167 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.167 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T18:12:44.167 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T18:12:44.167 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.167 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.167 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.168 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.168 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.168 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.168 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.168 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.168 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.168 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.168 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.168 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T18:12:44.168 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T18:12:44.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.168 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.169 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.169 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.169 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.169 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.169 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.169 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.169 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.169 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.169 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.169 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T18:12:44.169 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T18:12:44.169 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.170 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.170 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.170 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.170 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.170 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.170 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.170 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.170 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.170 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.170 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.170 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T18:12:44.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T18:12:44.170 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.171 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.171 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.171 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.171 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.171 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.171 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.171 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.171 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.171 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.171 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.171 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T18:12:44.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T18:12:44.172 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.172 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.172 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.172 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.172 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.172 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.172 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.172 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.172 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.172 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.172 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.172 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T18:12:44.173 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T18:12:44.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.173 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.173 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.173 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.173 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.173 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.173 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.173 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.173 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.173 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.173 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.174 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T18:12:44.174 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T18:12:44.174 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.174 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.174 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.174 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.174 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.174 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.174 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.174 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.174 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.174 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.174 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.175 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T18:12:44.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T18:12:44.175 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.175 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.175 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.175 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.175 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.175 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.175 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.175 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.175 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.176 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.176 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.176 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T18:12:44.176 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T18:12:44.176 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.176 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.176 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.176 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.176 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.176 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.176 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.176 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.176 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T18:12:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T18:12:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T18:12:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.177 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.178 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.178 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.178 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T18:12:44.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T18:12:44.178 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.178 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.178 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.178 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.179 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.179 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.179 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.179 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.179 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.179 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.179 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.179 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T18:12:44.179 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T18:12:44.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.179 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.179 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.180 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2024-04-26T18:12:44.180 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.180 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.180 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.180 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.180 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.180 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.180 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.180 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T18:12:44.180 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T18:12:44.180 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.180 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.180 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.181 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2024-04-26T18:12:44.181 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.181 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.181 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.181 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.181 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.181 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.181 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.181 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T18:12:44.181 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T18:12:44.181 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.181 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.181 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.182 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.182 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.182 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.182 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.182 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.182 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.182 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.182 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.182 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T18:12:44.182 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T18:12:44.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.182 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.182 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.183 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-26T18:12:44.183 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.183 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.183 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.183 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.183 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.183 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.183 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.183 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T18:12:44.183 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T18:12:44.183 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.183 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.183 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T18:12:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T18:12:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.184 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.185 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.185 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.185 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.185 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.185 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.185 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.185 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.185 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.185 INFO:teuthology.orchestra.run.smithi031.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T18:12:44.185 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T18:12:44.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.185 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.185 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.186 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.186 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.186 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.186 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.186 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.186 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.186 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.186 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.186 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T18:12:44.186 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T18:12:44.186 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.186 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.187 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.187 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:44.187 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.187 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.187 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.187 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.187 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.187 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.187 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.187 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T18:12:44.187 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T18:12:44.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.187 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.188 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.188 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.188 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.188 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.188 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.188 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.188 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.188 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.188 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.188 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T18:12:44.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T18:12:44.188 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.189 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.189 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.189 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-26T18:12:44.189 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.189 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.189 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.189 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.189 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.189 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.189 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.189 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2024-04-26T18:12:44.189 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2024-04-26T18:12:44.189 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.190 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.190 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.190 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.190 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.190 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.190 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.190 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.190 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.190 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.190 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.190 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2024-04-26T18:12:44.190 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2024-04-26T18:12:44.191 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.191 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.191 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.191 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.191 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.191 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.191 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.191 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.191 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.191 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.191 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.191 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2024-04-26T18:12:44.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2024-04-26T18:12:44.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.192 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.192 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.192 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.192 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.192 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.192 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.192 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2024-04-26T18:12:44.192 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.192 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.192 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.193 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2024-04-26T18:12:44.193 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2024-04-26T18:12:44.193 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.193 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.193 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.193 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2024-04-26T18:12:44.193 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.193 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.193 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.193 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.193 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.193 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.193 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.194 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-26T18:12:44.194 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-26T18:12:44.194 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.194 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.194 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.194 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.194 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.194 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.194 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.194 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.194 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.194 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.194 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.195 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.195 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.195 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.195 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.195 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.195 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.195 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.195 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.195 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.195 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.195 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.195 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.195 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.197 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.197 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.197 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.197 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.197 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.197 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.197 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.197 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.197 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.197 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.197 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.197 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.197 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.198 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.198 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.198 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.198 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.198 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.198 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.198 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.198 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.198 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.198 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.198 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.198 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.198 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.199 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.199 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.199 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.199 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.199 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.199 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.199 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.199 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.199 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.199 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.199 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.199 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.200 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.200 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2024-04-26T18:12:44.200 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2024-04-26T18:12:44.200 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.200 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.200 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.200 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.200 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.200 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.200 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.201 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2024-04-26T18:12:44.201 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.201 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.201 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.201 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2024-04-26T18:12:44.201 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T18:12:44.201 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.201 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.201 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.201 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.201 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.201 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.202 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.202 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.202 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.202 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.202 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.202 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-26T18:12:44.202 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-26T18:12:44.202 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.202 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.202 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.202 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-26T18:12:44.202 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.203 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.203 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.203 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.203 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.203 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.203 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.203 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-26T18:12:44.203 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-26T18:12:44.203 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.204 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.204 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.204 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2024-04-26T18:12:44.204 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.204 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.204 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.204 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.204 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.204 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.204 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.204 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-26T18:12:44.204 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-26T18:12:44.204 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.205 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.205 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.205 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.205 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.205 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.205 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.205 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.205 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.205 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.205 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.205 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2024-04-26T18:12:44.205 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2024-04-26T18:12:44.205 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.206 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.206 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.206 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2024-04-26T18:12:44.206 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.206 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.206 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.206 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.206 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.206 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.206 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.206 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-26T18:12:44.206 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-26T18:12:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2024-04-26T18:12:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2024-04-26T18:12:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.207 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.208 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-26T18:12:44.208 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-26T18:12:44.208 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.208 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.208 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.208 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-26T18:12:44.208 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-26T18:12:44.208 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-26T18:12:44.208 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.208 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.208 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.208 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.208 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.209 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2024-04-26T18:12:44.209 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2024-04-26T18:12:44.209 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.209 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.209 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.209 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.209 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.209 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.209 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.209 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.209 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.209 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.210 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.211 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.211 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.211 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.211 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2024-04-26T18:12:44.211 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.211 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.211 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.211 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.211 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.211 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.211 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.211 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.212 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.212 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.212 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.212 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.212 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.212 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.212 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.212 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.212 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.212 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.212 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.212 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.212 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.212 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.213 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.214 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.215 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.216 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.216 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.216 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.216 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.216 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.216 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.216 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.216 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2024-04-26T18:12:44.216 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2024-04-26T18:12:44.216 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.216 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.216 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.216 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2024-04-26T18:12:44.217 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.217 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.217 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.217 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T18:12:44.217 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.217 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.217 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.217 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2024-04-26T18:12:44.217 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2024-04-26T18:12:44.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.217 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.217 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.218 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2024-04-26T18:12:44.218 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.218 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.218 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.218 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T18:12:44.218 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.218 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.218 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.218 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2024-04-26T18:12:44.218 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2024-04-26T18:12:44.218 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.218 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.218 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.219 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2024-04-26T18:12:44.219 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.219 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.219 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.219 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2024-04-26T18:12:44.219 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.219 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.219 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.219 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2024-04-26T18:12:44.219 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2024-04-26T18:12:44.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.219 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.219 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.220 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-26T18:12:44.220 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.220 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.220 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.220 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T18:12:44.220 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.220 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.220 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.220 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2024-04-26T18:12:44.220 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2024-04-26T18:12:44.220 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.220 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.220 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.221 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.221 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.221 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.221 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.221 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T18:12:44.221 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.221 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.222 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.222 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:44.222 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:44.222 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.222 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.223 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.223 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:44.223 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.223 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.223 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.223 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T18:12:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2024-04-26T18:12:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2024-04-26T18:12:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.225 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.225 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.225 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2024-04-26T18:12:44.225 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.225 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.225 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.226 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T18:12:44.226 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.226 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.226 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.227 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2024-04-26T18:12:44.227 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.227 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.227 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.228 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.228 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.228 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.228 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.228 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.228 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.229 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.229 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.229 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.229 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.229 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.229 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.230 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.230 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.230 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.230 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.230 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.230 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.231 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.231 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.231 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.231 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.232 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.232 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.232 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.232 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.232 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.232 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.233 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.233 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.233 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.233 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.234 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.234 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.234 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.234 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.234 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.234 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.235 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.235 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.235 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.235 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.235 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.235 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.236 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.236 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.236 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.236 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.236 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.236 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.237 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.237 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.237 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.237 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.237 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.238 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.238 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.238 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.238 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.238 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.238 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.239 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.239 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.239 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.239 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.239 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2024-04-26T18:12:44.239 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2024-04-26T18:12:44.240 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.240 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.240 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.240 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-26T18:12:44.240 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.240 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.240 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.240 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.240 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.240 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.240 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.240 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2024-04-26T18:12:44.241 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2024-04-26T18:12:44.241 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.241 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.241 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.241 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2024-04-26T18:12:44.241 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.241 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.241 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.241 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.241 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.241 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.241 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.243 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.243 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2024-04-26T18:12:44.243 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2024-04-26T18:12:44.243 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2024-04-26T18:12:44.243 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.243 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2024-04-26T18:12:44.243 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2024-04-26T18:12:44.243 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.243 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.243 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.244 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2024-04-26T18:12:44.244 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.244 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.244 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.244 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T18:12:44.244 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.244 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.244 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.244 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2024-04-26T18:12:44.244 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2024-04-26T18:12:44.244 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.244 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.244 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.245 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2024-04-26T18:12:44.245 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.245 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.245 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.245 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T18:12:44.245 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.245 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.245 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.245 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2024-04-26T18:12:44.245 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2024-04-26T18:12:44.245 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.245 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.245 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.246 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.246 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.246 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.246 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.246 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2024-04-26T18:12:44.246 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.246 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.246 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.246 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:44.246 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:44.246 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.246 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.246 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.247 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2024-04-26T18:12:44.247 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2024-04-26T18:12:44.247 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.247 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.247 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T18:12:44.247 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.247 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.247 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.247 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.247 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.247 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.247 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.247 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.248 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.249 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.249 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.249 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.249 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.249 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.249 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.249 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.249 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.249 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.249 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.249 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.249 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.249 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.250 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.250 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.250 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.250 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.250 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.250 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.250 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.250 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.250 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.250 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.250 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.250 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.251 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.251 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.251 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.251 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.251 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.251 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.251 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.251 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.251 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.251 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.251 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.251 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.251 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.252 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.252 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.252 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.252 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.252 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.252 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.252 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.252 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.252 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2024-04-26T18:12:44.252 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2024-04-26T18:12:44.252 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.252 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.252 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2024-04-26T18:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2024-04-26T18:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2024-04-26T18:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.254 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2024-04-26T18:12:44.254 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.254 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.254 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.254 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2024-04-26T18:12:44.254 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.254 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.254 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.254 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-26T18:12:44.254 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-26T18:12:44.254 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.254 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.255 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.255 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2024-04-26T18:12:44.255 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.255 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.255 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.255 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T18:12:44.255 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.255 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.255 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.255 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2024-04-26T18:12:44.255 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2024-04-26T18:12:44.255 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.255 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.256 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.256 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-26T18:12:44.256 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-26T18:12:44.256 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2024-04-26T18:12:44.256 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.256 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T18:12:44.256 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.256 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.256 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.256 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2024-04-26T18:12:44.256 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2024-04-26T18:12:44.256 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.256 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.257 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.257 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.257 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.257 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.257 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.257 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2024-04-26T18:12:44.257 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.257 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.257 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.257 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2024-04-26T18:12:44.257 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2024-04-26T18:12:44.257 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.257 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2024-04-26T18:12:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T18:12:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2024-04-26T18:12:44.259 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.259 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.259 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.259 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.259 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.259 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.259 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.259 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.259 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.259 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.259 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.259 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.259 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.259 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.260 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.260 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.260 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.260 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.260 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.260 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.260 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.260 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.260 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.260 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.260 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.260 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.261 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.262 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.262 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.262 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.262 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.262 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.262 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.262 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.262 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.262 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.262 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.262 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.262 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.262 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.262 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.263 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.264 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.264 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2024-04-26T18:12:44.264 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.264 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.264 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.264 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.264 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.264 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.264 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.265 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.265 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.265 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.265 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.265 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.265 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.265 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.265 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.265 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.265 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.265 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.265 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.265 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.266 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2024-04-26T18:12:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2024-04-26T18:12:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2024-04-26T18:12:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2024-04-26T18:12:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T18:12:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2024-04-26T18:12:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2024-04-26T18:12:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2024-04-26T18:12:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T18:12:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.274 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.274 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.274 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.274 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.274 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.274 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.274 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.274 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.274 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.274 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.274 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.274 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.274 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.275 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.275 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.275 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.275 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.275 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.275 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.275 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.275 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.275 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.275 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.275 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.275 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.275 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.276 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.276 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.276 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.276 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.276 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.276 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.276 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.276 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.276 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.276 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.276 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.276 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.277 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.277 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.277 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.277 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.277 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.277 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.277 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.277 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.277 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.277 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2024-04-26T18:12:44.277 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.277 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2024-04-26T18:12:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2024-04-26T18:12:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2024-04-26T18:12:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2024-04-26T18:12:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T18:12:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2024-04-26T18:12:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2024-04-26T18:12:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.285 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2024-04-26T18:12:44.285 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T18:12:44.285 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.285 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.285 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2024-04-26T18:12:44.285 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2024-04-26T18:12:44.285 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.285 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.285 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.285 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-04-26T18:12:44.285 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.285 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.286 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.286 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2024-04-26T18:12:44.286 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.286 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.286 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.286 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2024-04-26T18:12:44.286 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2024-04-26T18:12:44.286 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.286 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.286 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.286 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2024-04-26T18:12:44.286 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.287 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.287 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.287 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T18:12:44.287 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.287 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.287 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.287 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2024-04-26T18:12:44.287 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2024-04-26T18:12:44.287 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.287 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.287 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.287 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.287 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.288 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.288 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.288 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2024-04-26T18:12:44.288 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.288 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.288 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.288 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2024-04-26T18:12:44.288 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2024-04-26T18:12:44.288 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.288 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.288 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.288 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2024-04-26T18:12:44.289 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.289 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.289 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.289 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T18:12:44.289 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T18:12:44.289 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.289 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.289 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.289 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.289 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.289 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2024-04-26T18:12:44.289 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.290 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.290 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.290 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.290 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.290 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.290 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.290 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.290 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.290 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.290 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.290 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.290 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.290 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.290 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.291 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.291 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.291 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.291 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.291 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.291 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.291 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.291 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.291 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.291 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.291 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.291 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.291 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.292 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.293 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.293 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.293 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.293 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.293 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.293 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.293 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.293 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.293 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.293 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.293 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.293 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.293 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.293 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.294 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.294 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.294 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.294 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.294 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.294 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.294 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.294 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.294 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.294 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.294 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.294 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.294 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.295 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.296 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.296 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.296 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.296 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.296 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.296 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.296 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.296 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.296 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.296 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.296 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.296 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.297 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.297 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.297 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.297 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.297 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.297 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.297 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.297 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.297 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.297 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.297 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.297 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.297 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.298 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.298 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.298 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.298 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.298 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.298 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.298 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.298 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.298 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.298 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.298 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.298 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.298 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.299 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.299 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.299 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.299 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.299 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.299 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.299 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.299 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.299 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.299 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.299 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.299 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.299 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.300 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.300 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.300 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.300 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.300 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.300 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.300 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.300 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.300 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.300 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.300 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2024-04-26T18:12:44.300 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.301 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.302 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.302 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.302 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.302 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.302 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.302 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.302 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.302 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.302 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.302 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.302 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.302 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.304 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.304 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.304 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.304 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.304 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.304 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.304 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.304 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.304 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.304 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.304 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.304 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.304 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.304 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.305 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.306 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-26T18:12:44.306 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.306 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.306 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.306 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2024-04-26T18:12:44.306 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2024-04-26T18:12:44.306 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.306 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.306 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.306 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T18:12:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2024-04-26T18:12:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-26T18:12:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-04-26T18:12:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2024-04-26T18:12:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2024-04-26T18:12:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2024-04-26T18:12:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2024-04-26T18:12:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2024-04-26T18:12:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2024-04-26T18:12:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2024-04-26T18:12:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2024-04-26T18:12:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-26T18:12:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2024-04-26T18:12:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2024-04-26T18:12:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T18:12:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2024-04-26T18:12:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2024-04-26T18:12:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2024-04-26T18:12:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T18:12:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-26T18:12:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-26T18:12:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2024-04-26T18:12:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2024-04-26T18:12:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-26T18:12:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2024-04-26T18:12:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2024-04-26T18:12:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2024-04-26T18:12:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2024-04-26T18:12:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "exclude_perf_counters": { 2024-04-26T18:12:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "exclude_perf_counters", 2024-04-26T18:12:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T18:12:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T18:12:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2024-04-26T18:12:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2024-04-26T18:12:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T18:12:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T18:12:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2024-04-26T18:12:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2024-04-26T18:12:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2024-04-26T18:12:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2024-04-26T18:12:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-26T18:12:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-26T18:12:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-26T18:12:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T18:12:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-26T18:12:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-26T18:12:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2024-04-26T18:12:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T18:12:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2024-04-26T18:12:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2024-04-26T18:12:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2024-04-26T18:12:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-26T18:12:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-26T18:12:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2024-04-26T18:12:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2024-04-26T18:12:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2024-04-26T18:12:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-26T18:12:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-26T18:12:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-26T18:12:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-26T18:12:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-26T18:12:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-04-26T18:12:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2024-04-26T18:12:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T18:12:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2024-04-26T18:12:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2024-04-26T18:12:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2024-04-26T18:12:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2024-04-26T18:12:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2024-04-26T18:12:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-26T18:12:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-26T18:12:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-26T18:12:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-26T18:12:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-26T18:12:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-26T18:12:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2024-04-26T18:12:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T18:12:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2024-04-26T18:12:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2024-04-26T18:12:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2024-04-26T18:12:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2024-04-26T18:12:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2024-04-26T18:12:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2024-04-26T18:12:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2024-04-26T18:12:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2024-04-26T18:12:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2024-04-26T18:12:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2024-04-26T18:12:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2024-04-26T18:12:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2024-04-26T18:12:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2024-04-26T18:12:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2024-04-26T18:12:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2024-04-26T18:12:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2024-04-26T18:12:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2024-04-26T18:12:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2024-04-26T18:12:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2024-04-26T18:12:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2024-04-26T18:12:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2024-04-26T18:12:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2024-04-26T18:12:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2024-04-26T18:12:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-04-26T18:12:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2024-04-26T18:12:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2024-04-26T18:12:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T18:12:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2024-04-26T18:12:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2024-04-26T18:12:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2024-04-26T18:12:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.412 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.412 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.412 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.412 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.412 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.412 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.412 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.412 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.412 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.412 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.412 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.412 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.412 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2024-04-26T18:12:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.416 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.417 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.419 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.420 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2024-04-26T18:12:44.420 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.420 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.420 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.420 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.420 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.420 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.420 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.420 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.420 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.420 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.420 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.421 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.422 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.422 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.422 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.422 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.422 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.422 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.422 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.422 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.422 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.422 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.422 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.422 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.422 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.423 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.423 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.423 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.423 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.423 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.423 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.423 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.423 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.423 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.423 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.423 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.423 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.424 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2024-04-26T18:12:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.425 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2024-04-26T18:12:44.426 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2024-04-26T18:12:44.426 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.426 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.426 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.426 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T18:12:44.426 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.426 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.426 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.426 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.426 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.426 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.426 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.427 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:44.427 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:44.427 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.427 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.427 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.427 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2024-04-26T18:12:44.427 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.427 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.427 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.427 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.427 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.427 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.427 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.428 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.428 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.428 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.428 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.428 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.428 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.428 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.428 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.428 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.428 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.428 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.428 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.428 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.428 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.429 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.430 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.430 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.430 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.430 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.430 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.430 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.430 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.430 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.430 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.430 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.430 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.430 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.430 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.431 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.431 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.431 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.431 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.431 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.431 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.431 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.431 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.431 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.431 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.431 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.431 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.431 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.433 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2024-04-26T18:12:44.433 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.433 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.433 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.433 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2024-04-26T18:12:44.433 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2024-04-26T18:12:44.433 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.433 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.433 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.433 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.433 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.433 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.433 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.434 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T18:12:44.434 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.434 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.434 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.434 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2024-04-26T18:12:44.434 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2024-04-26T18:12:44.434 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.434 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.434 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.434 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.434 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.434 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.434 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.435 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T18:12:44.435 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.435 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.435 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.435 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2024-04-26T18:12:44.435 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2024-04-26T18:12:44.435 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.435 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.435 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.435 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.435 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.435 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.435 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.436 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T18:12:44.436 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.436 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.436 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.436 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2024-04-26T18:12:44.436 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2024-04-26T18:12:44.436 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.436 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.436 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.436 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.436 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.436 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.437 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.437 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T18:12:44.437 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.437 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.437 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.437 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2024-04-26T18:12:44.437 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2024-04-26T18:12:44.437 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.437 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.437 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.437 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.437 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.438 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.438 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.438 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T18:12:44.438 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.438 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.438 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.438 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2024-04-26T18:12:44.438 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2024-04-26T18:12:44.438 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.438 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.438 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.438 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.439 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.439 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.439 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.439 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.439 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.439 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.439 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.439 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2024-04-26T18:12:44.439 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2024-04-26T18:12:44.439 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.439 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.439 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.439 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.440 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.440 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.440 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.440 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.440 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.440 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.440 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.440 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2024-04-26T18:12:44.440 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2024-04-26T18:12:44.440 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.440 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.440 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.440 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T18:12:44.441 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.441 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.441 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.441 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.441 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.441 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.441 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.441 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-26T18:12:44.441 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-26T18:12:44.441 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.441 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.441 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.441 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.442 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.442 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.442 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.442 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.442 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.442 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.442 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.442 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.442 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:44.442 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:44.442 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.442 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.442 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.442 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2024-04-26T18:12:44.443 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2024-04-26T18:12:44.443 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.443 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.443 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.443 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.443 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.443 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.443 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2024-04-26T18:12:44.443 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2024-04-26T18:12:44.443 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.443 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.443 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2024-04-26T18:12:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2024-04-26T18:12:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2024-04-26T18:12:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.444 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.445 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.445 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.445 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.445 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.445 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.445 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.445 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.445 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.445 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2024-04-26T18:12:44.445 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2024-04-26T18:12:44.445 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.445 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.445 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.446 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.446 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.446 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.446 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.446 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.446 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.446 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.446 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.446 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.446 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.446 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.446 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.446 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.447 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.447 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.447 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.447 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.447 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.447 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.447 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.447 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.447 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.447 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.447 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.447 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.447 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.447 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.448 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.448 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.448 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.448 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.448 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.448 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.448 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.448 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.448 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.448 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.448 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.448 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.448 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.451 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.451 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.451 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.451 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.451 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.451 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.451 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.451 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.451 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.451 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2024-04-26T18:12:44.451 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2024-04-26T18:12:44.451 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.451 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.452 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.452 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.452 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.452 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.452 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.452 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.452 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.452 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.452 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.452 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2024-04-26T18:12:44.452 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2024-04-26T18:12:44.452 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.453 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.453 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.453 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.453 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.453 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.453 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.453 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.453 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.453 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.453 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.453 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2024-04-26T18:12:44.453 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2024-04-26T18:12:44.453 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T18:12:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2024-04-26T18:12:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.456 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.456 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.456 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.456 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.456 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.456 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.456 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.456 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.456 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.456 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.456 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.456 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.456 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.457 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.457 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.457 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.457 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.457 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.457 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.457 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.457 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.457 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.457 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.457 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.457 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.457 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.458 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.458 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.458 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.458 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.458 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.458 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.458 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.458 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.458 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.458 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.458 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.458 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.458 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.458 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.459 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.459 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.459 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.459 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.459 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.459 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.459 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.459 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.459 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.459 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.459 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.459 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.459 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.460 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.460 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.460 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.460 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.460 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.460 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.460 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-04-26T18:12:44.460 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.460 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.460 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.460 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.460 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.461 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.461 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.461 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.461 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.461 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.461 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.461 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.461 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.461 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.461 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.461 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.461 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.461 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.461 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.462 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.462 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.462 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.462 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.462 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.462 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.462 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.462 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.462 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.462 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.462 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.462 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.462 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.463 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.464 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.464 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.464 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.464 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.464 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.464 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.464 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.464 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.464 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.464 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.464 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.464 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.465 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.465 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.465 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.465 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.465 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.465 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.465 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.465 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.465 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.465 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.465 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.465 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2024-04-26T18:12:44.465 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2024-04-26T18:12:44.466 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.466 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.466 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.466 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2024-04-26T18:12:44.466 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.466 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.466 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.466 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T18:12:44.466 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.466 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.466 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.466 INFO:teuthology.orchestra.run.smithi031.stdout: "periodic_async_work": { 2024-04-26T18:12:44.466 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "periodic_async_work", 2024-04-26T18:12:44.467 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.467 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.467 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.467 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.467 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.467 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.467 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.467 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Periodically check for async work", 2024-04-26T18:12:44.467 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.467 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.467 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.467 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2024-04-26T18:12:44.467 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2024-04-26T18:12:44.468 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.468 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.468 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.468 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:44.468 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.468 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.468 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.468 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T18:12:44.468 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.468 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.468 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.468 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_no_wait": { 2024-04-26T18:12:44.468 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T18:12:44.469 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.469 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.469 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.469 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.469 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.469 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.469 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.469 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T18:12:44.469 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.469 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.469 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.469 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.469 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2024-04-26T18:12:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2024-04-26T18:12:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2024-04-26T18:12:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2024-04-26T18:12:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.471 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.471 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.471 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.471 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.471 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2024-04-26T18:12:44.471 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2024-04-26T18:12:44.471 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.471 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.471 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.471 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.471 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.471 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.471 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.472 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.472 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.472 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.472 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.472 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:44.472 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:44.472 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.472 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.472 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.472 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:44.472 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.472 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.472 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.473 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.473 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.473 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.473 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.473 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.473 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.473 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.473 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.473 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.473 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.473 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.473 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.473 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.474 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.474 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.474 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.474 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.474 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.474 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.474 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.474 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.474 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.474 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.474 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.474 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.475 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.476 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.476 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.476 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.476 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.476 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.476 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.476 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.476 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.476 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.476 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.476 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.476 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.476 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.476 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.477 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.478 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.478 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.478 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.478 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2024-04-26T18:12:44.478 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2024-04-26T18:12:44.478 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.478 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.478 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.478 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.478 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.478 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.479 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.479 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.479 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.479 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.479 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.479 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2024-04-26T18:12:44.479 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2024-04-26T18:12:44.479 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.479 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.479 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.479 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2024-04-26T18:12:44.479 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.479 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2024-04-26T18:12:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2024-04-26T18:12:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T18:12:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-26T18:12:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "gid": 24869, 2024-04-26T18:12:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "y", 2024-04-26T18:12:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_features": 4540701547738038271, 2024-04-26T18:12:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2024-04-26T18:12:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2024-04-26T18:12:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T18:12:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.486 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.486 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.486 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.486 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.486 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.486 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.486 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.486 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.486 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.486 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.486 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.486 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.486 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.487 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.487 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.487 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.487 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.487 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.487 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.487 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.487 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.487 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.487 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.487 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.487 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.487 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.488 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2024-04-26T18:12:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2024-04-26T18:12:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2024-04-26T18:12:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2024-04-26T18:12:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.490 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.490 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.490 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2024-04-26T18:12:44.490 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.490 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.490 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.490 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2024-04-26T18:12:44.490 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2024-04-26T18:12:44.490 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.490 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.490 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.490 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.490 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.491 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.491 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.491 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2024-04-26T18:12:44.491 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.491 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.491 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.491 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2024-04-26T18:12:44.491 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2024-04-26T18:12:44.491 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.491 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.491 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.491 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.491 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.492 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.492 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.492 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2024-04-26T18:12:44.492 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.492 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.492 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.492 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2024-04-26T18:12:44.492 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2024-04-26T18:12:44.492 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.492 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.492 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.492 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2024-04-26T18:12:44.493 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.493 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.493 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.493 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2024-04-26T18:12:44.493 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.493 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.493 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.493 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2024-04-26T18:12:44.493 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2024-04-26T18:12:44.493 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.493 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.493 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.493 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.494 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.494 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.494 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.494 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2024-04-26T18:12:44.494 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.494 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.494 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.494 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2024-04-26T18:12:44.494 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2024-04-26T18:12:44.494 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.494 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.495 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.495 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.495 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.495 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.495 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.495 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T18:12:44.495 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.495 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.495 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.496 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2024-04-26T18:12:44.496 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2024-04-26T18:12:44.496 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.496 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.496 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.497 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.497 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.497 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.497 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.497 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2024-04-26T18:12:44.498 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.498 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.498 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.498 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.498 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.498 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2024-04-26T18:12:44.498 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.499 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.499 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.499 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2024-04-26T18:12:44.499 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2024-04-26T18:12:44.499 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.499 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.499 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.499 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.499 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.499 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.499 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.499 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T18:12:44.499 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.499 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.500 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.500 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2024-04-26T18:12:44.500 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2024-04-26T18:12:44.500 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.500 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.500 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.500 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2024-04-26T18:12:44.500 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.500 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.500 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.500 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T18:12:44.500 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T18:12:44.500 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.501 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.501 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2024-04-26T18:12:44.501 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2024-04-26T18:12:44.501 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:44.501 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.501 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.501 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:44.501 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-26T18:12:44.501 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-26T18:12:44.501 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.501 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T18:12:44.501 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T18:12:44.501 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.502 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.502 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2024-04-26T18:12:44.502 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2024-04-26T18:12:44.502 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:44.502 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.502 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.502 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2024-04-26T18:12:44.502 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-26T18:12:44.502 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2024-04-26T18:12:44.502 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.502 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T18:12:44.502 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.503 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.503 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.503 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2024-04-26T18:12:44.503 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2024-04-26T18:12:44.503 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.503 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.503 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.503 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T18:12:44.503 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.503 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.503 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.503 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T18:12:44.503 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T18:12:44.504 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.505 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.505 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2024-04-26T18:12:44.505 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2024-04-26T18:12:44.505 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.505 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.505 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.505 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2024-04-26T18:12:44.505 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.505 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.505 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.505 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T18:12:44.505 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T18:12:44.505 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.506 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.506 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2024-04-26T18:12:44.506 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2024-04-26T18:12:44.506 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:44.506 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.506 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.506 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:44.506 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-26T18:12:44.506 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-26T18:12:44.506 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.506 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T18:12:44.507 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T18:12:44.507 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.507 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.507 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.507 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.507 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.507 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.507 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.507 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.507 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.507 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.507 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.509 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.509 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.509 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.509 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.509 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.509 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.509 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.509 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.509 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.509 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.509 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.509 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.509 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.510 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.510 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.510 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.510 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.510 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.510 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.510 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.510 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.510 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.510 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.510 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.510 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.510 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.510 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.511 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.511 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.511 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.511 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.511 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.511 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.511 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.511 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.511 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.511 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.511 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.511 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.511 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.512 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.512 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.512 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.512 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.512 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2024-04-26T18:12:44.512 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2024-04-26T18:12:44.512 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.512 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.512 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.512 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:44.512 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.512 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.513 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.513 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T18:12:44.513 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.513 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.513 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.513 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-26T18:12:44.513 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-26T18:12:44.513 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.513 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.513 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.513 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2024-04-26T18:12:44.513 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.513 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.514 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.514 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2024-04-26T18:12:44.514 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2024-04-26T18:12:44.514 INFO:teuthology.orchestra.run.smithi031.stdout: "read", 2024-04-26T18:12:44.514 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap", 2024-04-26T18:12:44.514 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap-read" 2024-04-26T18:12:44.514 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.514 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2024-04-26T18:12:44.514 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.514 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.514 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.514 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2024-04-26T18:12:44.514 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2024-04-26T18:12:44.515 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.515 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.515 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.515 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.515 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.515 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.515 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.515 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T18:12:44.515 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.515 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.515 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.515 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:44.516 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:44.516 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.516 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.516 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.516 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:44.516 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.516 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.516 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.516 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T18:12:44.516 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.516 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.516 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.516 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2024-04-26T18:12:44.517 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2024-04-26T18:12:44.517 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.517 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.517 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.517 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-26T18:12:44.517 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-26T18:12:44.517 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.517 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.517 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T18:12:44.517 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T18:12:44.517 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.517 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.518 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2024-04-26T18:12:44.518 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2024-04-26T18:12:44.518 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:44.518 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.518 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.518 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.518 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.518 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.518 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.518 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T18:12:44.518 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.518 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.518 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.519 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.519 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.519 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.519 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2024-04-26T18:12:44.519 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.519 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.519 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.519 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2024-04-26T18:12:44.519 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2024-04-26T18:12:44.519 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.519 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.519 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.519 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-26T18:12:44.520 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.520 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.520 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.520 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T18:12:44.520 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.520 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.520 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.520 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2024-04-26T18:12:44.520 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2024-04-26T18:12:44.520 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.520 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.520 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.521 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2024-04-26T18:12:44.521 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.521 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.521 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.521 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T18:12:44.521 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.521 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.521 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.521 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2024-04-26T18:12:44.521 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2024-04-26T18:12:44.521 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.521 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.521 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2024-04-26T18:12:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T18:12:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2024-04-26T18:12:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2024-04-26T18:12:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.523 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.523 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.523 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.523 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.523 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T18:12:44.523 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T18:12:44.523 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.523 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.523 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2024-04-26T18:12:44.523 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2024-04-26T18:12:44.523 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.523 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.523 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.524 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:44.524 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.524 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.524 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.524 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T18:12:44.524 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.524 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.524 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.524 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2024-04-26T18:12:44.524 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T18:12:44.524 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.524 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.525 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.525 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2024-04-26T18:12:44.525 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.525 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.525 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.525 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T18:12:44.525 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.525 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.525 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.525 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_log_destination": { 2024-04-26T18:12:44.525 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm_log_destination", 2024-04-26T18:12:44.525 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.526 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.526 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.526 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.526 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.526 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.526 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.526 INFO:teuthology.orchestra.run.smithi031.stdout: "file", 2024-04-26T18:12:44.526 INFO:teuthology.orchestra.run.smithi031.stdout: "file,syslog", 2024-04-26T18:12:44.526 INFO:teuthology.orchestra.run.smithi031.stdout: "syslog" 2024-04-26T18:12:44.526 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.526 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T18:12:44.526 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.526 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.526 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.527 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2024-04-26T18:12:44.527 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2024-04-26T18:12:44.527 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.527 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.527 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.527 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.527 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.527 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.527 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.527 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T18:12:44.527 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.527 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.527 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.528 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2024-04-26T18:12:44.528 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2024-04-26T18:12:44.528 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.528 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.528 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.528 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.528 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.528 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.528 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.528 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T18:12:44.528 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.528 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2024-04-26T18:12:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2024-04-26T18:12:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T18:12:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-26T18:12:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.531 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2024-04-26T18:12:44.531 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2024-04-26T18:12:44.531 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.531 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.531 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.531 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T18:12:44.531 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.531 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.531 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.531 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2024-04-26T18:12:44.531 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.531 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.531 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.532 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2024-04-26T18:12:44.532 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2024-04-26T18:12:44.532 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.532 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.532 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.532 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T18:12:44.532 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.532 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.532 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.532 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2024-04-26T18:12:44.532 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.533 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.533 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.533 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2024-04-26T18:12:44.533 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2024-04-26T18:12:44.533 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.533 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.533 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.533 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T18:12:44.533 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.533 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.533 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.533 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-26T18:12:44.533 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.534 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.534 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.534 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2024-04-26T18:12:44.534 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2024-04-26T18:12:44.534 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.534 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.534 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.534 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T18:12:44.534 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.534 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.534 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.534 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2024-04-26T18:12:44.535 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.535 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.535 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.535 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2024-04-26T18:12:44.535 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2024-04-26T18:12:44.535 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.535 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.535 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.535 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T18:12:44.535 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.535 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.535 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.536 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2024-04-26T18:12:44.536 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.536 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.536 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.536 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2024-04-26T18:12:44.536 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2024-04-26T18:12:44.536 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.536 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.536 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.536 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T18:12:44.536 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.536 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2024-04-26T18:12:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2024-04-26T18:12:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2024-04-26T18:12:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T18:12:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.538 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.538 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2024-04-26T18:12:44.538 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.538 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.538 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.538 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2024-04-26T18:12:44.538 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2024-04-26T18:12:44.538 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.538 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.538 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.538 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T18:12:44.538 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.539 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.539 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.539 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2024-04-26T18:12:44.539 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.539 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.539 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.539 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2024-04-26T18:12:44.539 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2024-04-26T18:12:44.539 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.539 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.539 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.539 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T18:12:44.540 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.540 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.540 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.540 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2024-04-26T18:12:44.540 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.540 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.540 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.540 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2024-04-26T18:12:44.540 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2024-04-26T18:12:44.540 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.540 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.540 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.541 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T18:12:44.541 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.541 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.541 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.541 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-26T18:12:44.541 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.541 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.541 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.541 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nvmeof": { 2024-04-26T18:12:44.541 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nvmeof", 2024-04-26T18:12:44.541 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.541 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.541 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.542 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T18:12:44.542 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.542 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.542 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.542 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nvme-of container image", 2024-04-26T18:12:44.542 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.542 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.542 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.542 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2024-04-26T18:12:44.542 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2024-04-26T18:12:44.542 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.542 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.543 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.543 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T18:12:44.543 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.543 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.543 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.543 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-26T18:12:44.543 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.543 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.543 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.543 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2024-04-26T18:12:44.543 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2024-04-26T18:12:44.543 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.543 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.544 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.544 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T18:12:44.544 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.544 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.544 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.544 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2024-04-26T18:12:44.544 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.544 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.544 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.544 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_samba": { 2024-04-26T18:12:44.544 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_samba", 2024-04-26T18:12:44.544 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.544 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.545 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.545 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T18:12:44.545 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.545 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.545 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.545 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Samba/SMB container image", 2024-04-26T18:12:44.545 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.545 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.545 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.545 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2024-04-26T18:12:44.545 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2024-04-26T18:12:44.545 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.546 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.546 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.546 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T18:12:44.546 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.546 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.546 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.546 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2024-04-26T18:12:44.546 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.546 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.546 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.546 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2024-04-26T18:12:44.546 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2024-04-26T18:12:44.547 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.547 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.547 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.547 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.547 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.547 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.547 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.547 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T18:12:44.547 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.547 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.547 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.547 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2024-04-26T18:12:44.548 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2024-04-26T18:12:44.548 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.548 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.548 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.548 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:44.548 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.548 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.548 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.548 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T18:12:44.548 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.548 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.548 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.549 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2024-04-26T18:12:44.549 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T18:12:44.549 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.549 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.549 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.549 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2024-04-26T18:12:44.549 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.549 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.549 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.549 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T18:12:44.549 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.549 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.549 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.550 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2024-04-26T18:12:44.550 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2024-04-26T18:12:44.550 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.550 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.550 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.550 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2024-04-26T18:12:44.550 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.550 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.550 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.550 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T18:12:44.550 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.550 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.551 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.551 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2024-04-26T18:12:44.551 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2024-04-26T18:12:44.551 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.551 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.551 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.551 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2024-04-26T18:12:44.551 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.551 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.551 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.551 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2024-04-26T18:12:44.551 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.552 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.552 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.552 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2024-04-26T18:12:44.552 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2024-04-26T18:12:44.552 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.552 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.552 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.552 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.552 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.552 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.552 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.552 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T18:12:44.553 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.553 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.553 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.553 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2024-04-26T18:12:44.553 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2024-04-26T18:12:44.553 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.553 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.553 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.553 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:44.553 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.553 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.553 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.553 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2024-04-26T18:12:44.554 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.554 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.554 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.554 INFO:teuthology.orchestra.run.smithi031.stdout: "grafana_dashboards_path": { 2024-04-26T18:12:44.554 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "grafana_dashboards_path", 2024-04-26T18:12:44.554 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.554 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.554 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.554 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T18:12:44.554 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.554 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.554 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.555 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T18:12:44.555 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.555 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.555 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.555 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2024-04-26T18:12:44.555 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2024-04-26T18:12:44.555 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.555 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.555 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.555 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:44.555 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.555 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.556 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.556 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2024-04-26T18:12:44.556 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.556 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.556 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.556 INFO:teuthology.orchestra.run.smithi031.stdout: "hw_monitoring": { 2024-04-26T18:12:44.556 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hw_monitoring", 2024-04-26T18:12:44.556 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.556 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.556 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.556 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.556 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.557 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.557 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.557 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T18:12:44.557 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.557 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.557 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.557 INFO:teuthology.orchestra.run.smithi031.stdout: "inventory_list_all": { 2024-04-26T18:12:44.557 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "inventory_list_all", 2024-04-26T18:12:44.557 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.557 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.557 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.557 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.558 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.558 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.558 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.558 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T18:12:44.558 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.558 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.558 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.558 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.558 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.558 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.558 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.559 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.559 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.559 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.559 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.559 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.559 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.559 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.559 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.559 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.559 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.559 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.559 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.559 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.560 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.560 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.560 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.560 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2024-04-26T18:12:44.560 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2024-04-26T18:12:44.560 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.560 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.560 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.560 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.560 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.560 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.560 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.561 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T18:12:44.561 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.561 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.561 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.561 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.561 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.561 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.561 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.561 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.561 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.561 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.561 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.562 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.562 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T18:12:44.562 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.562 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.562 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.562 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.562 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.562 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.562 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.562 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.562 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.563 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.563 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.563 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.563 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.563 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.563 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.563 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.563 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.563 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.563 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.563 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.563 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.563 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.564 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.564 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.564 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.564 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.564 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.564 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.564 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.564 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.564 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.564 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.564 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.564 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.565 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.565 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.565 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T18:12:44.565 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T18:12:44.565 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.565 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.565 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.565 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.565 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.565 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.565 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.565 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T18:12:44.565 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.566 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.566 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.566 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T18:12:44.566 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T18:12:44.566 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.566 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.566 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.566 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2024-04-26T18:12:44.566 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.566 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.566 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.566 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T18:12:44.567 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.567 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.567 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.567 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2024-04-26T18:12:44.567 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2024-04-26T18:12:44.567 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.567 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.567 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.567 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.567 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.567 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.567 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2024-04-26T18:12:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2024-04-26T18:12:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2024-04-26T18:12:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T18:12:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2024-04-26T18:12:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2024-04-26T18:12:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2024-04-26T18:12:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-26T18:12:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-26T18:12:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2024-04-26T18:12:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2024-04-26T18:12:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2024-04-26T18:12:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T18:12:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "oob_default_addr": { 2024-04-26T18:12:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "oob_default_addr", 2024-04-26T18:12:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "169.254.1.1", 2024-04-26T18:12:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T18:12:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2024-04-26T18:12:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2024-04-26T18:12:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T18:12:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2024-04-26T18:12:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2024-04-26T18:12:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.575 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.575 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.575 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.575 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.575 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.575 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T18:12:44.575 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.575 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.575 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.575 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2024-04-26T18:12:44.575 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2024-04-26T18:12:44.575 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.575 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.576 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.576 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.576 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.576 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.576 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.576 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T18:12:44.576 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.576 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.576 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.576 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2024-04-26T18:12:44.576 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2024-04-26T18:12:44.576 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.577 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.577 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.577 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.577 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.577 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.577 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.577 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T18:12:44.577 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.577 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.577 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.577 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2024-04-26T18:12:44.577 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2024-04-26T18:12:44.577 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.578 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.578 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.578 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.578 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.578 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.578 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.578 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T18:12:44.578 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.578 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.578 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.578 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2024-04-26T18:12:44.578 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2024-04-26T18:12:44.578 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.579 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.579 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.579 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2024-04-26T18:12:44.579 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.579 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.579 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.579 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2024-04-26T18:12:44.579 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.579 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.579 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.579 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2024-04-26T18:12:44.579 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2024-04-26T18:12:44.580 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.580 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.580 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.580 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.580 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.580 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.580 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.580 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T18:12:44.580 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.580 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.580 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.580 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2024-04-26T18:12:44.581 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2024-04-26T18:12:44.581 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.581 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.581 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.581 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.581 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.581 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.581 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.581 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T18:12:44.581 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.581 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.581 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.581 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2024-04-26T18:12:44.582 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2024-04-26T18:12:44.582 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.582 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.582 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.582 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.582 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.582 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.582 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.582 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T18:12:44.582 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.582 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.582 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.583 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2024-04-26T18:12:44.583 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2024-04-26T18:12:44.583 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.583 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.583 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.583 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.583 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.583 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.583 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.583 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T18:12:44.583 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.583 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.583 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.584 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2024-04-26T18:12:44.584 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2024-04-26T18:12:44.584 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.584 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.584 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.584 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.584 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.584 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.584 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.584 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T18:12:44.584 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.584 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.585 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.585 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2024-04-26T18:12:44.585 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2024-04-26T18:12:44.585 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.585 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.585 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.585 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.585 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.585 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.585 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.585 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T18:12:44.585 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.586 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.586 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.586 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.586 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.586 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2024-04-26T18:12:44.586 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.586 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.586 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.586 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.586 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.586 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.587 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.587 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.587 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.587 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.587 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.587 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.587 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.587 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.587 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.587 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.587 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.587 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.587 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.587 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.588 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.588 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.588 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.588 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.588 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.588 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.588 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.588 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.588 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.588 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.588 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.589 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.589 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.589 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.589 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.589 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.589 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.589 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.589 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.589 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.589 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.589 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.589 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.589 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.590 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.590 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.590 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.590 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.590 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.590 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.590 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.590 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.590 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.590 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.590 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.590 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.591 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.591 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.591 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.591 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.591 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.591 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.591 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.591 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.591 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.591 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.591 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.591 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.591 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2024-04-26T18:12:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2024-04-26T18:12:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2024-04-26T18:12:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T18:12:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2024-04-26T18:12:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2024-04-26T18:12:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2024-04-26T18:12:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T18:12:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2024-04-26T18:12:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T18:12:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T18:12:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T18:12:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T18:12:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T18:12:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T18:12:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2024-04-26T18:12:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T18:12:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T18:12:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T18:12:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T18:12:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T18:12:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T18:12:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T18:12:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T18:12:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T18:12:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.603 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.603 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.603 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T18:12:44.603 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T18:12:44.603 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.603 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.603 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.603 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.603 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.603 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.603 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.603 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.603 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.604 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.604 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.604 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T18:12:44.604 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T18:12:44.604 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.604 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.604 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.604 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.604 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.604 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.604 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.604 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.605 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.605 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.605 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.605 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T18:12:44.605 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T18:12:44.605 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.605 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.605 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.605 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.605 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.605 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.605 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T18:12:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T18:12:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.607 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.607 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.607 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.607 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.607 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.607 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T18:12:44.607 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T18:12:44.607 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.607 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.607 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.607 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.607 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.607 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.608 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.608 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.608 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.608 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.608 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.608 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T18:12:44.608 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T18:12:44.608 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.608 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.608 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.608 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.608 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.609 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.609 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.609 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.609 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.609 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.609 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.609 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T18:12:44.609 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T18:12:44.609 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.609 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.609 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.609 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-26T18:12:44.609 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.610 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.610 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.610 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.610 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.610 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.610 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.610 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T18:12:44.610 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T18:12:44.610 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.610 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.610 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.610 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.610 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2024-04-26T18:12:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2024-04-26T18:12:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.612 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.613 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.613 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.613 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.613 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.613 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.613 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.613 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T18:12:44.613 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T18:12:44.614 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.614 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.614 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.614 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-26T18:12:44.614 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.614 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.614 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.614 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.614 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.614 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.614 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.614 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T18:12:44.615 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T18:12:44.615 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.615 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.615 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.615 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.615 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.615 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.615 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.615 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.615 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.615 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.615 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.615 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T18:12:44.616 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T18:12:44.616 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.616 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.616 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.616 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.616 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.616 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.616 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.616 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.616 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.616 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.616 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.617 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T18:12:44.617 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T18:12:44.617 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.617 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.617 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.617 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.617 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.617 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.617 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.617 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.617 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.617 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.617 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.618 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T18:12:44.618 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T18:12:44.618 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.618 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.618 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.618 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.618 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.618 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.618 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.618 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.618 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.618 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.619 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.619 INFO:teuthology.orchestra.run.smithi031.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T18:12:44.619 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T18:12:44.619 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.619 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.619 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.619 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "[]", 2024-04-26T18:12:44.619 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.619 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.619 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.619 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.619 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.620 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.620 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.620 INFO:teuthology.orchestra.run.smithi031.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T18:12:44.620 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T18:12:44.620 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.620 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.620 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.620 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "{}", 2024-04-26T18:12:44.620 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.620 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.620 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.620 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.620 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.621 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.621 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.621 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T18:12:44.621 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T18:12:44.621 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.621 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.621 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.621 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.621 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.621 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.621 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.621 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.622 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.622 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.622 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.622 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T18:12:44.622 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T18:12:44.622 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.622 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.622 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.622 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.622 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.622 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.622 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.623 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.623 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.623 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.623 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.623 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T18:12:44.623 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T18:12:44.623 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.623 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.623 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.623 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.623 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.623 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.623 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.624 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.624 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.624 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.624 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.624 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T18:12:44.624 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T18:12:44.624 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.624 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.624 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.624 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.624 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.624 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.624 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.625 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.625 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.625 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.625 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.625 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T18:12:44.625 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T18:12:44.625 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.625 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.625 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.625 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.625 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.625 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.625 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.626 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.626 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.626 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.626 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.626 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T18:12:44.626 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T18:12:44.626 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.626 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.626 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.626 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.626 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.626 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.627 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.627 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.627 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.627 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.627 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.627 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T18:12:44.627 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T18:12:44.627 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.627 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.627 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.627 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.627 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.627 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.628 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.628 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.628 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.628 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.628 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.628 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T18:12:44.628 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T18:12:44.628 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.628 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.628 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.628 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.628 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.628 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.629 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.629 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.629 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.629 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.629 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.629 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T18:12:44.629 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T18:12:44.629 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.629 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.629 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.629 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.629 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.629 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.630 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.630 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.630 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.630 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.630 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.630 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T18:12:44.630 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T18:12:44.630 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.630 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.630 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.630 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.630 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.631 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.631 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.631 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.631 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.631 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.631 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.631 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T18:12:44.631 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T18:12:44.631 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.631 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.631 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.631 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T18:12:44.631 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.632 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.632 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.632 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.632 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.632 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.632 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.632 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T18:12:44.632 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T18:12:44.632 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.632 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.632 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.632 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.633 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.633 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.633 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.633 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.633 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.633 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.633 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.633 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T18:12:44.633 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T18:12:44.633 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.633 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.633 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.634 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2024-04-26T18:12:44.634 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.634 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.634 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.634 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.634 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.634 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.634 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.634 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T18:12:44.634 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T18:12:44.634 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.634 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.634 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2024-04-26T18:12:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T18:12:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T18:12:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.635 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.636 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.636 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.636 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.636 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.636 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.636 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.636 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.636 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.636 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.636 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T18:12:44.636 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T18:12:44.636 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.637 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.637 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.637 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-26T18:12:44.637 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.637 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.637 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.637 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.637 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.637 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.637 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.637 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T18:12:44.637 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T18:12:44.637 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T18:12:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T18:12:44.639 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.639 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.639 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.639 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.639 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.639 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.639 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.639 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.639 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.639 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.639 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.639 INFO:teuthology.orchestra.run.smithi031.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T18:12:44.639 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T18:12:44.640 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.640 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.640 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.640 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.640 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.640 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.640 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.640 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.640 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.640 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.640 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.641 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T18:12:44.641 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T18:12:44.641 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.641 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.641 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.641 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:44.641 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.641 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.641 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.641 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.641 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.641 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.641 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.642 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T18:12:44.642 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T18:12:44.642 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.642 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.642 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.642 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.642 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.642 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.642 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.642 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.642 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.643 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.643 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.643 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T18:12:44.643 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T18:12:44.643 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.643 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.643 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.643 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-26T18:12:44.643 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.643 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.643 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.643 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.644 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.644 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.644 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.644 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2024-04-26T18:12:44.644 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2024-04-26T18:12:44.644 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.644 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.644 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.644 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.644 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.644 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.644 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.644 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.645 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.645 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.645 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.645 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2024-04-26T18:12:44.645 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2024-04-26T18:12:44.645 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.645 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.645 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.645 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.645 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.645 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.645 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.646 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.646 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.646 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.646 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.646 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2024-04-26T18:12:44.646 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2024-04-26T18:12:44.646 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.646 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.646 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.646 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.646 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.646 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.647 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.647 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2024-04-26T18:12:44.647 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.647 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.647 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.647 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2024-04-26T18:12:44.647 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2024-04-26T18:12:44.647 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.647 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.647 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.647 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2024-04-26T18:12:44.647 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.648 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.648 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.648 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.648 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.648 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.648 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.648 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-26T18:12:44.648 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-26T18:12:44.648 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.648 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.648 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.648 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.648 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.649 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.649 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.649 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.649 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.649 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.649 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.649 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.649 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.649 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.649 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.649 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.649 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.649 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.650 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.650 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.650 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.650 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.650 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.650 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.650 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.650 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.650 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.650 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.650 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.650 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.650 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.651 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.651 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.651 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.651 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.651 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.651 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.651 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.651 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.651 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.651 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.651 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.651 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.652 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.653 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.653 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.653 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.653 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.653 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.653 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.653 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.653 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.653 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.653 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.653 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.653 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.653 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.654 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.654 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.654 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.654 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.654 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.654 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.654 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.654 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.654 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.654 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2024-04-26T18:12:44.654 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2024-04-26T18:12:44.654 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.654 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.655 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.655 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.655 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.655 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.655 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.655 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2024-04-26T18:12:44.655 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.655 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.655 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.655 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2024-04-26T18:12:44.655 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T18:12:44.655 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.656 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.656 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.656 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.656 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.656 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.656 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.656 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.656 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.656 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.656 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.656 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-26T18:12:44.656 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-26T18:12:44.657 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.657 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.657 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.657 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-26T18:12:44.657 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.657 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.657 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.657 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.657 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.657 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.657 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.657 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-26T18:12:44.657 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-26T18:12:44.658 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.658 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.658 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.658 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2024-04-26T18:12:44.658 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.658 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.658 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.658 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.658 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.658 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.658 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.658 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-26T18:12:44.659 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-26T18:12:44.659 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.659 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.659 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.659 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.659 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.659 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.659 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.659 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.659 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.659 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.659 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.659 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2024-04-26T18:12:44.660 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2024-04-26T18:12:44.660 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.660 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.660 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.660 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2024-04-26T18:12:44.660 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.660 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.660 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.660 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.660 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.660 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.660 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.660 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-26T18:12:44.661 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-26T18:12:44.661 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.661 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.661 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.661 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2024-04-26T18:12:44.661 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.661 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.661 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.661 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.661 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2024-04-26T18:12:44.661 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.661 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.661 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.662 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.662 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.662 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-26T18:12:44.662 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-26T18:12:44.662 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.662 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.662 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.662 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-26T18:12:44.662 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-26T18:12:44.662 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-26T18:12:44.662 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.662 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.662 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.663 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.663 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.663 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2024-04-26T18:12:44.663 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2024-04-26T18:12:44.663 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.663 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.663 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.663 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.663 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.663 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.663 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.663 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.664 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.664 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.664 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.664 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-04-26T18:12:44.664 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.664 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.664 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.664 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2024-04-26T18:12:44.664 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2024-04-26T18:12:44.665 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.665 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.665 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.665 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.665 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.665 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.665 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.665 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2024-04-26T18:12:44.665 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.665 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.665 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.665 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.666 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.666 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.666 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.666 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.666 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.666 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.666 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.666 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.666 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.666 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.666 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.666 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.666 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.667 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.667 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.667 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.667 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.667 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.667 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.667 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.667 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.667 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.667 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.667 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.667 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.667 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2024-04-26T18:12:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2024-04-26T18:12:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2024-04-26T18:12:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T18:12:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2024-04-26T18:12:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2024-04-26T18:12:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2024-04-26T18:12:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T18:12:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2024-04-26T18:12:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2024-04-26T18:12:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2024-04-26T18:12:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2024-04-26T18:12:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.674 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.674 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.674 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2024-04-26T18:12:44.674 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2024-04-26T18:12:44.674 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.674 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.674 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.674 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-26T18:12:44.674 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.674 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.674 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.674 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T18:12:44.674 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.675 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.675 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.675 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2024-04-26T18:12:44.675 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2024-04-26T18:12:44.675 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.675 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.675 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.675 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.675 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.675 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.675 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.675 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T18:12:44.675 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.676 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.676 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.676 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:44.676 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:44.676 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.676 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.676 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.676 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:44.676 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.676 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.676 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.676 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T18:12:44.676 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.677 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.678 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.678 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.678 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.678 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2024-04-26T18:12:44.678 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.678 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.678 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.678 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.678 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.678 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.678 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.679 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.679 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.679 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.679 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.679 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.679 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.679 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.679 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.679 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.679 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.679 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.679 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.679 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.680 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.680 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.680 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.680 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.680 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.680 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.680 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.680 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.680 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.680 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.680 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.680 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.680 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.681 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.681 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.681 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.681 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.681 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.681 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.681 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.681 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.681 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.681 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.681 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.681 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.682 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.682 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.682 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.682 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.682 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.682 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.682 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.682 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.682 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.682 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.682 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.682 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.683 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.683 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.683 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.683 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.683 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.683 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.683 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.683 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.683 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.683 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.683 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.683 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.684 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2024-04-26T18:12:44.684 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2024-04-26T18:12:44.684 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.684 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.684 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.684 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-26T18:12:44.684 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.684 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.684 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.684 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.684 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.684 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2024-04-26T18:12:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2024-04-26T18:12:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2024-04-26T18:12:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2024-04-26T18:12:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2024-04-26T18:12:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2024-04-26T18:12:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2024-04-26T18:12:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2024-04-26T18:12:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2024-04-26T18:12:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T18:12:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2024-04-26T18:12:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2024-04-26T18:12:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2024-04-26T18:12:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T18:12:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2024-04-26T18:12:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2024-04-26T18:12:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2024-04-26T18:12:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2024-04-26T18:12:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2024-04-26T18:12:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T18:12:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.691 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.692 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.693 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.693 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.693 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.693 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.693 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.693 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.693 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.693 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.693 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.693 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.693 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.693 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.694 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.694 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.694 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.694 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.694 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.694 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.694 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.694 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.694 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.694 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.694 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.694 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.694 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.695 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.695 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.695 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.695 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.695 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.695 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.695 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.695 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.695 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.695 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.695 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.695 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.695 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.695 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.696 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.696 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.696 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.696 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.696 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.696 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.696 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.696 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.696 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.696 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.696 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.696 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.697 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.697 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.697 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.697 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2024-04-26T18:12:44.697 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2024-04-26T18:12:44.697 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.697 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.697 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.697 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.697 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.697 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.697 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.698 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2024-04-26T18:12:44.698 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.698 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.698 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.698 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2024-04-26T18:12:44.698 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2024-04-26T18:12:44.698 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.698 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.698 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.698 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2024-04-26T18:12:44.698 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.698 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.698 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.699 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2024-04-26T18:12:44.699 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.699 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.699 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.699 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-26T18:12:44.699 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-26T18:12:44.699 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.699 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.699 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.699 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2024-04-26T18:12:44.699 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.699 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.699 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.700 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T18:12:44.700 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.700 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.700 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.700 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2024-04-26T18:12:44.700 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2024-04-26T18:12:44.700 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.700 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.700 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.700 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-26T18:12:44.700 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-26T18:12:44.700 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2024-04-26T18:12:44.700 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.701 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T18:12:44.701 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.701 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.701 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.701 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2024-04-26T18:12:44.701 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2024-04-26T18:12:44.701 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.701 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.701 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.701 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.701 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.701 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.702 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.702 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2024-04-26T18:12:44.702 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.702 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.702 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.702 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2024-04-26T18:12:44.702 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2024-04-26T18:12:44.702 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.702 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.702 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.702 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2024-04-26T18:12:44.702 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.703 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.703 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.703 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T18:12:44.703 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.703 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.703 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.703 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.703 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.703 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2024-04-26T18:12:44.703 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.703 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.703 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.704 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.704 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.704 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.704 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.704 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.704 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.704 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.704 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.704 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.704 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.704 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.704 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.704 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.704 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.705 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.705 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.705 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.705 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.705 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.705 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.705 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.705 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.705 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.705 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.705 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.705 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.705 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.706 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.706 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.706 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.706 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.706 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.706 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.706 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.706 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.706 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.706 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.706 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.706 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.706 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.708 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.708 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.708 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.708 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.708 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.708 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.708 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.708 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.708 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.708 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.708 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.708 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.708 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.708 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.709 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.709 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2024-04-26T18:12:44.709 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.709 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.709 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.709 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.709 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.709 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.709 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.709 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.709 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.710 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.710 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.710 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.710 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.710 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.710 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.710 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.710 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.710 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.710 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.710 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.710 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.710 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.711 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.711 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.711 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.711 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.711 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.711 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.711 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.711 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.711 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.711 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.711 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.712 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.712 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.712 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.712 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.712 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.712 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.712 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.712 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.712 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.712 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.712 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.712 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.713 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.713 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.713 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.713 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.713 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.713 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.713 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.713 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.713 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.713 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.713 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.713 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.713 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.714 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.714 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.714 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.714 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.714 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.714 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.714 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.714 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.714 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.714 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.714 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.714 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.714 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.715 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2024-04-26T18:12:44.715 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.715 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.715 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.715 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2024-04-26T18:12:44.715 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2024-04-26T18:12:44.715 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.715 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.715 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.715 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2024-04-26T18:12:44.716 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.716 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.716 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.716 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T18:12:44.716 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.716 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.716 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.716 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2024-04-26T18:12:44.716 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2024-04-26T18:12:44.716 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.716 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.716 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.716 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.717 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2024-04-26T18:12:44.717 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.717 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.717 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T18:12:44.717 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.717 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.717 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.717 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.717 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.717 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.717 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.717 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.718 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.718 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.718 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.718 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.718 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.718 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.718 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.718 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.718 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.718 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.718 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.718 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.718 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.718 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.719 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.719 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.719 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.719 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.719 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.719 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.719 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.719 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.719 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.719 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.719 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.720 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.720 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.720 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.720 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.720 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.720 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.720 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.720 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.720 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.720 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.720 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.720 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.721 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.721 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.721 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.721 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.721 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.721 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.721 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.721 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.721 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.721 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.721 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.721 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.722 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.722 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.722 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.722 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.722 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.722 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.722 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.722 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.722 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.722 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.722 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.722 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.722 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.723 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.723 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2024-04-26T18:12:44.723 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.723 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.723 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.723 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2024-04-26T18:12:44.723 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2024-04-26T18:12:44.723 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.723 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.723 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.723 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2024-04-26T18:12:44.724 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.724 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.724 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.724 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2024-04-26T18:12:44.724 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T18:12:44.724 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.724 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.724 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.724 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.724 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.724 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.724 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.725 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.725 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.725 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.725 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.725 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.725 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.725 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.725 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.725 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.725 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.725 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.725 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.725 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.726 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.726 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.726 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.726 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.726 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.726 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.726 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.726 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.727 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.727 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.727 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.727 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.727 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.727 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.727 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.727 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.727 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.727 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.727 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.727 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.728 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.728 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.728 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.728 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.728 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.728 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.728 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.728 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.728 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.728 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.728 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.728 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.728 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.729 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.729 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.729 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.729 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.729 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.729 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.729 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.729 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.729 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.729 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.729 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.729 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.729 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.730 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.730 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.730 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2024-04-26T18:12:44.730 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2024-04-26T18:12:44.730 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.730 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.730 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.730 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.730 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.730 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.730 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.730 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2024-04-26T18:12:44.731 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T18:12:44.731 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.731 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.731 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2024-04-26T18:12:44.731 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2024-04-26T18:12:44.731 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.731 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.731 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.731 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-04-26T18:12:44.731 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.731 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.731 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.732 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2024-04-26T18:12:44.732 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.732 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.732 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.732 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2024-04-26T18:12:44.732 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2024-04-26T18:12:44.732 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.732 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.732 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.732 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2024-04-26T18:12:44.732 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.732 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.732 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.733 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T18:12:44.733 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.733 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.733 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.733 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2024-04-26T18:12:44.733 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2024-04-26T18:12:44.733 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.733 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.733 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.733 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.733 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.733 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.733 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.734 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2024-04-26T18:12:44.734 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.734 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.734 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.734 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2024-04-26T18:12:44.734 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2024-04-26T18:12:44.734 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.734 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.734 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.734 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2024-04-26T18:12:44.734 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.734 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.735 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.735 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T18:12:44.735 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T18:12:44.735 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.735 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.735 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.735 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.735 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2024-04-26T18:12:44.735 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.735 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.735 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.736 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.736 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.736 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.736 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.736 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.736 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.736 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.736 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.736 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.736 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.736 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.736 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.736 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.736 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.737 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.737 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.737 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.737 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.737 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.737 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.737 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.737 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.737 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.737 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.737 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.737 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.737 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.738 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.738 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.738 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.738 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.738 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.738 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.738 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.738 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.738 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.738 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.738 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.738 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.738 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.739 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.739 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.739 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.739 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.739 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.739 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.739 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.739 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.739 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.739 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.739 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.739 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.739 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.740 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.740 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.740 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.740 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.740 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.740 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.740 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.740 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.740 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.740 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.740 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.741 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.741 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.741 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.741 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.741 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2024-04-26T18:12:44.741 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.741 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.741 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.741 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.741 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.741 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.742 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.742 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.742 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.742 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.742 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.742 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.742 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.742 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.742 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.742 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.742 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.742 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.742 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.743 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.743 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.743 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.743 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.743 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.743 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.743 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.743 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.743 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.743 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.743 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.743 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.744 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.744 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.744 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.744 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.744 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.744 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.744 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.744 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.744 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.744 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.744 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.744 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.745 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.745 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.745 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.745 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.745 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.745 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.745 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.745 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.745 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.745 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.745 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.745 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.746 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.746 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.746 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.746 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.746 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.746 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.746 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.746 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.746 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.746 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.746 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.746 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.747 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.747 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.747 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.747 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.747 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2024-04-26T18:12:44.747 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.747 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.747 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.747 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.747 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.747 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.748 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.748 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.748 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.748 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.748 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.748 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.748 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.748 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.748 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.748 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.748 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.748 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.748 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.750 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.750 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.750 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.750 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.750 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.750 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.750 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.750 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.750 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.750 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.750 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.750 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.751 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.751 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.751 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.751 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.751 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.751 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.751 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.751 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.751 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.751 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.751 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.751 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.752 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.752 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.752 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.752 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.752 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.752 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.752 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.752 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.752 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.752 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.752 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.752 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.752 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.753 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.753 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.753 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-26T18:12:44.753 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.753 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.753 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.753 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2024-04-26T18:12:44.753 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2024-04-26T18:12:44.753 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.753 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.754 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.754 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.754 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.754 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.754 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.754 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T18:12:44.754 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.754 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.754 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.754 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.754 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.755 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.755 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.755 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.755 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.755 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.755 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.755 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.755 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.755 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.755 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.755 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.755 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.756 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.756 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.756 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.756 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.756 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.756 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.756 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.756 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.756 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.756 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.756 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.756 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.757 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.757 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.757 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.757 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.757 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.757 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.757 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.757 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.757 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.757 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.757 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.757 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.758 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.758 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.758 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.758 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.758 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.758 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.758 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.758 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.758 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.758 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.758 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.759 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.759 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.759 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.759 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.759 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.759 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.759 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.759 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.759 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.759 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.759 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.759 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.760 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.760 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.760 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.760 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.760 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.760 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2024-04-26T18:12:44.760 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-26T18:12:44.760 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.760 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.760 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.760 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.760 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.761 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.761 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.761 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-04-26T18:12:44.761 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2024-04-26T18:12:44.761 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2024-04-26T18:12:44.761 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.761 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2024-04-26T18:12:44.761 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.761 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.761 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.761 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.761 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.762 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2024-04-26T18:12:44.762 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.762 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.762 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.762 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.762 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.762 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.762 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.762 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.762 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.762 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.763 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.763 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.763 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.763 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.763 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.763 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.763 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.763 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.763 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.763 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.763 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.763 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.763 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.764 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.764 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.764 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.764 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.764 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.764 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.764 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.764 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.764 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.764 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.764 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.764 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.764 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.765 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.765 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.765 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.765 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.765 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.765 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.765 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.765 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.765 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.765 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.765 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.765 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.765 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.765 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.766 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.766 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.766 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.766 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.766 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.766 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.766 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.766 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.766 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.766 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.766 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.766 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.767 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.768 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.768 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.768 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.768 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.768 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.768 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.768 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.768 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.768 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.768 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.768 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.768 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.768 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.769 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.769 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.769 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.769 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.769 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.769 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.769 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.769 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.769 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.769 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.769 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.769 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.769 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.770 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.770 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.770 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.770 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.770 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.770 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.770 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.770 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.770 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.770 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.770 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.770 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.771 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.771 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.771 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.771 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.771 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.771 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.771 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.771 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.771 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.771 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.771 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.771 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.771 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.772 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.772 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.772 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.772 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.772 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.772 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.772 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.772 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.772 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.772 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.772 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.772 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.773 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.773 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.773 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.773 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2024-04-26T18:12:44.773 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.773 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.773 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.773 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.773 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.774 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.775 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.777 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.777 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.777 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.777 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.777 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.777 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.777 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.777 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.777 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.777 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.777 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.777 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.777 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.778 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.778 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.778 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.778 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.778 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.778 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.778 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.778 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.778 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.778 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.778 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.779 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.779 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.779 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.779 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:44.779 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:44.779 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.779 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.779 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.779 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:44.779 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.779 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.779 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.780 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.780 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.780 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.780 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.780 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2024-04-26T18:12:44.780 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2024-04-26T18:12:44.780 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.780 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.780 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.780 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-26T18:12:44.780 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2024-04-26T18:12:44.780 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.781 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.781 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2024-04-26T18:12:44.781 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T18:12:44.781 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.781 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.781 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.781 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.781 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2024-04-26T18:12:44.781 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.781 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.781 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.782 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2024-04-26T18:12:44.782 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2024-04-26T18:12:44.782 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.782 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.782 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.782 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.782 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.782 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.782 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.782 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T18:12:44.782 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.782 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.782 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.783 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-26T18:12:44.783 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-26T18:12:44.783 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.783 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.783 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.783 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.783 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.783 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.783 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.783 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.783 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.784 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.784 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.784 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.784 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.784 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.784 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.784 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.784 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.784 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.784 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.784 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.784 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.784 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.785 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.785 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.785 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.785 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.785 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.785 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.785 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.785 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.785 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.785 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.785 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.785 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.786 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.786 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.786 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.786 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.786 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.786 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.786 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.786 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.786 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.786 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.786 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.786 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.787 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.787 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.787 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.787 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.787 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.787 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.787 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.787 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.787 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.787 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.787 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.787 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.788 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.788 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.788 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.788 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.788 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.788 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.788 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.788 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.788 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.788 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.788 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.788 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2024-04-26T18:12:44.789 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.790 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.790 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.790 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2024-04-26T18:12:44.790 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.790 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.790 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.790 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:44.790 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:44.790 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.790 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.790 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.790 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-26T18:12:44.791 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.791 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.791 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.791 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2024-04-26T18:12:44.791 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.791 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.791 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.791 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2024-04-26T18:12:44.792 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.792 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.792 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.792 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2024-04-26T18:12:44.792 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2024-04-26T18:12:44.792 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.792 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.792 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.792 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.792 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.792 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.792 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.792 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.793 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.793 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.793 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.793 INFO:teuthology.orchestra.run.smithi031.stdout: "exclude_perf_counters": { 2024-04-26T18:12:44.793 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "exclude_perf_counters", 2024-04-26T18:12:44.793 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.793 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.793 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.793 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.793 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.793 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.793 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T18:12:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T18:12:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2024-04-26T18:12:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2024-04-26T18:12:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T18:12:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T18:12:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2024-04-26T18:12:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2024-04-26T18:12:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2024-04-26T18:12:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2024-04-26T18:12:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-26T18:12:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-26T18:12:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-26T18:12:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T18:12:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.804 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.804 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.804 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-26T18:12:44.804 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-26T18:12:44.804 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.804 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.804 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.804 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2024-04-26T18:12:44.804 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.804 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.804 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.804 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T18:12:44.805 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.805 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.805 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.805 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2024-04-26T18:12:44.805 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2024-04-26T18:12:44.805 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.805 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.805 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.805 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2024-04-26T18:12:44.805 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.805 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.805 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.805 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.806 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.806 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.806 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.806 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-26T18:12:44.806 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-26T18:12:44.806 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.806 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.806 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.806 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2024-04-26T18:12:44.806 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.806 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.806 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.807 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2024-04-26T18:12:44.807 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2024-04-26T18:12:44.807 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.807 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.807 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.807 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.807 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.807 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-26T18:12:44.807 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-26T18:12:44.807 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.807 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.807 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.808 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-26T18:12:44.808 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-26T18:12:44.808 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-26T18:12:44.808 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.808 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.808 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.808 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.808 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.808 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.808 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.809 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.810 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.810 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.810 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.810 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.810 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.810 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.810 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.810 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.810 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.810 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.810 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.811 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.811 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.811 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.811 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.811 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.811 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.811 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.811 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.811 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.811 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.811 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.811 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.812 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.812 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.812 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.812 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.812 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.812 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.812 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.812 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.812 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.812 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.812 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.812 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.813 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.813 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.813 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.813 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.813 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.813 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.813 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.813 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.813 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.813 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.813 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.813 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.814 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.814 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.814 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.814 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.814 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.814 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.814 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.814 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.814 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2024-04-26T18:12:44.814 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2024-04-26T18:12:44.815 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.815 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.815 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.815 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.815 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.815 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.815 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.815 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.815 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.815 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.815 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.815 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2024-04-26T18:12:44.816 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T18:12:44.816 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.816 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.816 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.816 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.816 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.816 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.816 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.816 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.816 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.816 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.816 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.816 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2024-04-26T18:12:44.817 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2024-04-26T18:12:44.817 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.817 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.817 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.817 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.817 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.817 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.817 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.817 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.817 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.817 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.817 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.817 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.817 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.818 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2024-04-26T18:12:44.818 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.818 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.818 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.818 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2024-04-26T18:12:44.818 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2024-04-26T18:12:44.818 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.818 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.818 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.818 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.819 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.819 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.819 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.819 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.819 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.819 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.819 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.819 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-26T18:12:44.819 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-26T18:12:44.819 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.819 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.820 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.820 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.820 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.820 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.820 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.820 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.820 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.820 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.820 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.820 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.820 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.820 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.821 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.821 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.821 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.821 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.821 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.821 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.821 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.821 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.821 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.821 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.821 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.821 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.821 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.822 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.822 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.822 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.822 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.822 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.822 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.822 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.822 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.822 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.822 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.822 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.822 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.823 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.823 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.823 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.823 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.823 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.823 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.823 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.823 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.823 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.823 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.823 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.823 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.824 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.824 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.824 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.824 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.824 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.824 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.824 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.824 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.824 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.824 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.824 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.824 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.825 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.825 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.825 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.825 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.825 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.825 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.825 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.825 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.825 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.825 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.825 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.825 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.825 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.826 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.826 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-26T18:12:44.826 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-26T18:12:44.826 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.826 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.826 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.826 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.826 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.826 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.826 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.826 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.826 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.826 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.827 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.827 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-26T18:12:44.827 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-26T18:12:44.827 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.827 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.827 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.827 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.827 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.827 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.827 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.827 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.827 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.827 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.828 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.828 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.828 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.828 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2024-04-26T18:12:44.828 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.828 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.828 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.828 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.828 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.828 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.828 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.829 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.829 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.829 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.829 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.829 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.829 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.829 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.829 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.829 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.829 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.829 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.829 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.830 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.830 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.830 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.830 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.830 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.830 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.830 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.830 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.830 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.830 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.830 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.831 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.831 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.831 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.831 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.831 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.831 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.831 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.831 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.831 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.831 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.831 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.831 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.832 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.832 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.832 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.832 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.832 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.832 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.832 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.832 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.832 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.832 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.832 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.832 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.832 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.833 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.833 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.833 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.833 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.833 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.833 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.833 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.833 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.833 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.833 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.833 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.833 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.834 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.834 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.834 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.834 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.834 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2024-04-26T18:12:44.834 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.834 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.834 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.834 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.834 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.834 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.835 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.835 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.835 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.835 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.835 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.835 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.835 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.835 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.835 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.835 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.835 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.835 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.835 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.836 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.836 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.836 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.836 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.836 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.836 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.836 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.836 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.836 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.836 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.836 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.836 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.836 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.837 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.837 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.837 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.837 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.837 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.837 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.837 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.837 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.837 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.837 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.837 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.837 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.838 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.838 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.838 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.838 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.838 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.838 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.838 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.838 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.838 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.838 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.838 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.838 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.839 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.839 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.839 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.839 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.839 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.839 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.839 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.839 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.839 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.839 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.839 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.839 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.839 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.840 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2024-04-26T18:12:44.840 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2024-04-26T18:12:44.840 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.840 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.840 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.840 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2024-04-26T18:12:44.840 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.840 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.840 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.840 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T18:12:44.840 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.840 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.841 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.841 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.841 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.841 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2024-04-26T18:12:44.841 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.841 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.841 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.841 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.841 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.841 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.841 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.842 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.842 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.842 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.842 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.842 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.842 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.842 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.842 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.842 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.842 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.842 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.842 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.842 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.843 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.843 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.843 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.843 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.843 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.843 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.843 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.843 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.843 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.843 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.843 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.843 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.843 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.844 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.844 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.844 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.844 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.844 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.844 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.844 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.844 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.844 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.844 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.845 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.845 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.845 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.845 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.845 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.845 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.845 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.845 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.845 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.845 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.845 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.845 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.845 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.846 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.846 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.846 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.846 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.846 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.846 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.846 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.846 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.846 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.846 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.846 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.846 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.847 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.847 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2024-04-26T18:12:44.847 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2024-04-26T18:12:44.847 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.847 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.847 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.847 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.847 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.847 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.847 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.847 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.847 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.848 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.848 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.848 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2024-04-26T18:12:44.848 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2024-04-26T18:12:44.848 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.848 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.848 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.848 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2024-04-26T18:12:44.848 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.848 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.848 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.848 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.849 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.849 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.849 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.849 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2024-04-26T18:12:44.849 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2024-04-26T18:12:44.849 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.849 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.849 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.849 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.849 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.849 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.849 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.850 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.850 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.850 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.850 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.850 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2024-04-26T18:12:44.850 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2024-04-26T18:12:44.850 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.850 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.850 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.850 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.850 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2024-04-26T18:12:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2024-04-26T18:12:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.852 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.852 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.852 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.852 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.852 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.852 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.852 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.852 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2024-04-26T18:12:44.852 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2024-04-26T18:12:44.852 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.852 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.852 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.852 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.853 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.853 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.853 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.853 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.853 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.853 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.853 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.853 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2024-04-26T18:12:44.853 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2024-04-26T18:12:44.853 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.853 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.853 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.853 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.854 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.854 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.854 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.854 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.854 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.854 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.854 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.854 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2024-04-26T18:12:44.854 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2024-04-26T18:12:44.854 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.854 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.854 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.854 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.855 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.855 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.855 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.855 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.855 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.855 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.855 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.855 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2024-04-26T18:12:44.855 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2024-04-26T18:12:44.855 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.855 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.855 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.856 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.856 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-26T18:12:44.856 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2024-04-26T18:12:44.856 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.856 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.856 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.856 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.856 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.856 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2024-04-26T18:12:44.856 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2024-04-26T18:12:44.856 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.856 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.857 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.857 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.857 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.857 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.857 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.857 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.857 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.857 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.857 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.857 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2024-04-26T18:12:44.857 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2024-04-26T18:12:44.857 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.857 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.858 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.858 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.858 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.858 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.858 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.858 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.858 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.858 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.858 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.858 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2024-04-26T18:12:44.858 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2024-04-26T18:12:44.858 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.858 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.859 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.859 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.859 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.859 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.859 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.859 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.859 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.859 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.859 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.859 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.859 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.860 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-04-26T18:12:44.860 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.860 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.860 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.860 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2024-04-26T18:12:44.860 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2024-04-26T18:12:44.860 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.860 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.860 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.860 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.860 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.860 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.860 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.861 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T18:12:44.861 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.861 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.861 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.861 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2024-04-26T18:12:44.861 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2024-04-26T18:12:44.861 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.861 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.861 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.861 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.861 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.861 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.862 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.862 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2024-04-26T18:12:44.862 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.862 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.862 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.862 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.862 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.862 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.862 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.862 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.862 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.862 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.864 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.864 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.864 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.864 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.864 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.864 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.864 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.864 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.864 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.864 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.864 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.864 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.864 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.865 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.865 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.865 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.865 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.865 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.865 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.865 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.865 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.865 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.865 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.865 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.865 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.865 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.865 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.866 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.866 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.866 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.866 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.866 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.866 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.866 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.866 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.866 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.866 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.866 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.866 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.867 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.867 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.867 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.867 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.867 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.867 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.867 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.867 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.867 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.867 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.867 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2024-04-26T18:12:44.867 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.868 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.868 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.868 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.868 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.868 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.868 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.868 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.868 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.868 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.868 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.868 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.868 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.868 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.868 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.869 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.869 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.869 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.869 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.869 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.869 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.869 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.869 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.869 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.869 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.869 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.869 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2024-04-26T18:12:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.875 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.875 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.875 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.875 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.875 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.875 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.875 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.875 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.875 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.875 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.875 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.875 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2024-04-26T18:12:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2024-04-26T18:12:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2024-04-26T18:12:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T18:12:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2024-04-26T18:12:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.882 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.882 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.882 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.882 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.882 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.882 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.882 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.882 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.882 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.882 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.882 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.882 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.882 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.882 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.883 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.883 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.883 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.883 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.883 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.883 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.883 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.883 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.883 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.883 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.883 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.883 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.884 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.884 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.884 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.884 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.884 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.884 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.884 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.884 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.884 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.884 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.884 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.884 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.884 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.885 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.885 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.885 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.885 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.885 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.885 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.885 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.885 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.885 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.885 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.885 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.885 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.885 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.886 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.886 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.886 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.886 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.886 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.886 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.886 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.886 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.886 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.886 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.886 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.886 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.886 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.887 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.887 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2024-04-26T18:12:44.887 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.887 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.887 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.887 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2024-04-26T18:12:44.887 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2024-04-26T18:12:44.887 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.887 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.887 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.887 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.887 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.887 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.888 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.888 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T18:12:44.888 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.888 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.888 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.888 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2024-04-26T18:12:44.888 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2024-04-26T18:12:44.888 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.888 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.888 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.888 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.888 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.888 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.889 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.889 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T18:12:44.889 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.889 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.889 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.889 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2024-04-26T18:12:44.889 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2024-04-26T18:12:44.889 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.889 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.889 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.889 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.890 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.890 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.890 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.890 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T18:12:44.890 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.890 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.890 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.890 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2024-04-26T18:12:44.890 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2024-04-26T18:12:44.891 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.891 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.891 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.891 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.891 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.891 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.891 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.891 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T18:12:44.891 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.891 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.891 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.891 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2024-04-26T18:12:44.892 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2024-04-26T18:12:44.892 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.892 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.892 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.892 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.892 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.892 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.892 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.892 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T18:12:44.892 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.892 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.892 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.892 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2024-04-26T18:12:44.893 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2024-04-26T18:12:44.893 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.893 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.893 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.893 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.893 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.893 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.893 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.893 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.893 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.893 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.893 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.894 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2024-04-26T18:12:44.894 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2024-04-26T18:12:44.894 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.894 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.894 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.894 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.894 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.894 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.894 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.894 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.894 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.894 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.894 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.895 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2024-04-26T18:12:44.895 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2024-04-26T18:12:44.895 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.895 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.895 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.895 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T18:12:44.895 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.895 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.895 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.895 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.895 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.895 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.895 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.896 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-26T18:12:44.896 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-26T18:12:44.896 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.896 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.896 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.896 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.896 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.896 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.896 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.896 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.896 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.896 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.896 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.897 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:44.897 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:44.897 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.897 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.897 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.897 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2024-04-26T18:12:44.897 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2024-04-26T18:12:44.897 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.897 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.897 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.897 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.897 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.897 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.898 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2024-04-26T18:12:44.898 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2024-04-26T18:12:44.898 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.898 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.898 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.898 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2024-04-26T18:12:44.898 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.898 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.898 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.898 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.898 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.898 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.899 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.899 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2024-04-26T18:12:44.899 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2024-04-26T18:12:44.899 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.899 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.899 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.899 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.899 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.899 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.899 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.899 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.899 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.899 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.900 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.900 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2024-04-26T18:12:44.900 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2024-04-26T18:12:44.900 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.900 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.900 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.900 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.900 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.900 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.900 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.900 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.901 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.901 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.901 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.901 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.901 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.901 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.901 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.901 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.901 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.901 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.901 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.901 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.902 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.902 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.902 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.902 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.902 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.902 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.902 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.902 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.902 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.902 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.902 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.902 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.903 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.903 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.903 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.903 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.903 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.903 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.903 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.903 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.903 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.903 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.903 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.903 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.904 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.904 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.904 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.904 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.904 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.904 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.904 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.905 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.905 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.905 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.905 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.905 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.905 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.905 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.905 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.905 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.906 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.906 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.906 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.906 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.906 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.906 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.906 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.906 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.906 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.906 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.906 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.908 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.908 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.908 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.908 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.908 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.908 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2024-04-26T18:12:44.908 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2024-04-26T18:12:44.908 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.908 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.908 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.908 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.908 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.909 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.909 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.909 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.909 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.909 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.909 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.909 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2024-04-26T18:12:44.909 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2024-04-26T18:12:44.909 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.909 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.909 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.909 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T18:12:44.909 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.910 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.910 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.910 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.910 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.910 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.910 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.910 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.910 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.910 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2024-04-26T18:12:44.910 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.910 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.911 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.911 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.911 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.911 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.911 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.911 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.911 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.911 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.911 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.911 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.911 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.911 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.911 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.911 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.912 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.912 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.912 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.912 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.912 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.912 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.912 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.912 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.912 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.912 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.912 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.912 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.912 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.914 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.914 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.914 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.914 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.914 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.914 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.914 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.914 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.914 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.914 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.914 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.914 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.914 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.915 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.915 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.915 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.915 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.915 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.915 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.915 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.915 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.915 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.915 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.915 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.915 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.915 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.916 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.916 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.916 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.916 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-04-26T18:12:44.916 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.916 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.916 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.916 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.916 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.916 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.916 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.916 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.916 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.917 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.917 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.917 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.917 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.917 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.917 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.917 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.917 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.917 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.917 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.917 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.917 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.917 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.917 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2024-04-26T18:12:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2024-04-26T18:12:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2024-04-26T18:12:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T18:12:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "periodic_async_work": { 2024-04-26T18:12:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "periodic_async_work", 2024-04-26T18:12:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Periodically check for async work", 2024-04-26T18:12:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2024-04-26T18:12:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2024-04-26T18:12:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T18:12:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_no_wait": { 2024-04-26T18:12:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T18:12:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T18:12:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2024-04-26T18:12:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2024-04-26T18:12:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2024-04-26T18:12:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.927 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.927 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2024-04-26T18:12:44.927 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.927 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.927 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.927 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.927 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.927 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.927 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.927 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2024-04-26T18:12:44.927 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2024-04-26T18:12:44.927 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.927 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.928 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.928 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.928 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.928 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.928 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.928 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.928 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.928 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.928 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.928 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:44.928 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:44.928 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.928 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.930 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.930 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.930 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.930 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.930 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.930 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.930 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.930 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.930 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.930 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.930 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.930 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.930 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.930 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.931 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.931 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.931 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.931 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.931 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.931 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.931 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.931 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.931 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.931 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.931 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.931 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.931 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.932 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.932 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.932 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.932 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.932 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.932 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.932 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.932 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.932 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.932 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.932 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.933 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.933 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.933 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.933 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.933 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.933 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.933 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.933 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.933 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.933 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.933 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.933 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.934 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.934 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.934 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.934 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.934 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.934 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.934 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.934 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.934 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.934 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.934 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.934 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.934 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.935 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2024-04-26T18:12:44.935 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2024-04-26T18:12:44.935 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.935 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.935 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.935 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.935 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.935 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.935 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.935 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.935 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.935 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.935 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.936 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2024-04-26T18:12:44.936 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2024-04-26T18:12:44.936 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.936 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.936 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.936 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2024-04-26T18:12:44.936 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.936 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.936 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.936 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.936 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.936 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.937 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.937 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2024-04-26T18:12:44.937 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2024-04-26T18:12:44.937 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.937 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.937 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.937 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T18:12:44.937 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.937 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.937 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.937 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.937 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.938 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.938 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.938 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.938 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.938 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.938 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-26T18:12:44.938 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.938 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.938 INFO:teuthology.orchestra.run.smithi031.stdout: "modules": [ 2024-04-26T18:12:44.938 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-04-26T18:12:44.938 INFO:teuthology.orchestra.run.smithi031.stdout: "dashboard", 2024-04-26T18:12:44.938 INFO:teuthology.orchestra.run.smithi031.stdout: "iostat", 2024-04-26T18:12:44.938 INFO:teuthology.orchestra.run.smithi031.stdout: "nfs", 2024-04-26T18:12:44.939 INFO:teuthology.orchestra.run.smithi031.stdout: "restful", 2024-04-26T18:12:44.939 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_schedule" 2024-04-26T18:12:44.939 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.939 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2024-04-26T18:12:44.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.939 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2024-04-26T18:12:44.939 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.939 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.939 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.939 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:44.939 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:44.939 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.939 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.939 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.940 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:44.940 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.940 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.940 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.940 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T18:12:44.940 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.940 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.940 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.940 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.940 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.940 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.940 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.941 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.941 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.941 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.941 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.941 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.941 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.941 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.941 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.941 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.941 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.941 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.941 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.941 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.942 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.942 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.942 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.942 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.942 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.942 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.942 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.942 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.942 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.942 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.942 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.942 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.942 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.943 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.943 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.943 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.943 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.943 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.943 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.943 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.943 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.943 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.943 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.943 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.943 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.943 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.944 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.944 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.944 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.944 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.944 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.944 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.944 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.944 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.944 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.944 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.944 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.944 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.944 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.945 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.945 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.945 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.945 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.945 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.945 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.945 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.945 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.945 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.945 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.945 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2024-04-26T18:12:44.945 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2024-04-26T18:12:44.945 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.946 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.946 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.946 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.946 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.946 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.946 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.946 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2024-04-26T18:12:44.946 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.946 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.946 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.946 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2024-04-26T18:12:44.946 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2024-04-26T18:12:44.946 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.947 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.947 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.947 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2024-04-26T18:12:44.947 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.947 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.947 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.947 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2024-04-26T18:12:44.947 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.947 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.947 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.947 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2024-04-26T18:12:44.947 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2024-04-26T18:12:44.947 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.948 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.949 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.949 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.949 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.949 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.949 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.949 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2024-04-26T18:12:44.949 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.949 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.949 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.949 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2024-04-26T18:12:44.949 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2024-04-26T18:12:44.949 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.950 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.950 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.950 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2024-04-26T18:12:44.950 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.950 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.950 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.950 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2024-04-26T18:12:44.950 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.950 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.950 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.950 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2024-04-26T18:12:44.950 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2024-04-26T18:12:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2024-04-26T18:12:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2024-04-26T18:12:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2024-04-26T18:12:44.952 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.952 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.952 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.952 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.952 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.952 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.952 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.952 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T18:12:44.952 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.952 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.952 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.952 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2024-04-26T18:12:44.953 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2024-04-26T18:12:44.953 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.953 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.953 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.953 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.953 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.953 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.953 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.953 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2024-04-26T18:12:44.953 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.953 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.953 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.953 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.954 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.954 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2024-04-26T18:12:44.954 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.954 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.954 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.954 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2024-04-26T18:12:44.954 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2024-04-26T18:12:44.954 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.954 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.954 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.954 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.955 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.955 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.955 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.955 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T18:12:44.955 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.955 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.955 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.955 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2024-04-26T18:12:44.955 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2024-04-26T18:12:44.955 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.955 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.955 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.956 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2024-04-26T18:12:44.956 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.956 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.956 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.956 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T18:12:44.956 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T18:12:44.956 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.956 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.956 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2024-04-26T18:12:44.956 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2024-04-26T18:12:44.956 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:44.956 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.956 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.957 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:44.957 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-26T18:12:44.957 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-26T18:12:44.957 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.957 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T18:12:44.957 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T18:12:44.957 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.957 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.957 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2024-04-26T18:12:44.957 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2024-04-26T18:12:44.957 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:44.957 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.957 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.958 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2024-04-26T18:12:44.958 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-26T18:12:44.958 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2024-04-26T18:12:44.958 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.958 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T18:12:44.958 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.958 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.958 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.958 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2024-04-26T18:12:44.958 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2024-04-26T18:12:44.958 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.958 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.959 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.959 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T18:12:44.959 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.959 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.959 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.959 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T18:12:44.959 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T18:12:44.959 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.959 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.959 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2024-04-26T18:12:44.959 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2024-04-26T18:12:44.959 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.960 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.960 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.960 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2024-04-26T18:12:44.960 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2024-04-26T18:12:44.960 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2024-04-26T18:12:44.960 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.960 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2024-04-26T18:12:44.960 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T18:12:44.960 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.960 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.960 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2024-04-26T18:12:44.960 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2024-04-26T18:12:44.961 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.961 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.961 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.961 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2024-04-26T18:12:44.961 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.961 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.961 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.961 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T18:12:44.961 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T18:12:44.961 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.961 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.961 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2024-04-26T18:12:44.962 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2024-04-26T18:12:44.962 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:44.962 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.962 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.962 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:44.962 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-26T18:12:44.962 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-26T18:12:44.962 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.962 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T18:12:44.962 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T18:12:44.962 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.962 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.963 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:44.963 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:44.963 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.963 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.963 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.963 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.963 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.963 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.963 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.963 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.963 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.963 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.963 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.964 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.964 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.964 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.964 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.964 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.964 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.964 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.964 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:44.964 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:44.964 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.964 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.964 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.966 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.966 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:44.966 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.966 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.966 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.966 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:44.966 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:44.966 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:44.966 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:44.966 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:44.966 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:44.966 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.966 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.967 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.967 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.967 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.967 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:44.967 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:44.967 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.967 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.967 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.967 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.967 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.967 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.967 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.968 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:44.968 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.968 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.968 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.968 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2024-04-26T18:12:44.968 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2024-04-26T18:12:44.968 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.968 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.968 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.968 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:44.968 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.968 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.968 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.969 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T18:12:44.969 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.969 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.969 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.969 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-26T18:12:44.969 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-26T18:12:44.969 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.969 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.969 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.969 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2024-04-26T18:12:44.969 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.969 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.969 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.970 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2024-04-26T18:12:44.970 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2024-04-26T18:12:44.970 INFO:teuthology.orchestra.run.smithi031.stdout: "read", 2024-04-26T18:12:44.970 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap", 2024-04-26T18:12:44.970 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap-read" 2024-04-26T18:12:44.970 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.970 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2024-04-26T18:12:44.970 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.970 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.970 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.970 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2024-04-26T18:12:44.970 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2024-04-26T18:12:44.971 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.971 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.971 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.971 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.971 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.971 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.971 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.971 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T18:12:44.971 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.971 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.971 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.971 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:44.972 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:44.972 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.972 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.972 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.972 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:44.972 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.972 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.972 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.972 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T18:12:44.972 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.972 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.972 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.972 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2024-04-26T18:12:44.973 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2024-04-26T18:12:44.973 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.973 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.973 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.973 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-26T18:12:44.973 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-26T18:12:44.973 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.973 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.973 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T18:12:44.973 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T18:12:44.973 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.973 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.973 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2024-04-26T18:12:44.974 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2024-04-26T18:12:44.974 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:44.974 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.974 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.974 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:44.974 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.974 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.974 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.974 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T18:12:44.974 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.974 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.974 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.974 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.974 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:44.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:44.975 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2024-04-26T18:12:44.975 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:44.975 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:44.975 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:44.975 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2024-04-26T18:12:44.975 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2024-04-26T18:12:44.975 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.975 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.975 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.975 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-26T18:12:44.975 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.975 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.976 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.976 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T18:12:44.976 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.976 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.976 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.976 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2024-04-26T18:12:44.976 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2024-04-26T18:12:44.976 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.976 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.976 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.976 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2024-04-26T18:12:44.976 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.977 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.977 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.977 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T18:12:44.977 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.977 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.977 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.977 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2024-04-26T18:12:44.977 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2024-04-26T18:12:44.977 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:44.977 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.977 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.977 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2024-04-26T18:12:44.977 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.978 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.978 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.978 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T18:12:44.978 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.978 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.978 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.978 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2024-04-26T18:12:44.978 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2024-04-26T18:12:44.978 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.978 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.978 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.978 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.978 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.979 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.979 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.979 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T18:12:44.979 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T18:12:44.979 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.979 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.979 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2024-04-26T18:12:44.979 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2024-04-26T18:12:44.979 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:44.979 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.979 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.979 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:44.979 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T18:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2024-04-26T18:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T18:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2024-04-26T18:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T18:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_log_destination": { 2024-04-26T18:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm_log_destination", 2024-04-26T18:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:44.982 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.982 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.982 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:44.982 INFO:teuthology.orchestra.run.smithi031.stdout: "file", 2024-04-26T18:12:44.982 INFO:teuthology.orchestra.run.smithi031.stdout: "file,syslog", 2024-04-26T18:12:44.982 INFO:teuthology.orchestra.run.smithi031.stdout: "syslog" 2024-04-26T18:12:44.982 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:44.982 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T18:12:44.982 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.982 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.982 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.982 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2024-04-26T18:12:44.983 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2024-04-26T18:12:44.983 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.983 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.983 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.983 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.983 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.983 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.983 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.983 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T18:12:44.983 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.983 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.983 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.984 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2024-04-26T18:12:44.985 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2024-04-26T18:12:44.985 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:44.985 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.985 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.985 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:44.985 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.985 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.985 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.985 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T18:12:44.985 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.985 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.985 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.986 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2024-04-26T18:12:44.986 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2024-04-26T18:12:44.986 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.986 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.986 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.986 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T18:12:44.986 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.986 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.986 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.986 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-26T18:12:44.986 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.986 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.986 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.987 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2024-04-26T18:12:44.987 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2024-04-26T18:12:44.987 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.987 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.987 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:44.987 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T18:12:44.987 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.987 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.987 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.987 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2024-04-26T18:12:44.987 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.987 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.987 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2024-04-26T18:12:44.989 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2024-04-26T18:12:44.989 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.989 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.989 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.989 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T18:12:44.989 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.989 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.989 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.989 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-26T18:12:44.989 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.989 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.989 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.990 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2024-04-26T18:12:44.990 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2024-04-26T18:12:44.990 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.990 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.990 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.990 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T18:12:44.990 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.990 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.990 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.990 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2024-04-26T18:12:44.990 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.990 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.990 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.991 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2024-04-26T18:12:44.991 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2024-04-26T18:12:44.991 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.991 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.991 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.991 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T18:12:44.991 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.991 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.991 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.991 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2024-04-26T18:12:44.991 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.991 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.991 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.992 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2024-04-26T18:12:44.992 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2024-04-26T18:12:44.992 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.992 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.992 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.992 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T18:12:44.992 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.992 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.992 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.992 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2024-04-26T18:12:44.992 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.992 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.992 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.993 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2024-04-26T18:12:44.993 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2024-04-26T18:12:44.993 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.993 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.993 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.993 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T18:12:44.993 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.993 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.993 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.993 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2024-04-26T18:12:44.993 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.993 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.994 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.994 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2024-04-26T18:12:44.994 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2024-04-26T18:12:44.994 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.994 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.994 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.994 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T18:12:44.994 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.994 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.994 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.994 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2024-04-26T18:12:44.994 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.994 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.995 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.995 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2024-04-26T18:12:44.995 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2024-04-26T18:12:44.995 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.995 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.995 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.995 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T18:12:44.995 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.995 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.995 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.995 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2024-04-26T18:12:44.996 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.996 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.996 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.996 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2024-04-26T18:12:44.996 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2024-04-26T18:12:44.996 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.996 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.996 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.996 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T18:12:44.996 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.996 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.996 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.997 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-26T18:12:44.997 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.997 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.997 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.997 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nvmeof": { 2024-04-26T18:12:44.997 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nvmeof", 2024-04-26T18:12:44.997 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.997 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.997 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.997 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T18:12:44.997 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.997 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.997 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.998 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nvme-of container image", 2024-04-26T18:12:44.998 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.998 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.998 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.998 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2024-04-26T18:12:44.998 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2024-04-26T18:12:44.998 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.998 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.998 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.998 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T18:12:44.998 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.998 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:44.998 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:44.999 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-26T18:12:44.999 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:44.999 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:44.999 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:44.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:44.999 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2024-04-26T18:12:44.999 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2024-04-26T18:12:44.999 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:44.999 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:44.999 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:44.999 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T18:12:44.999 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:44.999 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.000 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.000 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2024-04-26T18:12:45.000 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.000 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.000 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.000 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_samba": { 2024-04-26T18:12:45.000 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_samba", 2024-04-26T18:12:45.000 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.000 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.000 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.000 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T18:12:45.000 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.001 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.001 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.001 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Samba/SMB container image", 2024-04-26T18:12:45.001 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.001 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.001 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.001 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2024-04-26T18:12:45.001 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2024-04-26T18:12:45.001 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.001 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.001 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.001 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T18:12:45.002 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.002 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.002 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.002 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2024-04-26T18:12:45.002 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.002 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.002 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.002 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2024-04-26T18:12:45.002 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2024-04-26T18:12:45.002 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.002 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.002 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.003 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.003 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.003 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.003 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.003 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T18:12:45.003 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.003 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.003 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.003 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2024-04-26T18:12:45.003 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2024-04-26T18:12:45.003 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.003 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.004 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.004 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:45.004 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.004 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.004 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.004 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T18:12:45.004 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.004 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.004 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.004 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2024-04-26T18:12:45.004 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T18:12:45.004 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.004 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.005 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.005 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2024-04-26T18:12:45.005 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.005 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.005 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.005 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T18:12:45.005 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.005 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.005 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.005 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2024-04-26T18:12:45.005 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2024-04-26T18:12:45.005 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.006 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.006 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.006 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2024-04-26T18:12:45.006 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.006 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.006 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.006 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T18:12:45.006 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.006 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.006 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.006 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2024-04-26T18:12:45.006 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2024-04-26T18:12:45.007 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.007 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.007 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.007 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2024-04-26T18:12:45.007 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.007 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.007 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.007 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2024-04-26T18:12:45.007 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.007 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.007 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.007 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2024-04-26T18:12:45.008 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2024-04-26T18:12:45.008 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.008 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.008 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.008 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.008 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.008 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.008 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.008 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T18:12:45.008 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.008 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.008 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.008 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2024-04-26T18:12:45.009 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2024-04-26T18:12:45.009 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.009 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.009 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.009 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:45.009 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.009 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.009 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.009 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2024-04-26T18:12:45.009 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.009 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.009 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.009 INFO:teuthology.orchestra.run.smithi031.stdout: "grafana_dashboards_path": { 2024-04-26T18:12:45.010 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "grafana_dashboards_path", 2024-04-26T18:12:45.010 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.010 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.010 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.010 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T18:12:45.010 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.010 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.010 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.010 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T18:12:45.010 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.010 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.010 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.011 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2024-04-26T18:12:45.011 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2024-04-26T18:12:45.011 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.011 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.011 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.011 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:45.011 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.011 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.011 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.011 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2024-04-26T18:12:45.011 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.011 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.011 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.012 INFO:teuthology.orchestra.run.smithi031.stdout: "hw_monitoring": { 2024-04-26T18:12:45.012 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hw_monitoring", 2024-04-26T18:12:45.012 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.012 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.012 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.012 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.012 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.012 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.012 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.012 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T18:12:45.012 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.013 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.013 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.013 INFO:teuthology.orchestra.run.smithi031.stdout: "inventory_list_all": { 2024-04-26T18:12:45.013 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "inventory_list_all", 2024-04-26T18:12:45.013 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.013 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.013 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.013 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.013 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.013 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.013 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.013 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T18:12:45.013 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.014 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.014 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.014 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.014 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.014 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.014 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.014 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.014 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.014 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.014 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.014 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.014 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.015 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.015 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.015 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.015 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.015 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.015 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.015 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.015 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.015 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.015 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.015 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2024-04-26T18:12:45.015 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2024-04-26T18:12:45.015 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T18:12:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.017 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.017 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.017 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.017 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.017 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.017 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.017 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T18:12:45.017 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.017 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.017 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.017 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.017 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.017 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.018 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.018 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.018 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.018 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.018 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.018 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.018 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.018 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.018 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.018 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.018 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.018 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.018 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.019 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.019 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.019 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.019 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.019 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.019 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.019 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.019 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.019 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.019 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.019 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.019 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.019 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.020 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.020 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.020 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.020 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.020 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T18:12:45.020 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T18:12:45.020 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.020 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.020 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.020 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.020 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.020 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.020 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.021 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T18:12:45.021 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.021 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.021 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.021 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T18:12:45.021 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T18:12:45.021 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.021 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.021 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.021 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2024-04-26T18:12:45.021 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.021 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.022 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.022 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T18:12:45.022 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.022 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.022 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.022 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2024-04-26T18:12:45.022 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2024-04-26T18:12:45.022 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.022 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.022 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.022 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:45.023 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.023 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.023 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.023 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2024-04-26T18:12:45.023 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.023 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.023 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.023 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2024-04-26T18:12:45.023 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2024-04-26T18:12:45.023 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.023 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.023 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.023 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:45.024 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.024 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.024 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.024 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T18:12:45.024 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.024 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.024 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.024 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2024-04-26T18:12:45.024 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2024-04-26T18:12:45.024 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.024 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.024 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.025 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.025 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.025 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.025 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.025 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2024-04-26T18:12:45.025 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.025 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.025 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.025 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-26T18:12:45.025 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-26T18:12:45.025 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.025 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.026 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.026 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2024-04-26T18:12:45.026 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.026 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.026 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.026 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2024-04-26T18:12:45.026 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2024-04-26T18:12:45.026 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.026 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T18:12:45.026 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.026 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.026 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.027 INFO:teuthology.orchestra.run.smithi031.stdout: "oob_default_addr": { 2024-04-26T18:12:45.027 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "oob_default_addr", 2024-04-26T18:12:45.027 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.027 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.027 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.027 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "169.254.1.1", 2024-04-26T18:12:45.027 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.027 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.027 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.027 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T18:12:45.027 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.027 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.028 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.028 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2024-04-26T18:12:45.028 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2024-04-26T18:12:45.028 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.028 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.028 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.028 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T18:12:45.028 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.028 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.028 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.028 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T18:12:45.028 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.028 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.029 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.029 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2024-04-26T18:12:45.029 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2024-04-26T18:12:45.029 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.029 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.029 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.029 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.029 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.029 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.029 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.029 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T18:12:45.029 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.030 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.030 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.030 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2024-04-26T18:12:45.030 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2024-04-26T18:12:45.030 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.030 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.030 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.030 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.030 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.030 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.030 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.030 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T18:12:45.030 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.031 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.031 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.031 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2024-04-26T18:12:45.031 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2024-04-26T18:12:45.031 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.031 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.031 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.031 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.031 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.031 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.031 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.031 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T18:12:45.032 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.032 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.032 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.032 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2024-04-26T18:12:45.032 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2024-04-26T18:12:45.032 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.032 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.032 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.032 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.032 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.032 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.033 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.034 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T18:12:45.034 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.034 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.034 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.034 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2024-04-26T18:12:45.034 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2024-04-26T18:12:45.034 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.034 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.034 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.034 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2024-04-26T18:12:45.034 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.034 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2024-04-26T18:12:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2024-04-26T18:12:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2024-04-26T18:12:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.035 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.036 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.036 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T18:12:45.036 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.036 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.036 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.036 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2024-04-26T18:12:45.036 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2024-04-26T18:12:45.036 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.036 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.036 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.036 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.036 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.037 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.037 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.037 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T18:12:45.037 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.037 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.037 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.037 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2024-04-26T18:12:45.037 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2024-04-26T18:12:45.037 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.037 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.037 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.037 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T18:12:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2024-04-26T18:12:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2024-04-26T18:12:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.040 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.040 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.040 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.040 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.040 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T18:12:45.040 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.040 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.040 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.040 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2024-04-26T18:12:45.040 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2024-04-26T18:12:45.040 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.040 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.040 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.041 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.041 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.041 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.041 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.041 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T18:12:45.041 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.041 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.041 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.041 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.041 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.041 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.041 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2024-04-26T18:12:45.041 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.042 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.042 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.042 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.042 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.042 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.042 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.042 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.042 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.042 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.042 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.042 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.042 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.042 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.043 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.043 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.043 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.043 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.043 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.043 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.043 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.043 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.043 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.043 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.043 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.043 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.043 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.044 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.044 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.044 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.044 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.044 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.044 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.044 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.044 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.044 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.044 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.044 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.044 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.044 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.045 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.045 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.045 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.045 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.045 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.045 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.045 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.045 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.045 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.045 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.045 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.045 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.045 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.045 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.046 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.046 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.046 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.046 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.046 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.046 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.046 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.046 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.046 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.046 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.046 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.046 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.047 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.047 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.047 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.047 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2024-04-26T18:12:45.047 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2024-04-26T18:12:45.047 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.047 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.047 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.047 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2024-04-26T18:12:45.047 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.047 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.047 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.047 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T18:12:45.048 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.048 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.048 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.048 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2024-04-26T18:12:45.048 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2024-04-26T18:12:45.048 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.048 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.048 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.048 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2024-04-26T18:12:45.048 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.048 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.048 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.048 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T18:12:45.049 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.049 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.049 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.049 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.049 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.049 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2024-04-26T18:12:45.049 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.049 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.049 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.049 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T18:12:45.049 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T18:12:45.050 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.050 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.050 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.050 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:45.050 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.050 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.050 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.050 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.050 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.050 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.050 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.050 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T18:12:45.051 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T18:12:45.051 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.051 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.051 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.051 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.051 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.051 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.051 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.051 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.051 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.051 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.051 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.052 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T18:12:45.052 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T18:12:45.052 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.052 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.052 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.052 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.052 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.052 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.052 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.052 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.052 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.052 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.052 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2024-04-26T18:12:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T18:12:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.054 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T18:12:45.054 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T18:12:45.054 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.054 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.054 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.054 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.054 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.054 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.054 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.054 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.054 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.054 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.055 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.055 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T18:12:45.055 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T18:12:45.055 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.055 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.055 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.055 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.055 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.055 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.055 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.055 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.055 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.055 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.056 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.056 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T18:12:45.056 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T18:12:45.056 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.056 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.056 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.056 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.056 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.056 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.056 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.056 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.056 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.056 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.057 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.057 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T18:12:45.057 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T18:12:45.057 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.057 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.057 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.057 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.057 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.057 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.057 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.057 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.057 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.057 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.058 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.058 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T18:12:45.058 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T18:12:45.058 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.058 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.058 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.058 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.058 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.058 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.058 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.058 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.058 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.059 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.059 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.059 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T18:12:45.059 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T18:12:45.059 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.059 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.059 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.059 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.059 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.059 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.059 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.059 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.060 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.060 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.060 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.060 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T18:12:45.060 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T18:12:45.060 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.060 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.060 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.060 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.060 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.060 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.060 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.061 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.061 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.061 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.061 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.061 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T18:12:45.061 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T18:12:45.061 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.061 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.061 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.061 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.061 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.061 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.062 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.062 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.062 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.062 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.062 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.062 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T18:12:45.062 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T18:12:45.062 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.062 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.062 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.062 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.062 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.062 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.063 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.063 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.063 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.063 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.063 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.063 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T18:12:45.063 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T18:12:45.063 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.063 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.063 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.063 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.063 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.064 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.064 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.064 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.064 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.064 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.064 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.064 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T18:12:45.064 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T18:12:45.064 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.064 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.064 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.064 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-26T18:12:45.064 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.065 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.065 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.065 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.065 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.065 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.065 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.065 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T18:12:45.065 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T18:12:45.065 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.065 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.065 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.065 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.066 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.066 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.066 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.066 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.066 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.066 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.066 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.066 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2024-04-26T18:12:45.066 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2024-04-26T18:12:45.066 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.066 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.066 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.067 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.067 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.067 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.067 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.067 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.067 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.067 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.067 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.067 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T18:12:45.067 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T18:12:45.067 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.067 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.067 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.068 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-26T18:12:45.068 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.068 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.068 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.068 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.068 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.068 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.068 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.068 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T18:12:45.068 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T18:12:45.068 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.068 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.069 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.069 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.069 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.069 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.069 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.069 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.069 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.069 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.069 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.069 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T18:12:45.069 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T18:12:45.069 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.069 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.070 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.070 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.070 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.070 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.070 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.070 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.070 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.070 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.070 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.070 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T18:12:45.070 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T18:12:45.070 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.071 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.071 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.071 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.071 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.071 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.071 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.071 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.071 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.071 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.071 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.071 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T18:12:45.071 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T18:12:45.071 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.072 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.072 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.072 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.072 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.072 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.072 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.072 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.072 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.072 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.072 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.072 INFO:teuthology.orchestra.run.smithi031.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T18:12:45.072 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T18:12:45.073 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.073 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.073 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.073 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "[]", 2024-04-26T18:12:45.073 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.073 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.073 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.073 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.073 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.073 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.073 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.073 INFO:teuthology.orchestra.run.smithi031.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T18:12:45.074 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T18:12:45.074 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.074 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.074 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.074 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "{}", 2024-04-26T18:12:45.074 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.074 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.074 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.074 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.074 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.074 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.074 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.075 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T18:12:45.075 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T18:12:45.075 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.075 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.075 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.075 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.075 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.075 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.075 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.075 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.075 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.075 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.075 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.076 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T18:12:45.076 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T18:12:45.076 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.076 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.076 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.076 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.076 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.076 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.076 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.076 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.076 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.076 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.077 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.077 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T18:12:45.077 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T18:12:45.077 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.077 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.077 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.077 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.077 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.077 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.077 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.077 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.077 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.077 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T18:12:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T18:12:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T18:12:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T18:12:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T18:12:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T18:12:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T18:12:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T18:12:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T18:12:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T18:12:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T18:12:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T18:12:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T18:12:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T18:12:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T18:12:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T18:12:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T18:12:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T18:12:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T18:12:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2024-04-26T18:12:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T18:12:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T18:12:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2024-04-26T18:12:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.089 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.089 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.089 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.089 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.089 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T18:12:45.089 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T18:12:45.089 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.089 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.089 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.089 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.089 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.089 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T18:12:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T18:12:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-26T18:12:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T18:12:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T18:12:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T18:12:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T18:12:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T18:12:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T18:12:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T18:12:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T18:12:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T18:12:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T18:12:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T18:12:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T18:12:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-26T18:12:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2024-04-26T18:12:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2024-04-26T18:12:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2024-04-26T18:12:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2024-04-26T18:12:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2024-04-26T18:12:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2024-04-26T18:12:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2024-04-26T18:12:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2024-04-26T18:12:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2024-04-26T18:12:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2024-04-26T18:12:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-26T18:12:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-26T18:12:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2024-04-26T18:12:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2024-04-26T18:12:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2024-04-26T18:12:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2024-04-26T18:12:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T18:12:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-26T18:12:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-26T18:12:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-26T18:12:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-26T18:12:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-26T18:12:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2024-04-26T18:12:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-26T18:12:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-26T18:12:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2024-04-26T18:12:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2024-04-26T18:12:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2024-04-26T18:12:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-26T18:12:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-26T18:12:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2024-04-26T18:12:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2024-04-26T18:12:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-26T18:12:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-26T18:12:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-26T18:12:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-26T18:12:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-26T18:12:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2024-04-26T18:12:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2024-04-26T18:12:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-04-26T18:12:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2024-04-26T18:12:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2024-04-26T18:12:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2024-04-26T18:12:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2024-04-26T18:12:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2024-04-26T18:12:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2024-04-26T18:12:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T18:12:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2024-04-26T18:12:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2024-04-26T18:12:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2024-04-26T18:12:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T18:12:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2024-04-26T18:12:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2024-04-26T18:12:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2024-04-26T18:12:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2024-04-26T18:12:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2024-04-26T18:12:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2024-04-26T18:12:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-26T18:12:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T18:12:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2024-04-26T18:12:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2024-04-26T18:12:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T18:12:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T18:12:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2024-04-26T18:12:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2024-04-26T18:12:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2024-04-26T18:12:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T18:12:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2024-04-26T18:12:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2024-04-26T18:12:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2024-04-26T18:12:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-26T18:12:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2024-04-26T18:12:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2024-04-26T18:12:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2024-04-26T18:12:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-26T18:12:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2024-04-26T18:12:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2024-04-26T18:12:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2024-04-26T18:12:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2024-04-26T18:12:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2024-04-26T18:12:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2024-04-26T18:12:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T18:12:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2024-04-26T18:12:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2024-04-26T18:12:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2024-04-26T18:12:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2024-04-26T18:12:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2024-04-26T18:12:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T18:12:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2024-04-26T18:12:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2024-04-26T18:12:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2024-04-26T18:12:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2024-04-26T18:12:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2024-04-26T18:12:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2024-04-26T18:12:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2024-04-26T18:12:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-26T18:12:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-26T18:12:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2024-04-26T18:12:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T18:12:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2024-04-26T18:12:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2024-04-26T18:12:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-26T18:12:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-26T18:12:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2024-04-26T18:12:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T18:12:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2024-04-26T18:12:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2024-04-26T18:12:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2024-04-26T18:12:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2024-04-26T18:12:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2024-04-26T18:12:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2024-04-26T18:12:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T18:12:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2024-04-26T18:12:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2024-04-26T18:12:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2024-04-26T18:12:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2024-04-26T18:12:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2024-04-26T18:12:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2024-04-26T18:12:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T18:12:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2024-04-26T18:12:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2024-04-26T18:12:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2024-04-26T18:12:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.173 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T18:12:45.173 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.173 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.173 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.173 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.173 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.173 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.173 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.173 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.173 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.173 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.174 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.174 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.174 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.174 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.174 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.174 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.174 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.174 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.174 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.174 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.174 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.174 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.174 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.175 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.175 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.175 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.175 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.175 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.175 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.175 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.175 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.175 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.175 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.175 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.176 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.176 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.176 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.176 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.176 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.176 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.176 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.176 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.176 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.176 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.176 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.176 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.176 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.176 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.177 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.177 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.177 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.177 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.177 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.177 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.177 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.177 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.177 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.177 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.177 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.177 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.178 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.178 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.178 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.178 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.178 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.178 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.178 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2024-04-26T18:12:45.178 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.179 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.179 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.179 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2024-04-26T18:12:45.179 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2024-04-26T18:12:45.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.179 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.179 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.179 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2024-04-26T18:12:45.179 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.179 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.179 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.179 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2024-04-26T18:12:45.179 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T18:12:45.179 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.180 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.180 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.180 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.180 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.180 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.180 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.180 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.180 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.180 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.180 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.180 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.180 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.180 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.181 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.181 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.181 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.181 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.181 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.181 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.181 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.181 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.181 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.181 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.181 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.181 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.182 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.182 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.182 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.182 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.182 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.182 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.182 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.182 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.182 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.182 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.182 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.182 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.183 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.183 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.183 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.183 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.183 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.183 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.183 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.183 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.183 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.183 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.183 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.183 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.183 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.183 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.185 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.185 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.185 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2024-04-26T18:12:45.185 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2024-04-26T18:12:45.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.185 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.185 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.185 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.185 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.185 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.185 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.185 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2024-04-26T18:12:45.185 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T18:12:45.186 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.186 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.186 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2024-04-26T18:12:45.186 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2024-04-26T18:12:45.186 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.186 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.186 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.186 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-04-26T18:12:45.186 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.186 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.186 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.186 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2024-04-26T18:12:45.186 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.187 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.187 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.187 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2024-04-26T18:12:45.187 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2024-04-26T18:12:45.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.187 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.187 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.187 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2024-04-26T18:12:45.187 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.187 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.187 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.187 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T18:12:45.188 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.188 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.188 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.188 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2024-04-26T18:12:45.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2024-04-26T18:12:45.188 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.188 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.188 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.188 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.188 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.188 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.188 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.188 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2024-04-26T18:12:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2024-04-26T18:12:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2024-04-26T18:12:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2024-04-26T18:12:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T18:12:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T18:12:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2024-04-26T18:12:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.196 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2024-04-26T18:12:45.196 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.196 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.196 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.196 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.196 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.196 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.196 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.196 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.196 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.196 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.196 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.196 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.196 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.197 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.198 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.198 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.198 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.198 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.198 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.198 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.198 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.198 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.198 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.198 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.198 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.198 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.200 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.200 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.200 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.200 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.200 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.200 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.200 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.200 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.200 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.200 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.200 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.200 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.200 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.201 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.201 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.201 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.201 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.201 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.201 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2024-04-26T18:12:45.201 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.201 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.201 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.201 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.201 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.202 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.202 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.202 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.202 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.202 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.202 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.202 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.202 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.202 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.202 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.202 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.202 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.202 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.202 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.203 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.203 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.203 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.203 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.203 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.203 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.203 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.203 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.203 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.203 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.203 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-26T18:12:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2024-04-26T18:12:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2024-04-26T18:12:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T18:12:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.210 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.210 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.210 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.210 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.210 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.210 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.210 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.210 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.210 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.210 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.210 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.210 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.212 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.212 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.212 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.212 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.212 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.212 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.212 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.212 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.212 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.212 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.212 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.212 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.212 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.213 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.213 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.213 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.213 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.213 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.213 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.213 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.213 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.213 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.213 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.213 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.213 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.213 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2024-04-26T18:12:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-26T18:12:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-04-26T18:12:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.217 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.217 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.217 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.217 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.217 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.217 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.217 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.217 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.217 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.217 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.217 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.217 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.218 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.218 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.218 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.218 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.218 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.218 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.218 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.218 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.218 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.218 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.218 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.218 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.219 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.219 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.219 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.219 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.219 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.219 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.219 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.219 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.219 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.219 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.219 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.219 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.219 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.220 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.220 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.220 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.220 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.220 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.220 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.220 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.220 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.220 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.220 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.220 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.220 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.220 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.221 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.221 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.221 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2024-04-26T18:12:45.221 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.221 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.221 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.221 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.221 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.221 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.221 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.221 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.221 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.222 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.222 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.222 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.222 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.222 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.222 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.222 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.222 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.222 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.222 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.222 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.222 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.222 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.224 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.224 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.224 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.224 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.224 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.224 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.224 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.224 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.224 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.224 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.224 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.224 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.225 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.225 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.225 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.225 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.225 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.225 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.225 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.225 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.225 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.225 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.225 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.225 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.225 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.226 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.226 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.226 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.226 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.226 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.226 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.226 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.226 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.226 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.226 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2024-04-26T18:12:45.227 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.227 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.227 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.227 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.227 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.227 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.227 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.227 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.227 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.227 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.227 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.227 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.227 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.228 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.228 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.228 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.228 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.228 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.228 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.228 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.228 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.228 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.228 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.228 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.228 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.228 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.229 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.230 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.230 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.230 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.230 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.230 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.230 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.230 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.230 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.230 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.230 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.230 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.230 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.230 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.231 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.232 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.232 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.232 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:45.232 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:45.232 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.232 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.232 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.232 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:45.232 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.232 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.232 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.232 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.232 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.233 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.233 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.233 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2024-04-26T18:12:45.233 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2024-04-26T18:12:45.233 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:45.233 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.233 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.233 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-26T18:12:45.233 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2024-04-26T18:12:45.233 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.233 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.233 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2024-04-26T18:12:45.233 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T18:12:45.234 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.234 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.234 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.234 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.234 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2024-04-26T18:12:45.234 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.234 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.234 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.234 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2024-04-26T18:12:45.234 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2024-04-26T18:12:45.234 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.234 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.235 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.235 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.235 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.235 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.235 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.235 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T18:12:45.235 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.235 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.235 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.235 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-26T18:12:45.235 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-26T18:12:45.235 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.236 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.236 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.236 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.236 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.236 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.236 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.236 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.236 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.236 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.236 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.236 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.236 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.237 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.237 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.237 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.237 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.237 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.237 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.237 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.237 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.237 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.237 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.237 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.237 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.237 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.237 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.238 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.238 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.238 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.238 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.238 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.238 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.238 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.238 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.238 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.238 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.238 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.238 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.239 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.239 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.239 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.239 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.239 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.239 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.239 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.239 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.239 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.239 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.239 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.239 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.239 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.240 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.240 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.240 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.240 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.240 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.240 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.240 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.240 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.240 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.240 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.240 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.240 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.240 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2024-04-26T18:12:45.241 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2024-04-26T18:12:45.242 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.242 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.242 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.242 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2024-04-26T18:12:45.242 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.242 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.242 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.242 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2024-04-26T18:12:45.242 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.242 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.242 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.242 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.244 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2024-04-26T18:12:45.244 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.244 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.244 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.244 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2024-04-26T18:12:45.244 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2024-04-26T18:12:45.244 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.244 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.244 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.244 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.244 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.244 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: "exclude_perf_counters": { 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "exclude_perf_counters", 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.245 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.246 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T18:12:45.246 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T18:12:45.246 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.246 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.246 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.246 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.246 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.246 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.246 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.246 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.246 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.246 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.247 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.247 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.247 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.247 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.247 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.247 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.247 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.247 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.247 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.247 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.247 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.247 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.247 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.248 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.248 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.248 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.248 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.248 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.248 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.248 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.248 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.248 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.248 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.248 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.248 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.248 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.249 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.249 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.249 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.249 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.249 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.249 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.249 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.249 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.249 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.249 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.249 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.249 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.249 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.249 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.250 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.250 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.250 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.250 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.250 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.250 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.250 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.250 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.250 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.250 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.250 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.250 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.251 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.251 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.251 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.251 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.251 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.251 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.251 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2024-04-26T18:12:45.251 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2024-04-26T18:12:45.251 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.251 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.251 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.251 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.251 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.252 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.252 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.252 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.252 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.252 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.252 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T18:12:45.252 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T18:12:45.252 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.252 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.252 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.252 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2024-04-26T18:12:45.252 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.253 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.253 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.253 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.253 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.253 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.253 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.253 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2024-04-26T18:12:45.253 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2024-04-26T18:12:45.253 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:45.253 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.253 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.253 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2024-04-26T18:12:45.254 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.254 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.254 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.254 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.254 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.254 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.254 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.254 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-26T18:12:45.254 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-26T18:12:45.254 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.254 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.254 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-26T18:12:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T18:12:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-26T18:12:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-26T18:12:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.256 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2024-04-26T18:12:45.256 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.256 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.256 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.256 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T18:12:45.256 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.256 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.256 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.256 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2024-04-26T18:12:45.256 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2024-04-26T18:12:45.256 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.256 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.256 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.257 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2024-04-26T18:12:45.257 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.257 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.257 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.257 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.257 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.257 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.257 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.257 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-26T18:12:45.257 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-26T18:12:45.257 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.257 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.257 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.258 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2024-04-26T18:12:45.258 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.258 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.258 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.258 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2024-04-26T18:12:45.258 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2024-04-26T18:12:45.258 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.258 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.258 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.258 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.258 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.258 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-26T18:12:45.258 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-26T18:12:45.259 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.259 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.259 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.259 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-26T18:12:45.259 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-26T18:12:45.259 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-26T18:12:45.259 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.259 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.259 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.259 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.259 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.259 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.259 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.260 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-04-26T18:12:45.260 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.260 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.260 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.260 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.260 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.260 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.260 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.260 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.260 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.260 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.260 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.261 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.261 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.261 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.261 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.261 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.261 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.261 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.261 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.261 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.261 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.261 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.261 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.261 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.262 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.262 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.262 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.262 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.262 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.262 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.262 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.262 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.262 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.262 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.262 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.262 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.263 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.263 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.263 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.263 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.263 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.263 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.263 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.263 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.263 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.263 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.263 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.263 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.263 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.264 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.264 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.264 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.264 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.264 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.264 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.264 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.264 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.264 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.264 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.264 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2024-04-26T18:12:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2024-04-26T18:12:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-26T18:12:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2024-04-26T18:12:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T18:12:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2024-04-26T18:12:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2024-04-26T18:12:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2024-04-26T18:12:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2024-04-26T18:12:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2024-04-26T18:12:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-26T18:12:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-26T18:12:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.271 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.271 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.271 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.271 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.271 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.271 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.271 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.271 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.271 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.271 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.271 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.271 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.271 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.272 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.272 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.272 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.272 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.272 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.272 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.272 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.272 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.272 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.272 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.272 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.272 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.272 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.273 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.273 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.273 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.273 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.273 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.273 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.273 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.273 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.273 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.273 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.273 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.273 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.273 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.274 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.274 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.274 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.274 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.274 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.274 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.274 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.274 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.274 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.274 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.274 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.274 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.274 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.275 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.275 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.275 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.275 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.275 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.275 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.275 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.275 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.275 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.275 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.275 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.275 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.275 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-26T18:12:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-26T18:12:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.277 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.277 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.277 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.277 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-26T18:12:45.277 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-26T18:12:45.277 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.277 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.277 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.277 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.277 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.277 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.277 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.277 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.278 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.278 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.278 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.278 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.278 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.278 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2024-04-26T18:12:45.278 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.278 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.278 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.278 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.278 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.279 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.279 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.279 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.279 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.279 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.279 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.279 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.279 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.279 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.279 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.279 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.279 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.279 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.279 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.280 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.281 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.281 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.281 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.281 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.281 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.281 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.281 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.281 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.281 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.281 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.281 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.281 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.281 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.282 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.282 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.282 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.282 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.282 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.282 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.282 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.282 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.282 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.282 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.282 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.282 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.282 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.283 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2024-04-26T18:12:45.284 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.284 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.284 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.284 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.284 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.284 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.284 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.284 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.284 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.284 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.284 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.284 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.284 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.284 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.285 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.285 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.285 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.285 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.285 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.285 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.285 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.285 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.285 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.285 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.285 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.285 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.285 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.286 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.286 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.286 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.286 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.286 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.286 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.286 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.286 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.286 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.286 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.286 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.286 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.286 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.287 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.287 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.287 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.287 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.287 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.287 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.287 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.287 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.287 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.287 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.287 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.287 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.287 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.287 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.288 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.288 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.288 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.288 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.288 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.288 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.288 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.288 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.288 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.288 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.288 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.288 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.288 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.289 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.290 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.290 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.290 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.290 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2024-04-26T18:12:45.290 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.290 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.290 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.290 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.290 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.290 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.290 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.291 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.291 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.291 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.291 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.291 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.291 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.291 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.291 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.291 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.291 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.291 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.291 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.291 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.291 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.292 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.292 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.292 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.292 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.292 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.292 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.292 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.292 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.292 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.292 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.292 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.292 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.292 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.294 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.294 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.294 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.294 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.294 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.294 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.294 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.294 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.294 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.294 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.294 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.294 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.295 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.295 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.295 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.295 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.295 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.295 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.295 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.295 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.295 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.295 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.295 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2024-04-26T18:12:45.295 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2024-04-26T18:12:45.295 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.296 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.296 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.296 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.296 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.296 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.296 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.296 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.296 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.296 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.296 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.296 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2024-04-26T18:12:45.296 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2024-04-26T18:12:45.296 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2024-04-26T18:12:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2024-04-26T18:12:45.297 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2024-04-26T18:12:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2024-04-26T18:12:45.298 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2024-04-26T18:12:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2024-04-26T18:12:45.299 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2024-04-26T18:12:45.300 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.301 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.301 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.301 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.301 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.301 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.301 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.301 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.301 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.301 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.301 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.301 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2024-04-26T18:12:45.301 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2024-04-26T18:12:45.302 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.302 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.302 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.302 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.302 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.302 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.302 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.302 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.302 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.302 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.302 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.302 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2024-04-26T18:12:45.302 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2024-04-26T18:12:45.303 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.303 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.303 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.303 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.303 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.303 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.303 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.303 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.303 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.303 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.303 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.303 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2024-04-26T18:12:45.303 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2024-04-26T18:12:45.304 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.304 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.304 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.304 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.304 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-26T18:12:45.304 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2024-04-26T18:12:45.304 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.304 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.304 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.304 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.304 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.304 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2024-04-26T18:12:45.305 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2024-04-26T18:12:45.305 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.305 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.305 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.305 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.305 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.305 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.305 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.305 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.305 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.305 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.305 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.305 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2024-04-26T18:12:45.306 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2024-04-26T18:12:45.307 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.307 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.307 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.307 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.307 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.307 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.307 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.307 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.307 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.307 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.307 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.307 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.307 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.308 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-04-26T18:12:45.308 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.308 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.308 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.308 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2024-04-26T18:12:45.308 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2024-04-26T18:12:45.308 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.308 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.308 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.308 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.308 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.308 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.309 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.309 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T18:12:45.309 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.309 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.309 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.309 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2024-04-26T18:12:45.309 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2024-04-26T18:12:45.309 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.309 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.309 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.309 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.309 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.309 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.310 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.310 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2024-04-26T18:12:45.310 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.310 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.310 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.310 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.310 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.310 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.310 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.310 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.310 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.310 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.310 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.311 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.312 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.312 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.312 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.312 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.312 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.312 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.312 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.312 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.312 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.312 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.312 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.312 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.312 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.313 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.313 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.313 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.313 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.313 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.313 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.313 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.313 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.313 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.313 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.313 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.313 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.313 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.313 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.314 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.314 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.314 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.314 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.314 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.314 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.314 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.314 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.314 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.314 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.314 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.314 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.314 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.315 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.315 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.315 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.315 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.315 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.315 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.315 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2024-04-26T18:12:45.315 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.315 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.315 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.315 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.317 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.317 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.317 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.317 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.317 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.317 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.317 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.317 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.317 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.317 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.319 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.319 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.319 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.319 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.319 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.319 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.319 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.319 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.319 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.319 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.319 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.319 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.319 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.320 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.320 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.320 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.320 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.320 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.320 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.320 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.320 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.320 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.320 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.320 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.320 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.320 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.321 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.321 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.321 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.321 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.321 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.321 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.321 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.321 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.321 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.321 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.321 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.321 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.321 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2024-04-26T18:12:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.323 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.323 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.323 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.323 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.323 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.323 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.323 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.323 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.323 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.323 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.323 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.323 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.323 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.323 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.324 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.324 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.324 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.324 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.324 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.324 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.324 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.324 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.324 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.324 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.324 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.324 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.324 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.325 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.325 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.325 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.325 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.325 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.325 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.325 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.325 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.325 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.325 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.325 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.325 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.325 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.326 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.326 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.326 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.326 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.326 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.326 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.326 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.326 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.326 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.326 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.326 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.326 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.326 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.327 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.327 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.327 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.327 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.327 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.327 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.327 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.327 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.327 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.327 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.327 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.327 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.328 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2024-04-26T18:12:45.328 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.328 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.328 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.328 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2024-04-26T18:12:45.328 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2024-04-26T18:12:45.328 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.328 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.328 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.328 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T18:12:45.328 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.328 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.328 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.329 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.330 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.330 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.330 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.330 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.330 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.330 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.330 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.330 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.330 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.330 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.330 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.330 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.332 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.332 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.332 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.332 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.332 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.332 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.332 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.332 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.332 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.332 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.332 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.332 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.332 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.334 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.334 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.334 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.334 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.334 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.334 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.334 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.334 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.334 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.334 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.334 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.334 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2024-04-26T18:12:45.335 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.336 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.336 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.336 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.336 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.336 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.336 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.336 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T18:12:45.336 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.336 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.336 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.336 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2024-04-26T18:12:45.336 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2024-04-26T18:12:45.337 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.337 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.337 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.337 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.337 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.337 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.337 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.337 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T18:12:45.337 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.337 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.337 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.337 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2024-04-26T18:12:45.337 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2024-04-26T18:12:45.338 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.338 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.338 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.338 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.338 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.338 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.338 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.338 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T18:12:45.338 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.338 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.338 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.338 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2024-04-26T18:12:45.338 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2024-04-26T18:12:45.339 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.339 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.339 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.339 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.339 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.339 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.339 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.339 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T18:12:45.339 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.339 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.339 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.339 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2024-04-26T18:12:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2024-04-26T18:12:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T18:12:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2024-04-26T18:12:45.341 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2024-04-26T18:12:45.341 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.341 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.341 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.341 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.341 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.341 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.341 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.341 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.341 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.341 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.341 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2024-04-26T18:12:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2024-04-26T18:12:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2024-04-26T18:12:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2024-04-26T18:12:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T18:12:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-26T18:12:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-26T18:12:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.344 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2024-04-26T18:12:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2024-04-26T18:12:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.345 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2024-04-26T18:12:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2024-04-26T18:12:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2024-04-26T18:12:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.346 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2024-04-26T18:12:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2024-04-26T18:12:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.347 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2024-04-26T18:12:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2024-04-26T18:12:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.348 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.349 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.350 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.351 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.352 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.353 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2024-04-26T18:12:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2024-04-26T18:12:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.354 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2024-04-26T18:12:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2024-04-26T18:12:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2024-04-26T18:12:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2024-04-26T18:12:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T18:12:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.356 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2024-04-26T18:12:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.364 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.364 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.364 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.364 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.364 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.364 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.364 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.364 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.364 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.364 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.364 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.364 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.364 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.365 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.365 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.365 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.365 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.365 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.365 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.365 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.365 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.365 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.365 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.365 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.365 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.365 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.366 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.366 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.366 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.366 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.366 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.366 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.366 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.366 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.366 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.366 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.366 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.366 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.366 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.367 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.367 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.367 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.367 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.367 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.367 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.367 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.367 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.367 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.367 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.367 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.367 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2024-04-26T18:12:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2024-04-26T18:12:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.368 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.369 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.369 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2024-04-26T18:12:45.369 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.369 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.369 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.369 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T18:12:45.369 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.369 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.369 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.369 INFO:teuthology.orchestra.run.smithi031.stdout: "periodic_async_work": { 2024-04-26T18:12:45.369 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "periodic_async_work", 2024-04-26T18:12:45.369 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.370 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.370 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.370 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.370 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.370 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.370 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.370 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Periodically check for async work", 2024-04-26T18:12:45.370 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.370 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.370 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.370 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2024-04-26T18:12:45.370 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2024-04-26T18:12:45.370 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.371 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.371 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.371 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-26T18:12:45.371 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.371 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.371 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.371 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T18:12:45.371 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.371 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.371 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.371 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_no_wait": { 2024-04-26T18:12:45.371 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T18:12:45.371 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.372 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.372 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.372 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-26T18:12:45.372 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.372 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.372 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.372 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T18:12:45.372 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.372 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.372 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.372 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.372 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.373 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2024-04-26T18:12:45.373 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-26T18:12:45.373 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-26T18:12:45.373 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-26T18:12:45.373 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2024-04-26T18:12:45.373 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2024-04-26T18:12:45.373 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-26T18:12:45.373 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.373 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.373 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2024-04-26T18:12:45.373 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.373 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.373 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.373 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.374 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.374 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.374 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.374 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2024-04-26T18:12:45.374 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2024-04-26T18:12:45.374 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.374 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.374 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.374 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.374 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.374 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.374 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.375 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.375 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.375 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.375 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.375 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-26T18:12:45.375 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-26T18:12:45.375 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-26T18:12:45.375 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.375 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.375 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-26T18:12:45.375 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-26T18:12:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-26T18:12:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.377 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.377 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.377 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.377 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.377 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.377 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.377 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.377 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.377 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.377 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.377 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.377 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.377 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.378 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.378 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-26T18:12:45.378 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-26T18:12:45.378 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.378 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.378 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.378 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.378 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.378 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.378 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.378 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.378 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.379 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.379 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.379 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-26T18:12:45.379 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-26T18:12:45.379 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.379 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.379 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.379 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-26T18:12:45.379 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.379 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.379 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-26T18:12:45.379 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-26T18:12:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-26T18:12:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-26T18:12:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-26T18:12:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-26T18:12:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-26T18:12:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-26T18:12:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-26T18:12:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-26T18:12:45.381 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.381 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-26T18:12:45.381 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-26T18:12:45.381 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.381 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.381 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.381 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.381 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.381 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.381 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.381 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2024-04-26T18:12:45.381 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2024-04-26T18:12:45.381 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.382 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.382 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.382 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-26T18:12:45.382 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.382 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.382 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.382 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.382 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.382 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.382 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.382 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2024-04-26T18:12:45.383 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2024-04-26T18:12:45.383 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-26T18:12:45.383 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.383 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.383 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2024-04-26T18:12:45.383 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.383 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.383 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.383 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.383 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.383 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.383 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.383 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2024-04-26T18:12:45.384 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2024-04-26T18:12:45.384 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-26T18:12:45.384 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-26T18:12:45.384 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-26T18:12:45.384 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T18:12:45.384 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-26T18:12:45.384 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-26T18:12:45.384 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-26T18:12:45.384 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-26T18:12:45.384 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-26T18:12:45.384 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-26T18:12:45.384 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-26T18:12:45.384 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.384 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.385 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.385 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.385 INFO:teuthology.orchestra.run.smithi031.stdout: "services": { 2024-04-26T18:12:45.385 INFO:teuthology.orchestra.run.smithi031.stdout: "dashboard": "https://172.21.15.31:8443/" 2024-04-26T18:12:45.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.385 INFO:teuthology.orchestra.run.smithi031.stdout: "always_on_modules": { 2024-04-26T18:12:45.385 INFO:teuthology.orchestra.run.smithi031.stdout: "octopus": [ 2024-04-26T18:12:45.385 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-26T18:12:45.385 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-26T18:12:45.385 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-26T18:12:45.385 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-26T18:12:45.385 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-26T18:12:45.385 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-26T18:12:45.385 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-26T18:12:45.386 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-26T18:12:45.386 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-26T18:12:45.386 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-26T18:12:45.386 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.386 INFO:teuthology.orchestra.run.smithi031.stdout: "pacific": [ 2024-04-26T18:12:45.386 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-26T18:12:45.386 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-26T18:12:45.386 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-26T18:12:45.386 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-26T18:12:45.386 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-26T18:12:45.386 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-26T18:12:45.386 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-26T18:12:45.386 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-26T18:12:45.387 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-26T18:12:45.387 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-26T18:12:45.387 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.387 INFO:teuthology.orchestra.run.smithi031.stdout: "quincy": [ 2024-04-26T18:12:45.387 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-26T18:12:45.387 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-26T18:12:45.387 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-26T18:12:45.387 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-26T18:12:45.387 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-26T18:12:45.387 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-26T18:12:45.387 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-26T18:12:45.387 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-26T18:12:45.387 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-26T18:12:45.388 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-26T18:12:45.388 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.388 INFO:teuthology.orchestra.run.smithi031.stdout: "reef": [ 2024-04-26T18:12:45.388 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-26T18:12:45.388 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-26T18:12:45.388 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-26T18:12:45.388 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-26T18:12:45.388 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-26T18:12:45.388 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-26T18:12:45.388 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-26T18:12:45.388 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-26T18:12:45.388 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-26T18:12:45.388 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-26T18:12:45.388 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-26T18:12:45.389 INFO:teuthology.orchestra.run.smithi031.stdout: "squid": [ 2024-04-26T18:12:45.389 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-26T18:12:45.389 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-26T18:12:45.389 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-26T18:12:45.389 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-26T18:12:45.389 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-26T18:12:45.389 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-26T18:12:45.389 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-26T18:12:45.389 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-26T18:12:45.389 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-26T18:12:45.389 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-26T18:12:45.389 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-26T18:12:45.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.389 INFO:teuthology.orchestra.run.smithi031.stdout: "last_failure_osd_epoch": 91, 2024-04-26T18:12:45.390 INFO:teuthology.orchestra.run.smithi031.stdout: "active_clients": [ 2024-04-26T18:12:45.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.390 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-04-26T18:12:45.390 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-26T18:12:45.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.390 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-26T18:12:45.390 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-26T18:12:45.390 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1100642757 2024-04-26T18:12:45.390 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.390 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-26T18:12:45.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.390 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "libcephsqlite", 2024-04-26T18:12:45.390 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-26T18:12:45.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.391 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-26T18:12:45.391 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-26T18:12:45.391 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045273897 2024-04-26T18:12:45.391 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.391 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-26T18:12:45.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.391 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-04-26T18:12:45.391 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-26T18:12:45.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.391 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-26T18:12:45.391 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-26T18:12:45.392 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1453836499 2024-04-26T18:12:45.392 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.392 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-26T18:12:45.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.392 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-04-26T18:12:45.392 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-26T18:12:45.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.392 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-26T18:12:45.392 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-26T18:12:45.392 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3784632922 2024-04-26T18:12:45.392 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.393 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-26T18:12:45.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.393 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-04-26T18:12:45.393 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-26T18:12:45.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.393 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-26T18:12:45.393 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-26T18:12:45.393 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1539336114 2024-04-26T18:12:45.393 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.393 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-26T18:12:45.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:45.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.393 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-04-26T18:12:45.394 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-26T18:12:45.394 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-26T18:12:45.394 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-26T18:12:45.394 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-26T18:12:45.394 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3539645867 2024-04-26T18:12:45.394 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.394 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-26T18:12:45.394 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:45.394 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-26T18:12:45.394 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T18:12:45.396 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:44 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3887222010' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T18:12:45.396 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:44 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3465760541' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T18:12:45.396 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:44 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1375474921' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T18:12:45.396 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:44 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3887222010' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T18:12:45.396 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:44 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3465760541' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T18:12:45.396 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:44 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1375474921' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T18:12:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.993+0000 7f3174c25640 1 -- 172.21.15.31:0/1375474921 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f31580a3d20 msgr2=0x7f31580a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.993+0000 7f3174c25640 1 --2- 172.21.15.31:0/1375474921 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f31580a3d20 0x7f31580a61e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3160009900 tx=0x7f3160000bc0 comp rx=0 tx=0).stop 2024-04-26T18:12:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.993+0000 7f3174c25640 1 -- 172.21.15.31:0/1375474921 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31701021d0 msgr2=0x7f3170199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:12:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.993+0000 7f3174c25640 1 --2- 172.21.15.31:0/1375474921 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31701021d0 0x7f3170199670 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f316400c890 tx=0x7f316400cd50 comp rx=0 tx=0).stop 2024-04-26T18:12:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.993+0000 7f3174c25640 1 -- 172.21.15.31:0/1375474921 shutdown_connections 2024-04-26T18:12:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.993+0000 7f3174c25640 1 --2- 172.21.15.31:0/1375474921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3170107820 0x7f317019d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.993+0000 7f3174c25640 1 --2- 172.21.15.31:0/1375474921 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f31580a3d20 0x7f31580a61e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.993+0000 7f3174c25640 1 --2- 172.21.15.31:0/1375474921 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31701021d0 0x7f3170199670 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.993+0000 7f3174c25640 1 --2- 172.21.15.31:0/1375474921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3170101810 0x7f3170199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:12:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.993+0000 7f3174c25640 1 -- 172.21.15.31:0/1375474921 >> 172.21.15.31:0/1375474921 conn(0x7f3170078930 msgr2=0x7f3170069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:12:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.994+0000 7f3174c25640 1 -- 172.21.15.31:0/1375474921 shutdown_connections 2024-04-26T18:12:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:12:43.994+0000 7f3174c25640 1 -- 172.21.15.31:0/1375474921 wait complete. 2024-04-26T18:12:45.404 DEBUG:tasks.check_counter:Getting stats from a 2024-04-26T18:12:45.404 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-26T18:12:45.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:44 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3887222010' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T18:12:45.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:44 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3465760541' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T18:12:45.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:44 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1375474921' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T18:12:45.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:45 smithi186 ceph-mon[32797]: pgmap v510: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 10 MiB/s wr, 649 op/s 2024-04-26T18:12:45.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:45 smithi031 ceph-mon[29164]: pgmap v510: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 10 MiB/s wr, 649 op/s 2024-04-26T18:12:45.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:45 smithi097 ceph-mon[32419]: pgmap v510: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 10 MiB/s wr, 649 op/s 2024-04-26T18:12:45.668 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:12:47.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:47 smithi031 ceph-mon[29164]: pgmap v511: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.8 MiB/s wr, 123 op/s 2024-04-26T18:12:47.302 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T18:12:47.302 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T18:12:47.302 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 0, 2024-04-26T18:12:47.302 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 2, 2024-04-26T18:12:47.302 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 0, 2024-04-26T18:12:47.302 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 0, 2024-04-26T18:12:47.303 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2024-04-26T18:12:47.303 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 0, 2024-04-26T18:12:47.303 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.001609003, 2024-04-26T18:12:47.303 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T18:12:47.303 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.000366501, 2024-04-26T18:12:47.303 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T18:12:47.303 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:47.303 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:47.303 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:47.303 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:47.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:47.303 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:47.303 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:47.304 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:47.304 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:47.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:47.304 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T18:12:47.304 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T18:12:47.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:47.304 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T18:12:47.304 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 3, 2024-04-26T18:12:47.304 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 245, 2024-04-26T18:12:47.304 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 480, 2024-04-26T18:12:47.304 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 278848, 2024-04-26T18:12:47.304 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 6, 2024-04-26T18:12:47.304 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 3, 2024-04-26T18:12:47.304 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.018095581, 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.012751960, 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.002185781, 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 242, 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.028937422, 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000119576 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T18:12:47.305 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 278848 2024-04-26T18:12:47.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:47.306 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T18:12:47.306 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 334, 2024-04-26T18:12:47.306 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 321, 2024-04-26T18:12:47.306 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1623440, 2024-04-26T18:12:47.306 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 197488, 2024-04-26T18:12:47.306 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 2, 2024-04-26T18:12:47.306 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 1, 2024-04-26T18:12:47.306 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.060645125, 2024-04-26T18:12:47.306 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.021487404, 2024-04-26T18:12:47.306 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.039815969, 2024-04-26T18:12:47.306 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.007084705, 2024-04-26T18:12:47.306 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 321, 2024-04-26T18:12:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.019909757, 2024-04-26T18:12:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000062024 2024-04-26T18:12:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 1623440, 2024-04-26T18:12:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 197488 2024-04-26T18:12:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-04-26T18:12:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 0, 2024-04-26T18:12:47.308 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-04-26T18:12:47.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:47.308 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-04-26T18:12:47.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-04-26T18:12:47.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-04-26T18:12:47.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T18:12:47.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-04-26T18:12:47.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T18:12:47.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-04-26T18:12:47.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T18:12:47.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T18:12:47.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T18:12:47.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T18:12:47.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T18:12:47.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-04-26T18:12:47.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T18:12:47.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T18:12:47.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-04-26T18:12:47.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-04-26T18:12:47.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-04-26T18:12:47.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-04-26T18:12:47.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T18:12:47.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T18:12:47.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T18:12:47.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T18:12:47.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T18:12:47.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-04-26T18:12:47.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-04-26T18:12:47.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-04-26T18:12:47.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T18:12:47.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T18:12:47.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-04-26T18:12:47.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-04-26T18:12:47.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-04-26T18:12:47.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-04-26T18:12:47.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 147970, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 29, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 0, 2024-04-26T18:12:47.311 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 0, 2024-04-26T18:12:47.312 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T18:12:47.312 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-04-26T18:12:47.312 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-04-26T18:12:47.312 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-04-26T18:12:47.312 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 928, 2024-04-26T18:12:47.312 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 2, 2024-04-26T18:12:47.312 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-04-26T18:12:47.312 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-04-26T18:12:47.312 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-04-26T18:12:47.312 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-04-26T18:12:47.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:47.312 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T18:12:47.312 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-26T18:12:47.312 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 333, 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 1573932, 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 333, 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 333, 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1573932, 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:47.313 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:47.314 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T18:12:47.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:47 smithi186 ceph-mon[32797]: pgmap v511: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.8 MiB/s wr, 123 op/s 2024-04-26T18:12:47.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:47 smithi097 ceph-mon[32419]: pgmap v511: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.8 MiB/s wr, 123 op/s 2024-04-26T18:12:47.950 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-26T18:12:47.950 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-26T18:12:47.950 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-26T18:12:47.951 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-26T18:12:47.951 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-26T18:12:47.951 DEBUG:tasks.check_counter:Getting stats from d 2024-04-26T18:12:47.951 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-26T18:12:48.204 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:12:49.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:49 smithi186 ceph-mon[32797]: pgmap v512: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 1.8 MiB/s wr, 130 op/s 2024-04-26T18:12:49.553 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:49 smithi031 ceph-mon[29164]: pgmap v512: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 1.8 MiB/s wr, 130 op/s 2024-04-26T18:12:49.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:49 smithi097 ceph-mon[32419]: pgmap v512: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 1.8 MiB/s wr, 130 op/s 2024-04-26T18:12:49.798 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T18:12:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T18:12:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 0, 2024-04-26T18:12:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 2, 2024-04-26T18:12:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 0, 2024-04-26T18:12:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 0, 2024-04-26T18:12:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2024-04-26T18:12:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 0, 2024-04-26T18:12:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.001422195, 2024-04-26T18:12:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T18:12:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.000344330, 2024-04-26T18:12:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T18:12:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 337, 2024-04-26T18:12:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 326, 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1622400, 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 201520, 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 2, 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 1, 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.059225607, 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.020141068, 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.038759733, 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.007404940, 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 325, 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.024694599, 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000075983 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 1622400, 2024-04-26T18:12:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 201520 2024-04-26T18:12:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T18:12:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 7, 2024-04-26T18:12:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 252, 2024-04-26T18:12:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 3424, 2024-04-26T18:12:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 280176, 2024-04-26T18:12:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 6, 2024-04-26T18:12:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 3, 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.019278179, 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.013226279, 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.003033799, 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 250, 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.031261434, 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000125045 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 3424, 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 280176 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 0, 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T18:12:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T18:12:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T18:12:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-04-26T18:12:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-04-26T18:12:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-04-26T18:12:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T18:12:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T18:12:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-04-26T18:12:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-04-26T18:12:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-04-26T18:12:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-04-26T18:12:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T18:12:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-04-26T18:12:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T18:12:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-04-26T18:12:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 147987, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 30, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 0, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 0, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-04-26T18:12:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 928, 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 2, 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 340, 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 1574842, 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-26T18:12:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 340, 2024-04-26T18:12:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-26T18:12:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-26T18:12:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 340, 2024-04-26T18:12:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1574842, 2024-04-26T18:12:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-26T18:12:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:49.809 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T18:12:50.488 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-26T18:12:50.488 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-26T18:12:50.488 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-26T18:12:50.488 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-26T18:12:50.488 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-26T18:12:50.489 DEBUG:tasks.check_counter:Getting stats from g 2024-04-26T18:12:50.489 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-26T18:12:50.743 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:12:51.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:51 smithi186 ceph-mon[32797]: pgmap v513: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 13 op/s 2024-04-26T18:12:51.529 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:51 smithi031 ceph-mon[29164]: pgmap v513: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 13 op/s 2024-04-26T18:12:51.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:51 smithi097 ceph-mon[32419]: pgmap v513: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 13 op/s 2024-04-26T18:12:52.399 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T18:12:52.399 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T18:12:52.399 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 0, 2024-04-26T18:12:52.399 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 2, 2024-04-26T18:12:52.399 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 0, 2024-04-26T18:12:52.399 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 0, 2024-04-26T18:12:52.399 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2024-04-26T18:12:52.400 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 0, 2024-04-26T18:12:52.400 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.001342759, 2024-04-26T18:12:52.400 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T18:12:52.400 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.000231232, 2024-04-26T18:12:52.400 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T18:12:52.400 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:52.400 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:52.400 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:52.400 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:52.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:52.400 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:52.400 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:52.400 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:52.401 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:52.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:52.401 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T18:12:52.401 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T18:12:52.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:52.401 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T18:12:52.401 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 333, 2024-04-26T18:12:52.401 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 322, 2024-04-26T18:12:52.401 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1621312, 2024-04-26T18:12:52.401 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 198960, 2024-04-26T18:12:52.401 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 2, 2024-04-26T18:12:52.401 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 1, 2024-04-26T18:12:52.401 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.059157679, 2024-04-26T18:12:52.401 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.019774470, 2024-04-26T18:12:52.402 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.040203638, 2024-04-26T18:12:52.402 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.007212354, 2024-04-26T18:12:52.402 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:52.402 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 321, 2024-04-26T18:12:52.402 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.026489618, 2024-04-26T18:12:52.402 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000082522 2024-04-26T18:12:52.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:52.402 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:52.402 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:52.402 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:52.402 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:52.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:52.402 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 1621312, 2024-04-26T18:12:52.402 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 198960 2024-04-26T18:12:52.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:52.403 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T18:12:52.403 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 7, 2024-04-26T18:12:52.403 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 249, 2024-04-26T18:12:52.403 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 3424, 2024-04-26T18:12:52.403 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 279648, 2024-04-26T18:12:52.403 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 6, 2024-04-26T18:12:52.403 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 3, 2024-04-26T18:12:52.403 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.019279351, 2024-04-26T18:12:52.403 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.013691532, 2024-04-26T18:12:52.403 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.002497969, 2024-04-26T18:12:52.403 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T18:12:52.403 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:52.403 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 247, 2024-04-26T18:12:52.404 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.026359430, 2024-04-26T18:12:52.404 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000106718 2024-04-26T18:12:52.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:52.404 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:52.404 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:52.404 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:52.404 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:52.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:52.404 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 3424, 2024-04-26T18:12:52.404 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 279648 2024-04-26T18:12:52.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:52.404 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-04-26T18:12:52.404 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 0, 2024-04-26T18:12:52.405 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-04-26T18:12:52.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:52.405 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-04-26T18:12:52.405 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-04-26T18:12:52.405 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-04-26T18:12:52.405 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T18:12:52.405 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-04-26T18:12:52.405 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T18:12:52.405 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-04-26T18:12:52.405 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T18:12:52.405 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T18:12:52.405 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T18:12:52.405 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T18:12:52.406 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T18:12:52.406 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-04-26T18:12:52.406 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T18:12:52.406 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T18:12:52.406 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-04-26T18:12:52.406 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-04-26T18:12:52.406 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-04-26T18:12:52.406 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-04-26T18:12:52.406 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T18:12:52.406 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T18:12:52.406 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T18:12:52.406 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T18:12:52.406 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 147964, 2024-04-26T18:12:52.407 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 30, 2024-04-26T18:12:52.408 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-04-26T18:12:52.408 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-04-26T18:12:52.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-04-26T18:12:52.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-04-26T18:12:52.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-04-26T18:12:52.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-04-26T18:12:52.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-04-26T18:12:52.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-04-26T18:12:52.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 0, 2024-04-26T18:12:52.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 0, 2024-04-26T18:12:52.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T18:12:52.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-04-26T18:12:52.408 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-04-26T18:12:52.408 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-04-26T18:12:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 928, 2024-04-26T18:12:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 2, 2024-04-26T18:12:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-04-26T18:12:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-04-26T18:12:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-04-26T18:12:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-04-26T18:12:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T18:12:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-26T18:12:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-26T18:12:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-26T18:12:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 336, 2024-04-26T18:12:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 1574322, 2024-04-26T18:12:52.409 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-26T18:12:52.410 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 336, 2024-04-26T18:12:52.410 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-26T18:12:52.410 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-26T18:12:52.410 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 336, 2024-04-26T18:12:52.410 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1574322, 2024-04-26T18:12:52.410 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-26T18:12:52.410 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:52.410 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:52.410 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:52.410 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:52.410 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:52.410 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T18:12:53.096 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-26T18:12:53.096 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-26T18:12:53.097 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-26T18:12:53.097 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-26T18:12:53.097 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-26T18:12:53.097 DEBUG:tasks.check_counter:Getting stats from j 2024-04-26T18:12:53.097 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-26T18:12:53.348 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:12:53.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:53 smithi031 ceph-mon[29164]: pgmap v514: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 65 KiB/s wr, 18 op/s 2024-04-26T18:12:53.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:53 smithi186 ceph-mon[32797]: pgmap v514: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 65 KiB/s wr, 18 op/s 2024-04-26T18:12:53.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:53 smithi097 ceph-mon[32419]: pgmap v514: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 65 KiB/s wr, 18 op/s 2024-04-26T18:12:54.965 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:54 smithi031 ceph-mon[29164]: pgmap v515: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-04-26T18:12:54.965 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-26T18:12:54.966 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T18:12:54.966 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 4, 2024-04-26T18:12:54.966 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 4, 2024-04-26T18:12:54.966 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 2944, 2024-04-26T18:12:54.966 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 448, 2024-04-26T18:12:54.966 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2024-04-26T18:12:54.966 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 0, 2024-04-26T18:12:54.966 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.001617203, 2024-04-26T18:12:54.966 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.000032835, 2024-04-26T18:12:54.966 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.000578938, 2024-04-26T18:12:54.966 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 3, 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000720400, 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000240133 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T18:12:54.967 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 3, 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 251, 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 480, 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 279904, 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 6, 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 3, 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.019889516, 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.014614026, 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.002157427, 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 248, 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.019765112, 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000079698 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:54.968 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:54.969 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:54.969 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:54.969 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:54.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:54.969 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T18:12:54.969 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 279904 2024-04-26T18:12:54.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:54.969 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T18:12:54.969 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 339, 2024-04-26T18:12:54.969 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 328, 2024-04-26T18:12:54.969 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1622944, 2024-04-26T18:12:54.970 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 202800, 2024-04-26T18:12:54.970 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 2, 2024-04-26T18:12:54.970 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 1, 2024-04-26T18:12:54.970 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.062962430, 2024-04-26T18:12:54.970 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.020641703, 2024-04-26T18:12:54.970 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.043468393, 2024-04-26T18:12:54.970 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.007788069, 2024-04-26T18:12:54.970 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:54.970 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 327, 2024-04-26T18:12:54.970 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.016906524, 2024-04-26T18:12:54.970 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000051701 2024-04-26T18:12:54.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:54.970 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:54.971 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:54.971 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:54.971 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:54.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:54.971 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 1622944, 2024-04-26T18:12:54.971 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 202800 2024-04-26T18:12:54.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:54.971 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-04-26T18:12:54.971 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 0, 2024-04-26T18:12:54.971 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-04-26T18:12:54.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:54.971 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-04-26T18:12:54.971 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-04-26T18:12:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-04-26T18:12:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T18:12:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-04-26T18:12:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T18:12:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-04-26T18:12:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T18:12:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T18:12:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T18:12:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T18:12:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T18:12:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-04-26T18:12:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T18:12:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T18:12:54.972 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-04-26T18:12:54.973 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-04-26T18:12:54.973 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-04-26T18:12:54.973 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-04-26T18:12:54.973 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T18:12:54.973 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T18:12:54.973 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T18:12:54.973 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T18:12:54.973 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T18:12:54.973 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-04-26T18:12:54.973 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-04-26T18:12:54.973 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-04-26T18:12:54.973 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T18:12:54.973 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T18:12:54.973 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-04-26T18:12:54.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-04-26T18:12:54.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-04-26T18:12:54.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-04-26T18:12:54.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T18:12:54.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-04-26T18:12:54.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T18:12:54.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-04-26T18:12:54.974 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 147930, 2024-04-26T18:12:54.974 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 28, 2024-04-26T18:12:54.974 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-04-26T18:12:54.974 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-04-26T18:12:54.975 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-04-26T18:12:54.975 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-04-26T18:12:54.975 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-04-26T18:12:54.975 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-04-26T18:12:54.975 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-04-26T18:12:54.975 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-04-26T18:12:54.975 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 0, 2024-04-26T18:12:54.975 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 0, 2024-04-26T18:12:54.975 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T18:12:54.975 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-04-26T18:12:54.975 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-04-26T18:12:54.975 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-04-26T18:12:54.975 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 928, 2024-04-26T18:12:54.975 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 2, 2024-04-26T18:12:54.976 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-04-26T18:12:54.976 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-04-26T18:12:54.976 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-04-26T18:12:54.976 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-04-26T18:12:54.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-26T18:12:54.976 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T18:12:54.976 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-26T18:12:54.976 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-26T18:12:54.976 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-26T18:12:54.976 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 342, 2024-04-26T18:12:54.976 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 1575102, 2024-04-26T18:12:54.976 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-26T18:12:54.976 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 342, 2024-04-26T18:12:54.976 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-26T18:12:54.977 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-26T18:12:54.977 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 342, 2024-04-26T18:12:54.977 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1575102, 2024-04-26T18:12:54.977 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-26T18:12:54.977 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-26T18:12:54.977 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-26T18:12:54.977 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-26T18:12:54.977 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:54.977 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-26T18:12:54.977 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-26T18:12:55.134 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:54 smithi097 ceph-mon[32419]: pgmap v515: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-04-26T18:12:55.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:54 smithi186 ceph-mon[32797]: pgmap v515: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-04-26T18:12:55.587 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-26T18:12:55.588 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-26T18:12:55.588 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-26T18:12:55.588 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-26T18:12:55.588 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-26T18:12:55.588 DEBUG:tasks.check_counter:Getting stats from b 2024-04-26T18:12:55.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-26T18:12:55.839 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T18:12:57.293 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:57 smithi097 ceph-mon[32419]: pgmap v516: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-04-26T18:12:57.294 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:57 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:12:57.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:57 smithi186 ceph-mon[32797]: pgmap v516: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-04-26T18:12:57.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:57 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:12:57.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:57 smithi031 ceph-mon[29164]: pgmap v516: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-04-26T18:12:57.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:57 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T18:12:57.787 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-26T18:12:57.787 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T18:12:57.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 1687, 2024-04-26T18:12:57.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 2297, 2024-04-26T18:12:57.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 867236, 2024-04-26T18:12:57.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 1740146, 2024-04-26T18:12:57.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 14, 2024-04-26T18:12:57.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 1, 2024-04-26T18:12:57.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.141135178, 2024-04-26T18:12:57.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.053756297, 2024-04-26T18:12:57.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.087152507, 2024-04-26T18:12:57.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.006921659, 2024-04-26T18:12:57.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:57.788 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 2295, 2024-04-26T18:12:57.788 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.104606588, 2024-04-26T18:12:57.789 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000045580 2024-04-26T18:12:57.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.789 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:57.789 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 3221, 2024-04-26T18:12:57.789 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000526457, 2024-04-26T18:12:57.789 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000163 2024-04-26T18:12:57.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.789 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 495952, 2024-04-26T18:12:57.789 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 1288320 2024-04-26T18:12:57.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.789 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T18:12:57.789 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 3308, 2024-04-26T18:12:57.789 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 4115, 2024-04-26T18:12:57.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 1273475, 2024-04-26T18:12:57.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 1491303, 2024-04-26T18:12:57.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 39, 2024-04-26T18:12:57.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 18446744073709551593, 2024-04-26T18:12:57.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.250879894, 2024-04-26T18:12:57.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.083049419, 2024-04-26T18:12:57.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.134697926, 2024-04-26T18:12:57.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.041373723, 2024-04-26T18:12:57.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:57.790 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 4114, 2024-04-26T18:12:57.790 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.218275287, 2024-04-26T18:12:57.790 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000053056 2024-04-26T18:12:57.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.791 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:57.791 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 6422, 2024-04-26T18:12:57.791 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.001529665, 2024-04-26T18:12:57.791 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000238 2024-04-26T18:12:57.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.791 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 263872, 2024-04-26T18:12:57.791 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 375296 2024-04-26T18:12:57.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.791 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T18:12:57.791 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 3169, 2024-04-26T18:12:57.791 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 3172, 2024-04-26T18:12:57.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 1639663, 2024-04-26T18:12:57.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 3833907, 2024-04-26T18:12:57.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 10, 2024-04-26T18:12:57.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 3, 2024-04-26T18:12:57.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.232543335, 2024-04-26T18:12:57.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.093856508, 2024-04-26T18:12:57.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.116955201, 2024-04-26T18:12:57.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.031472544, 2024-04-26T18:12:57.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:12:57.792 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 3171, 2024-04-26T18:12:57.792 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.219353655, 2024-04-26T18:12:57.792 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000069174 2024-04-26T18:12:57.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.793 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:12:57.793 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 4446, 2024-04-26T18:12:57.793 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.001012232, 2024-04-26T18:12:57.793 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000227 2024-04-26T18:12:57.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.793 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 256560, 2024-04-26T18:12:57.793 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 95104 2024-04-26T18:12:57.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.793 INFO:teuthology.orchestra.run.smithi097.stdout: "cct": { 2024-04-26T18:12:57.793 INFO:teuthology.orchestra.run.smithi097.stdout: "total_workers": 1, 2024-04-26T18:12:57.793 INFO:teuthology.orchestra.run.smithi097.stdout: "unhealthy_workers": 0 2024-04-26T18:12:57.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.793 INFO:teuthology.orchestra.run.smithi097.stdout: "finisher-MDSRank": { 2024-04-26T18:12:57.794 INFO:teuthology.orchestra.run.smithi097.stdout: "queue_len": 0, 2024-04-26T18:12:57.794 INFO:teuthology.orchestra.run.smithi097.stdout: "complete_latency": { 2024-04-26T18:12:57.794 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 1178, 2024-04-26T18:12:57.794 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.450953802, 2024-04-26T18:12:57.794 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000382813 2024-04-26T18:12:57.794 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:12:57.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.794 INFO:teuthology.orchestra.run.smithi097.stdout: "finisher-PurgeQueue": { 2024-04-26T18:12:57.794 INFO:teuthology.orchestra.run.smithi097.stdout: "queue_len": 0, 2024-04-26T18:12:57.794 INFO:teuthology.orchestra.run.smithi097.stdout: "complete_latency": { 2024-04-26T18:12:57.794 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 502, 2024-04-26T18:12:57.794 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.019221476, 2024-04-26T18:12:57.794 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000038289 2024-04-26T18:12:57.795 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:12:57.795 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.795 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": { 2024-04-26T18:12:57.795 INFO:teuthology.orchestra.run.smithi097.stdout: "request": 823, 2024-04-26T18:12:57.795 INFO:teuthology.orchestra.run.smithi097.stdout: "reply": 790, 2024-04-26T18:12:57.795 INFO:teuthology.orchestra.run.smithi097.stdout: "reply_latency": { 2024-04-26T18:12:57.795 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 790, 2024-04-26T18:12:57.795 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 4.380381122, 2024-04-26T18:12:57.795 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.005544786 2024-04-26T18:12:57.795 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.795 INFO:teuthology.orchestra.run.smithi097.stdout: "slow_reply": 0, 2024-04-26T18:12:57.795 INFO:teuthology.orchestra.run.smithi097.stdout: "forward": 32, 2024-04-26T18:12:57.795 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_fetch_complete": 26, 2024-04-26T18:12:57.796 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_fetch_keys": 12, 2024-04-26T18:12:57.796 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_commit": 0, 2024-04-26T18:12:57.796 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_split": 0, 2024-04-26T18:12:57.796 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_merge": 0, 2024-04-26T18:12:57.796 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes": 687, 2024-04-26T18:12:57.796 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_top": 72, 2024-04-26T18:12:57.796 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_bottom": 615, 2024-04-26T18:12:57.796 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_pin_tail": 0, 2024-04-26T18:12:57.796 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_pinned": 585, 2024-04-26T18:12:57.796 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_expired": 20, 2024-04-26T18:12:57.796 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_with_caps": 341, 2024-04-26T18:12:57.796 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": 342, 2024-04-26T18:12:57.796 INFO:teuthology.orchestra.run.smithi097.stdout: "subtrees": 23, 2024-04-26T18:12:57.796 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse": 1944, 2024-04-26T18:12:57.797 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_hit": 1576, 2024-04-26T18:12:57.797 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_forward": 32, 2024-04-26T18:12:57.797 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_discover": 18, 2024-04-26T18:12:57.797 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_dir_fetch": 4, 2024-04-26T18:12:57.797 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_remote_ino": 3, 2024-04-26T18:12:57.797 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_lock": 53, 2024-04-26T18:12:57.797 INFO:teuthology.orchestra.run.smithi097.stdout: "load_cent": 2433, 2024-04-26T18:12:57.797 INFO:teuthology.orchestra.run.smithi097.stdout: "q": 0, 2024-04-26T18:12:57.797 INFO:teuthology.orchestra.run.smithi097.stdout: "exported": 0, 2024-04-26T18:12:57.797 INFO:teuthology.orchestra.run.smithi097.stdout: "exported_inodes": 0, 2024-04-26T18:12:57.797 INFO:teuthology.orchestra.run.smithi097.stdout: "imported": 1, 2024-04-26T18:12:57.797 INFO:teuthology.orchestra.run.smithi097.stdout: "imported_inodes": 28, 2024-04-26T18:12:57.797 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_dir_fetch": 6, 2024-04-26T18:12:57.798 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_backtrace_fetch": 2, 2024-04-26T18:12:57.798 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_peer_discover": 8, 2024-04-26T18:12:57.798 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rfiles": 1008, 2024-04-26T18:12:57.798 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rbytes": 1353473565, 2024-04-26T18:12:57.798 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rsnaps": 7, 2024-04-26T18:12:57.798 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T18:12:57.798 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_set_tag": 0, 2024-04-26T18:12:57.798 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T18:12:57.798 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_inotable_repaired": 0, 2024-04-26T18:12:57.798 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dir_inodes": 0, 2024-04-26T18:12:57.798 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T18:12:57.798 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T18:12:57.799 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_file_inodes": 0, 2024-04-26T18:12:57.799 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_inode_file_caps": 4, 2024-04-26T18:12:57.799 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_revoke": 130, 2024-04-26T18:12:57.799 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_grant": 547, 2024-04-26T18:12:57.799 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T18:12:57.799 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T18:12:57.799 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T18:12:57.799 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_caps": 571, 2024-04-26T18:12:57.799 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_caps_dirty": 286, 2024-04-26T18:12:57.799 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_cap_release": 96, 2024-04-26T18:12:57.799 INFO:teuthology.orchestra.run.smithi097.stdout: "process_request_cap_release": 561 2024-04-26T18:12:57.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.799 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache": { 2024-04-26T18:12:57.799 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_update": 0, 2024-04-26T18:12:57.800 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_update_receipt": 0, 2024-04-26T18:12:57.800 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_try_discover": 0, 2024-04-26T18:12:57.800 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_send_discover": 27, 2024-04-26T18:12:57.800 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_handle_discover": 7, 2024-04-26T18:12:57.800 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays": 5, 2024-04-26T18:12:57.800 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays_delayed": 0, 2024-04-26T18:12:57.800 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays_enqueuing": 0, 2024-04-26T18:12:57.800 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_created": 145, 2024-04-26T18:12:57.800 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_enqueued": 135, 2024-04-26T18:12:57.800 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_reintegrated": 7, 2024-04-26T18:12:57.800 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_migrated": 5, 2024-04-26T18:12:57.800 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_processing": 0, 2024-04-26T18:12:57.800 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_enqueued": 0, 2024-04-26T18:12:57.800 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_prioritized": 0, 2024-04-26T18:12:57.801 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_started": 1, 2024-04-26T18:12:57.801 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_completed": 1, 2024-04-26T18:12:57.801 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_quiesce_path": 0, 2024-04-26T18:12:57.801 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_quiesce_inode": 0, 2024-04-26T18:12:57.801 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T18:12:57.801 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_exportdir": 0, 2024-04-26T18:12:57.801 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_flush": 0, 2024-04-26T18:12:57.801 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_fragmentdir": 0, 2024-04-26T18:12:57.801 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_fragstats": 0, 2024-04-26T18:12:57.801 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_inodestats": 0 2024-04-26T18:12:57.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.801 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log": { 2024-04-26T18:12:57.801 INFO:teuthology.orchestra.run.smithi097.stdout: "evlrg": 0, 2024-04-26T18:12:57.801 INFO:teuthology.orchestra.run.smithi097.stdout: "evadd": 1448, 2024-04-26T18:12:57.802 INFO:teuthology.orchestra.run.smithi097.stdout: "evex": 0, 2024-04-26T18:12:57.802 INFO:teuthology.orchestra.run.smithi097.stdout: "evtrm": 0, 2024-04-26T18:12:57.802 INFO:teuthology.orchestra.run.smithi097.stdout: "ev": 1598, 2024-04-26T18:12:57.802 INFO:teuthology.orchestra.run.smithi097.stdout: "evexg": 0, 2024-04-26T18:12:57.802 INFO:teuthology.orchestra.run.smithi097.stdout: "evexd": 0, 2024-04-26T18:12:57.802 INFO:teuthology.orchestra.run.smithi097.stdout: "segadd": 2, 2024-04-26T18:12:57.802 INFO:teuthology.orchestra.run.smithi097.stdout: "segex": 0, 2024-04-26T18:12:57.802 INFO:teuthology.orchestra.run.smithi097.stdout: "segtrm": 0, 2024-04-26T18:12:57.802 INFO:teuthology.orchestra.run.smithi097.stdout: "seg": 3, 2024-04-26T18:12:57.802 INFO:teuthology.orchestra.run.smithi097.stdout: "segmjr": 1, 2024-04-26T18:12:57.802 INFO:teuthology.orchestra.run.smithi097.stdout: "segexg": 0, 2024-04-26T18:12:57.802 INFO:teuthology.orchestra.run.smithi097.stdout: "segexd": 0, 2024-04-26T18:12:57.803 INFO:teuthology.orchestra.run.smithi097.stdout: "expos": 7738070, 2024-04-26T18:12:57.803 INFO:teuthology.orchestra.run.smithi097.stdout: "wrpos": 11163047, 2024-04-26T18:12:57.803 INFO:teuthology.orchestra.run.smithi097.stdout: "rdpos": 8018094, 2024-04-26T18:12:57.803 INFO:teuthology.orchestra.run.smithi097.stdout: "jlat": { 2024-04-26T18:12:57.803 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 433, 2024-04-26T18:12:57.803 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 6.747949353, 2024-04-26T18:12:57.803 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.015584178 2024-04-26T18:12:57.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.803 INFO:teuthology.orchestra.run.smithi097.stdout: "replayed": 150 2024-04-26T18:12:57.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.803 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_mem": { 2024-04-26T18:12:57.803 INFO:teuthology.orchestra.run.smithi097.stdout: "ino": 605, 2024-04-26T18:12:57.803 INFO:teuthology.orchestra.run.smithi097.stdout: "ino+": 760, 2024-04-26T18:12:57.804 INFO:teuthology.orchestra.run.smithi097.stdout: "ino-": 155, 2024-04-26T18:12:57.804 INFO:teuthology.orchestra.run.smithi097.stdout: "dir": 47, 2024-04-26T18:12:57.804 INFO:teuthology.orchestra.run.smithi097.stdout: "dir+": 47, 2024-04-26T18:12:57.804 INFO:teuthology.orchestra.run.smithi097.stdout: "dir-": 0, 2024-04-26T18:12:57.804 INFO:teuthology.orchestra.run.smithi097.stdout: "dn": 689, 2024-04-26T18:12:57.804 INFO:teuthology.orchestra.run.smithi097.stdout: "dn+": 843, 2024-04-26T18:12:57.804 INFO:teuthology.orchestra.run.smithi097.stdout: "dn-": 154, 2024-04-26T18:12:57.804 INFO:teuthology.orchestra.run.smithi097.stdout: "cap": 342, 2024-04-26T18:12:57.804 INFO:teuthology.orchestra.run.smithi097.stdout: "cap+": 503, 2024-04-26T18:12:57.804 INFO:teuthology.orchestra.run.smithi097.stdout: "cap-": 161, 2024-04-26T18:12:57.804 INFO:teuthology.orchestra.run.smithi097.stdout: "rss": 62636, 2024-04-26T18:12:57.804 INFO:teuthology.orchestra.run.smithi097.stdout: "heap": 248184 2024-04-26T18:12:57.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.805 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_server": { 2024-04-26T18:12:57.805 INFO:teuthology.orchestra.run.smithi097.stdout: "dispatch_client_request": 1511, 2024-04-26T18:12:57.805 INFO:teuthology.orchestra.run.smithi097.stdout: "dispatch_server_request": 159, 2024-04-26T18:12:57.805 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_request": 823, 2024-04-26T18:12:57.805 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_session": 28, 2024-04-26T18:12:57.805 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_peer_request": 2035, 2024-04-26T18:12:57.805 INFO:teuthology.orchestra.run.smithi097.stdout: "req_create_latency": { 2024-04-26T18:12:57.805 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 131, 2024-04-26T18:12:57.805 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 2.209292254, 2024-04-26T18:12:57.805 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.016864826 2024-04-26T18:12:57.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.805 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getattr_latency": { 2024-04-26T18:12:57.805 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 156, 2024-04-26T18:12:57.805 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.186210685, 2024-04-26T18:12:57.806 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.001193658 2024-04-26T18:12:57.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.806 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getfilelock_latency": { 2024-04-26T18:12:57.806 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.806 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.806 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.806 INFO:teuthology.orchestra.run.smithi097.stdout: "req_link_latency": { 2024-04-26T18:12:57.806 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 5, 2024-04-26T18:12:57.806 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.039553074, 2024-04-26T18:12:57.806 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.007910614 2024-04-26T18:12:57.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.806 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookup_latency": { 2024-04-26T18:12:57.807 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 201, 2024-04-26T18:12:57.807 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.701128690, 2024-04-26T18:12:57.807 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.003488202 2024-04-26T18:12:57.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.807 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookuphash_latency": { 2024-04-26T18:12:57.807 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.807 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.807 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.807 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupino_latency": { 2024-04-26T18:12:57.807 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.807 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.807 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.808 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupname_latency": { 2024-04-26T18:12:57.808 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.808 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.808 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.808 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupparent_latency": { 2024-04-26T18:12:57.808 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.808 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.808 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.808 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupsnap_latency": { 2024-04-26T18:12:57.808 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.809 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.809 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.809 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lssnap_latency": { 2024-04-26T18:12:57.809 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 8, 2024-04-26T18:12:57.809 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.001385533, 2024-04-26T18:12:57.810 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000173191 2024-04-26T18:12:57.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.810 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mkdir_latency": { 2024-04-26T18:12:57.810 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.810 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.810 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.810 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mknod_latency": { 2024-04-26T18:12:57.810 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.810 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.810 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.810 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mksnap_latency": { 2024-04-26T18:12:57.811 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 8, 2024-04-26T18:12:57.811 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.047749502, 2024-04-26T18:12:57.811 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.005968687 2024-04-26T18:12:57.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.811 INFO:teuthology.orchestra.run.smithi097.stdout: "req_open_latency": { 2024-04-26T18:12:57.811 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 94, 2024-04-26T18:12:57.811 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.051268556, 2024-04-26T18:12:57.811 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000545410 2024-04-26T18:12:57.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.811 INFO:teuthology.orchestra.run.smithi097.stdout: "req_readdir_latency": { 2024-04-26T18:12:57.811 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 21, 2024-04-26T18:12:57.811 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.024430243, 2024-04-26T18:12:57.811 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.001163344 2024-04-26T18:12:57.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.812 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rename_latency": { 2024-04-26T18:12:57.812 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 129, 2024-04-26T18:12:57.812 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.909505205, 2024-04-26T18:12:57.812 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.007050427 2024-04-26T18:12:57.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.812 INFO:teuthology.orchestra.run.smithi097.stdout: "req_renamesnap_latency": { 2024-04-26T18:12:57.812 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.812 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.812 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.812 INFO:teuthology.orchestra.run.smithi097.stdout: "req_snapdiff_latency": { 2024-04-26T18:12:57.812 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.813 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.813 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.813 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmdir_latency": { 2024-04-26T18:12:57.813 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.813 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.813 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.813 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmsnap_latency": { 2024-04-26T18:12:57.813 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 7, 2024-04-26T18:12:57.813 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.038742607, 2024-04-26T18:12:57.813 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.005534658 2024-04-26T18:12:57.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.814 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmxattr_latency": { 2024-04-26T18:12:57.814 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.814 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.814 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.814 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setattr_latency": { 2024-04-26T18:12:57.814 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.814 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.814 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.814 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setdirlayout_latency": { 2024-04-26T18:12:57.814 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.814 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.814 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.815 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setfilelock_latency": { 2024-04-26T18:12:57.815 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.815 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.815 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.815 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setlayout_latency": { 2024-04-26T18:12:57.815 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.815 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.815 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.815 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setxattr_latency": { 2024-04-26T18:12:57.815 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.815 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.816 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.816 INFO:teuthology.orchestra.run.smithi097.stdout: "req_symlink_latency": { 2024-04-26T18:12:57.816 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.816 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.816 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.816 INFO:teuthology.orchestra.run.smithi097.stdout: "req_unlink_latency": { 2024-04-26T18:12:57.816 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 30, 2024-04-26T18:12:57.816 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.171114773, 2024-04-26T18:12:57.816 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.005703825 2024-04-26T18:12:57.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.816 INFO:teuthology.orchestra.run.smithi097.stdout: "cap_revoke_eviction": 0, 2024-04-26T18:12:57.817 INFO:teuthology.orchestra.run.smithi097.stdout: "cap_acquisition_throttle": 0, 2024-04-26T18:12:57.817 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getvxattr_latency": { 2024-04-26T18:12:57.817 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.817 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.817 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.817 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:12:57.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.817 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_sessions": { 2024-04-26T18:12:57.817 INFO:teuthology.orchestra.run.smithi097.stdout: "session_count": 2, 2024-04-26T18:12:57.817 INFO:teuthology.orchestra.run.smithi097.stdout: "session_add": 2, 2024-04-26T18:12:57.817 INFO:teuthology.orchestra.run.smithi097.stdout: "session_remove": 0, 2024-04-26T18:12:57.817 INFO:teuthology.orchestra.run.smithi097.stdout: "sessions_open": 0, 2024-04-26T18:12:57.817 INFO:teuthology.orchestra.run.smithi097.stdout: "sessions_stale": 0, 2024-04-26T18:12:57.818 INFO:teuthology.orchestra.run.smithi097.stdout: "total_load": 195, 2024-04-26T18:12:57.818 INFO:teuthology.orchestra.run.smithi097.stdout: "average_load": 97, 2024-04-26T18:12:57.818 INFO:teuthology.orchestra.run.smithi097.stdout: "avg_session_uptime": 2192, 2024-04-26T18:12:57.818 INFO:teuthology.orchestra.run.smithi097.stdout: "mdthresh_evicted": 0 2024-04-26T18:12:57.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.818 INFO:teuthology.orchestra.run.smithi097.stdout: "mempool": { 2024-04-26T18:12:57.818 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_bytes": 0, 2024-04-26T18:12:57.818 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_items": 0, 2024-04-26T18:12:57.818 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T18:12:57.818 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_items": 0, 2024-04-26T18:12:57.818 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T18:12:57.818 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_items": 0, 2024-04-26T18:12:57.818 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T18:12:57.818 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T18:12:57.819 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T18:12:57.819 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T18:12:57.819 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T18:12:57.819 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_items": 0, 2024-04-26T18:12:57.819 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T18:12:57.819 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T18:12:57.819 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_bytes": 0, 2024-04-26T18:12:57.819 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_items": 0, 2024-04-26T18:12:57.819 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_bytes": 0, 2024-04-26T18:12:57.819 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_items": 0, 2024-04-26T18:12:57.819 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T18:12:57.819 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T18:12:57.819 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T18:12:57.819 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_items": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_bytes": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_items": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_bytes": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_items": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_bytes": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_items": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_items": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_items": 0, 2024-04-26T18:12:57.820 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_bytes": 505420, 2024-04-26T18:12:57.821 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_items": 795, 2024-04-26T18:12:57.821 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_bytes": 528, 2024-04-26T18:12:57.821 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_items": 6, 2024-04-26T18:12:57.821 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bytes": 0, 2024-04-26T18:12:57.821 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_items": 0, 2024-04-26T18:12:57.821 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_bytes": 0, 2024-04-26T18:12:57.821 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_items": 0, 2024-04-26T18:12:57.821 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_bytes": 0, 2024-04-26T18:12:57.821 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_items": 0, 2024-04-26T18:12:57.821 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_bytes": 8352, 2024-04-26T18:12:57.821 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_items": 189, 2024-04-26T18:12:57.821 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T18:12:57.821 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_items": 0, 2024-04-26T18:12:57.821 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_bytes": 0, 2024-04-26T18:12:57.822 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_items": 0, 2024-04-26T18:12:57.822 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_bytes": 2266247, 2024-04-26T18:12:57.822 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_items": 28631, 2024-04-26T18:12:57.822 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_bytes": 0, 2024-04-26T18:12:57.822 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_items": 0, 2024-04-26T18:12:57.822 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_bytes": 0, 2024-04-26T18:12:57.822 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_items": 0 2024-04-26T18:12:57.822 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.822 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter": { 2024-04-26T18:12:57.822 INFO:teuthology.orchestra.run.smithi097.stdout: "op_active": 0, 2024-04-26T18:12:57.822 INFO:teuthology.orchestra.run.smithi097.stdout: "op_laggy": 0, 2024-04-26T18:12:57.822 INFO:teuthology.orchestra.run.smithi097.stdout: "op_send": 1072, 2024-04-26T18:12:57.822 INFO:teuthology.orchestra.run.smithi097.stdout: "op_send_bytes": 3192620, 2024-04-26T18:12:57.822 INFO:teuthology.orchestra.run.smithi097.stdout: "op_resend": 0, 2024-04-26T18:12:57.823 INFO:teuthology.orchestra.run.smithi097.stdout: "op_reply": 1072, 2024-04-26T18:12:57.823 INFO:teuthology.orchestra.run.smithi097.stdout: "op_latency": { 2024-04-26T18:12:57.823 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 1072, 2024-04-26T18:12:57.823 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 10.997820657, 2024-04-26T18:12:57.823 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.010259161 2024-04-26T18:12:57.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.823 INFO:teuthology.orchestra.run.smithi097.stdout: "op_inflight": 0, 2024-04-26T18:12:57.823 INFO:teuthology.orchestra.run.smithi097.stdout: "oplen_avg": { 2024-04-26T18:12:57.823 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 1072, 2024-04-26T18:12:57.823 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 1261 2024-04-26T18:12:57.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.823 INFO:teuthology.orchestra.run.smithi097.stdout: "op": 1072, 2024-04-26T18:12:57.823 INFO:teuthology.orchestra.run.smithi097.stdout: "op_r": 234, 2024-04-26T18:12:57.823 INFO:teuthology.orchestra.run.smithi097.stdout: "op_w": 838, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "op_rmw": 0, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "op_pg": 0, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_stat": 75, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_create": 0, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_read": 115, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_write": 498, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_writefull": 79, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_writesame": 0, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_append": 0, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_zero": 2, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_truncate": 0, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_delete": 146, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_mapext": 0, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_sparse_read": 0, 2024-04-26T18:12:57.824 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_clonerange": 0, 2024-04-26T18:12:57.825 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_getxattr": 33, 2024-04-26T18:12:57.825 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_setxattr": 0, 2024-04-26T18:12:57.825 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_cmpxattr": 0, 2024-04-26T18:12:57.825 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_rmxattr": 0, 2024-04-26T18:12:57.825 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_resetxattrs": 0, 2024-04-26T18:12:57.825 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_call": 0, 2024-04-26T18:12:57.825 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_watch": 0, 2024-04-26T18:12:57.825 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_notify": 0, 2024-04-26T18:12:57.825 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T18:12:57.825 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_pgls": 0, 2024-04-26T18:12:57.825 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_pgls_filter": 0, 2024-04-26T18:12:57.825 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_other": 0, 2024-04-26T18:12:57.825 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_active": 0, 2024-04-26T18:12:57.825 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_send": 0, 2024-04-26T18:12:57.826 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_resend": 0, 2024-04-26T18:12:57.826 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_ping": 0, 2024-04-26T18:12:57.826 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_active": 0, 2024-04-26T18:12:57.826 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_send": 0, 2024-04-26T18:12:57.826 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_resend": 0, 2024-04-26T18:12:57.826 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_active": 0, 2024-04-26T18:12:57.826 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_send": 0, 2024-04-26T18:12:57.827 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_resend": 0, 2024-04-26T18:12:57.827 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_active": 0, 2024-04-26T18:12:57.827 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_send": 0, 2024-04-26T18:12:57.827 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_resend": 0, 2024-04-26T18:12:57.827 INFO:teuthology.orchestra.run.smithi097.stdout: "command_active": 0, 2024-04-26T18:12:57.827 INFO:teuthology.orchestra.run.smithi097.stdout: "command_send": 0, 2024-04-26T18:12:57.827 INFO:teuthology.orchestra.run.smithi097.stdout: "command_resend": 0, 2024-04-26T18:12:57.827 INFO:teuthology.orchestra.run.smithi097.stdout: "map_epoch": 111, 2024-04-26T18:12:57.827 INFO:teuthology.orchestra.run.smithi097.stdout: "map_full": 0, 2024-04-26T18:12:57.827 INFO:teuthology.orchestra.run.smithi097.stdout: "map_inc": 17, 2024-04-26T18:12:57.827 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_sessions": 12, 2024-04-26T18:12:57.827 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_session_open": 12, 2024-04-26T18:12:57.827 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_session_close": 0, 2024-04-26T18:12:57.827 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_laggy": 0, 2024-04-26T18:12:57.828 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_wr": 177, 2024-04-26T18:12:57.828 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_rd": 80, 2024-04-26T18:12:57.828 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_del": 56 2024-04-26T18:12:57.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.828 INFO:teuthology.orchestra.run.smithi097.stdout: "oft": { 2024-04-26T18:12:57.828 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_objs": 0, 2024-04-26T18:12:57.828 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_kv_pairs": 0, 2024-04-26T18:12:57.828 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_updates": 0, 2024-04-26T18:12:57.828 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_removes": 0 2024-04-26T18:12:57.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.828 INFO:teuthology.orchestra.run.smithi097.stdout: "osd-slow-ops": { 2024-04-26T18:12:57.828 INFO:teuthology.orchestra.run.smithi097.stdout: "slow_ops_count": 0 2024-04-26T18:12:57.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.828 INFO:teuthology.orchestra.run.smithi097.stdout: "purge_queue": { 2024-04-26T18:12:57.829 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_ops": 0, 2024-04-26T18:12:57.829 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_ops_high_water": 10, 2024-04-26T18:12:57.829 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing": 0, 2024-04-26T18:12:57.829 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_high_water": 10, 2024-04-26T18:12:57.829 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executed_ops": 135, 2024-04-26T18:12:57.829 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executed": 135, 2024-04-26T18:12:57.829 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_item_in_journal": 0 2024-04-26T18:12:57.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.829 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T18:12:57.829 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:12:57.829 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 104857600, 2024-04-26T18:12:57.829 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:12:57.829 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 8160, 2024-04-26T18:12:57.829 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 2941461, 2024-04-26T18:12:57.830 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:12:57.830 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 8160, 2024-04-26T18:12:57.830 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-04-26T18:12:57.830 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-04-26T18:12:57.830 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 8160, 2024-04-26T18:12:57.830 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 2941461, 2024-04-26T18:12:57.830 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:12:57.830 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.830 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.830 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.830 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:12:57.830 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.830 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-objecter_bytes": { 2024-04-26T18:12:57.831 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:12:57.831 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 104857600, 2024-04-26T18:12:57.831 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:12:57.831 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-04-26T18:12:57.831 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-04-26T18:12:57.831 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:12:57.831 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-04-26T18:12:57.831 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 1072, 2024-04-26T18:12:57.831 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 3486823, 2024-04-26T18:12:57.831 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 1072, 2024-04-26T18:12:57.831 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 3486823, 2024-04-26T18:12:57.831 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:12:57.831 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.831 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.832 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.832 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:12:57.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.832 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-objecter_ops": { 2024-04-26T18:12:57.832 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:12:57.832 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 1024, 2024-04-26T18:12:57.832 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:12:57.832 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-04-26T18:12:57.832 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-04-26T18:12:57.832 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:12:57.832 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-04-26T18:12:57.832 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 1072, 2024-04-26T18:12:57.832 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 1072, 2024-04-26T18:12:57.832 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 1072, 2024-04-26T18:12:57.833 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 1072, 2024-04-26T18:12:57.833 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:12:57.833 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.833 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.833 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.833 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:12:57.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.833 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-write_buf_throttle": { 2024-04-26T18:12:57.833 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:12:57.833 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 3758096384, 2024-04-26T18:12:57.833 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:12:57.833 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 135, 2024-04-26T18:12:57.833 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 14211, 2024-04-26T18:12:57.834 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:12:57.834 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 135, 2024-04-26T18:12:57.834 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-04-26T18:12:57.834 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-04-26T18:12:57.834 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 65, 2024-04-26T18:12:57.834 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 14211, 2024-04-26T18:12:57.834 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:12:57.834 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.834 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.834 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.834 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:12:57.834 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:12:57.834 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-write_buf_throttle-0x5556c816c780": { 2024-04-26T18:12:57.834 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 599, 2024-04-26T18:12:57.835 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 3758096384, 2024-04-26T18:12:57.835 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:12:57.835 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 1448, 2024-04-26T18:12:57.835 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 3144379, 2024-04-26T18:12:57.835 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:12:57.835 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 1448, 2024-04-26T18:12:57.835 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-04-26T18:12:57.835 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-04-26T18:12:57.835 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 433, 2024-04-26T18:12:57.835 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 3143780, 2024-04-26T18:12:57.835 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:12:57.835 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:12:57.835 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:12:57.835 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:12:57.836 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:12:57.836 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:12:57.836 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-26T18:12:58.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:58 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:12:58.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:58 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:12:58.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:58 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:12:58.541 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:58 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:12:58.541 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:58 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:12:58.541 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:58 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:12:58.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:58 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T18:12:58.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:58 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T18:12:58.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:58 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' 2024-04-26T18:12:58.571 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-04-26T18:12:58.571 INFO:tasks.check_counter:Daemon mds.b mds.imported=1 2024-04-26T18:12:58.571 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=7 2024-04-26T18:12:58.571 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=8 2024-04-26T18:12:58.571 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=7 2024-04-26T18:12:58.571 DEBUG:tasks.check_counter:Getting stats from e 2024-04-26T18:12:58.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-26T18:12:58.830 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T18:12:59.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:12:59 smithi097 ceph-mon[32419]: pgmap v517: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 82 KiB/s wr, 16 op/s 2024-04-26T18:12:59.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:12:59 smithi186 ceph-mon[32797]: pgmap v517: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 82 KiB/s wr, 16 op/s 2024-04-26T18:12:59.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:12:59 smithi031 ceph-mon[29164]: pgmap v517: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 82 KiB/s wr, 16 op/s 2024-04-26T18:13:00.496 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-26T18:13:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T18:13:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 939, 2024-04-26T18:13:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 941, 2024-04-26T18:13:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 12823217, 2024-04-26T18:13:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 298666, 2024-04-26T18:13:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 8, 2024-04-26T18:13:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 3, 2024-04-26T18:13:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.082315030, 2024-04-26T18:13:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.023412139, 2024-04-26T18:13:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.026322667, 2024-04-26T18:13:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.024893975, 2024-04-26T18:13:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 939, 2024-04-26T18:13:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.050053548, 2024-04-26T18:13:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000053305 2024-04-26T18:13:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T18:13:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T18:13:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T18:13:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 1530, 2024-04-26T18:13:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 1485, 2024-04-26T18:13:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 14707204, 2024-04-26T18:13:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 566519, 2024-04-26T18:13:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 5, 2024-04-26T18:13:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 4, 2024-04-26T18:13:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.150538781, 2024-04-26T18:13:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.043050449, 2024-04-26T18:13:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.074803342, 2024-04-26T18:13:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.035048386, 2024-04-26T18:13:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 1485, 2024-04-26T18:13:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.053760187, 2024-04-26T18:13:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000036202 2024-04-26T18:13:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 1785776, 2024-04-26T18:13:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 208128 2024-04-26T18:13:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T18:13:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 1665, 2024-04-26T18:13:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 1913, 2024-04-26T18:13:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 5345473, 2024-04-26T18:13:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 1126514, 2024-04-26T18:13:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 7, 2024-04-26T18:13:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 5, 2024-04-26T18:13:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.172005047, 2024-04-26T18:13:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.057791751, 2024-04-26T18:13:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.043500027, 2024-04-26T18:13:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.054453299, 2024-04-26T18:13:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 1910, 2024-04-26T18:13:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.088294685, 2024-04-26T18:13:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000046227 2024-04-26T18:13:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-26T18:13:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 597680 2024-04-26T18:13:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "cct": { 2024-04-26T18:13:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "total_workers": 1, 2024-04-26T18:13:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "unhealthy_workers": 0 2024-04-26T18:13:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "finisher-MDSRank": { 2024-04-26T18:13:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "queue_len": 0, 2024-04-26T18:13:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "complete_latency": { 2024-04-26T18:13:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 6387, 2024-04-26T18:13:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.161815855, 2024-04-26T18:13:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000025335 2024-04-26T18:13:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "finisher-PurgeQueue": { 2024-04-26T18:13:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "queue_len": 0, 2024-04-26T18:13:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "complete_latency": { 2024-04-26T18:13:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": { 2024-04-26T18:13:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "request": 0, 2024-04-26T18:13:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "reply": 0, 2024-04-26T18:13:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "reply_latency": { 2024-04-26T18:13:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "slow_reply": 0, 2024-04-26T18:13:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "forward": 0, 2024-04-26T18:13:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_fetch_complete": 0, 2024-04-26T18:13:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_fetch_keys": 0, 2024-04-26T18:13:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_commit": 0, 2024-04-26T18:13:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_split": 0, 2024-04-26T18:13:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_merge": 0, 2024-04-26T18:13:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes": 469, 2024-04-26T18:13:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_top": 4, 2024-04-26T18:13:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_bottom": 465, 2024-04-26T18:13:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_pin_tail": 0, 2024-04-26T18:13:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_pinned": 463, 2024-04-26T18:13:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_expired": 0, 2024-04-26T18:13:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_with_caps": 0, 2024-04-26T18:13:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": 0, 2024-04-26T18:13:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "subtrees": 16, 2024-04-26T18:13:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse": 0, 2024-04-26T18:13:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_hit": 0, 2024-04-26T18:13:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_forward": 0, 2024-04-26T18:13:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_discover": 0, 2024-04-26T18:13:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_dir_fetch": 0, 2024-04-26T18:13:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_remote_ino": 0, 2024-04-26T18:13:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_lock": 0, 2024-04-26T18:13:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "load_cent": 0, 2024-04-26T18:13:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "q": 0, 2024-04-26T18:13:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "exported": 0, 2024-04-26T18:13:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "exported_inodes": 0, 2024-04-26T18:13:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "imported": 0, 2024-04-26T18:13:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "imported_inodes": 0, 2024-04-26T18:13:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_dir_fetch": 0, 2024-04-26T18:13:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_backtrace_fetch": 0, 2024-04-26T18:13:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_peer_discover": 0, 2024-04-26T18:13:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rfiles": 0, 2024-04-26T18:13:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rbytes": 0, 2024-04-26T18:13:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rsnaps": 0, 2024-04-26T18:13:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T18:13:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_set_tag": 0, 2024-04-26T18:13:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T18:13:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_inotable_repaired": 0, 2024-04-26T18:13:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dir_inodes": 0, 2024-04-26T18:13:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T18:13:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T18:13:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_file_inodes": 0, 2024-04-26T18:13:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_inode_file_caps": 0, 2024-04-26T18:13:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T18:13:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_grant": 0, 2024-04-26T18:13:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T18:13:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T18:13:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T18:13:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_caps": 0, 2024-04-26T18:13:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_caps_dirty": 0, 2024-04-26T18:13:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_cap_release": 0, 2024-04-26T18:13:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "process_request_cap_release": 0 2024-04-26T18:13:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache": { 2024-04-26T18:13:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_update": 0, 2024-04-26T18:13:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_update_receipt": 0, 2024-04-26T18:13:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_try_discover": 0, 2024-04-26T18:13:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_send_discover": 0, 2024-04-26T18:13:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_handle_discover": 0, 2024-04-26T18:13:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays": 0, 2024-04-26T18:13:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays_delayed": 0, 2024-04-26T18:13:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays_enqueuing": 0, 2024-04-26T18:13:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_created": 0, 2024-04-26T18:13:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_enqueued": 0, 2024-04-26T18:13:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_reintegrated": 0, 2024-04-26T18:13:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_migrated": 0, 2024-04-26T18:13:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_processing": 0, 2024-04-26T18:13:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_enqueued": 0, 2024-04-26T18:13:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_prioritized": 0, 2024-04-26T18:13:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_started": 0, 2024-04-26T18:13:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_completed": 0, 2024-04-26T18:13:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_quiesce_path": 0, 2024-04-26T18:13:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_quiesce_inode": 0, 2024-04-26T18:13:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T18:13:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_exportdir": 0, 2024-04-26T18:13:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_flush": 0, 2024-04-26T18:13:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_fragmentdir": 0, 2024-04-26T18:13:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_fragstats": 0, 2024-04-26T18:13:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_inodestats": 0 2024-04-26T18:13:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log": { 2024-04-26T18:13:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "evlrg": 0, 2024-04-26T18:13:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "evadd": 0, 2024-04-26T18:13:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "evex": 0, 2024-04-26T18:13:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "evtrm": 0, 2024-04-26T18:13:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "ev": 4658, 2024-04-26T18:13:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "evexg": 0, 2024-04-26T18:13:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "evexd": 0, 2024-04-26T18:13:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "segadd": 0, 2024-04-26T18:13:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "segex": 0, 2024-04-26T18:13:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "segtrm": 0, 2024-04-26T18:13:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "seg": 10, 2024-04-26T18:13:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "segmjr": 2, 2024-04-26T18:13:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "segexg": 0, 2024-04-26T18:13:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "segexd": 0, 2024-04-26T18:13:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "expos": 4194304, 2024-04-26T18:13:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "wrpos": 34189545, 2024-04-26T18:13:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "rdpos": 34134612, 2024-04-26T18:13:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "jlat": { 2024-04-26T18:13:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "replayed": 4658 2024-04-26T18:13:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_mem": { 2024-04-26T18:13:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "ino": 132, 2024-04-26T18:13:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "ino+": 1238, 2024-04-26T18:13:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "ino-": 1106, 2024-04-26T18:13:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "dir": 44, 2024-04-26T18:13:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "dir+": 80, 2024-04-26T18:13:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "dir-": 36, 2024-04-26T18:13:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "dn": 469, 2024-04-26T18:13:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "dn+": 1600, 2024-04-26T18:13:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "dn-": 1131, 2024-04-26T18:13:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "cap": 0, 2024-04-26T18:13:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "cap+": 0, 2024-04-26T18:13:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "cap-": 0, 2024-04-26T18:13:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "rss": 53136, 2024-04-26T18:13:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "heap": 215416 2024-04-26T18:13:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_server": { 2024-04-26T18:13:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "dispatch_client_request": 0, 2024-04-26T18:13:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "dispatch_server_request": 0, 2024-04-26T18:13:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_request": 0, 2024-04-26T18:13:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_session": 0, 2024-04-26T18:13:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_peer_request": 0, 2024-04-26T18:13:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "req_create_latency": { 2024-04-26T18:13:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getattr_latency": { 2024-04-26T18:13:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getfilelock_latency": { 2024-04-26T18:13:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "req_link_latency": { 2024-04-26T18:13:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookup_latency": { 2024-04-26T18:13:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookuphash_latency": { 2024-04-26T18:13:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupino_latency": { 2024-04-26T18:13:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupname_latency": { 2024-04-26T18:13:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupparent_latency": { 2024-04-26T18:13:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupsnap_latency": { 2024-04-26T18:13:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lssnap_latency": { 2024-04-26T18:13:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mkdir_latency": { 2024-04-26T18:13:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mknod_latency": { 2024-04-26T18:13:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mksnap_latency": { 2024-04-26T18:13:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "req_open_latency": { 2024-04-26T18:13:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "req_readdir_latency": { 2024-04-26T18:13:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rename_latency": { 2024-04-26T18:13:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "req_renamesnap_latency": { 2024-04-26T18:13:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "req_snapdiff_latency": { 2024-04-26T18:13:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmdir_latency": { 2024-04-26T18:13:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmsnap_latency": { 2024-04-26T18:13:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmxattr_latency": { 2024-04-26T18:13:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setattr_latency": { 2024-04-26T18:13:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setdirlayout_latency": { 2024-04-26T18:13:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setfilelock_latency": { 2024-04-26T18:13:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setlayout_latency": { 2024-04-26T18:13:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setxattr_latency": { 2024-04-26T18:13:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "req_symlink_latency": { 2024-04-26T18:13:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "req_unlink_latency": { 2024-04-26T18:13:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "cap_revoke_eviction": 0, 2024-04-26T18:13:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "cap_acquisition_throttle": 0, 2024-04-26T18:13:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getvxattr_latency": { 2024-04-26T18:13:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_sessions": { 2024-04-26T18:13:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "session_count": 2, 2024-04-26T18:13:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "session_add": 3, 2024-04-26T18:13:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "session_remove": 1, 2024-04-26T18:13:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "sessions_open": 3, 2024-04-26T18:13:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "sessions_stale": 0, 2024-04-26T18:13:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "total_load": 0, 2024-04-26T18:13:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "average_load": 0, 2024-04-26T18:13:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "avg_session_uptime": 2796, 2024-04-26T18:13:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "mdthresh_evicted": 0 2024-04-26T18:13:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "mempool": { 2024-04-26T18:13:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_bytes": 0, 2024-04-26T18:13:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_items": 0, 2024-04-26T18:13:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T18:13:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_items": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_items": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_items": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_bytes": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_items": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_bytes": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_items": 0, 2024-04-26T18:13:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T18:13:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T18:13:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T18:13:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T18:13:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T18:13:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_items": 0, 2024-04-26T18:13:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_bytes": 0, 2024-04-26T18:13:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_items": 0, 2024-04-26T18:13:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T18:13:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T18:13:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_bytes": 0, 2024-04-26T18:13:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_items": 0, 2024-04-26T18:13:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_bytes": 0, 2024-04-26T18:13:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_items": 0, 2024-04-26T18:13:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T18:13:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_items": 0, 2024-04-26T18:13:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T18:13:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_items": 0, 2024-04-26T18:13:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_bytes": 230126, 2024-04-26T18:13:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_items": 432, 2024-04-26T18:13:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_bytes": 0, 2024-04-26T18:13:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_items": 0, 2024-04-26T18:13:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bytes": 0, 2024-04-26T18:13:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_items": 0, 2024-04-26T18:13:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_bytes": 0, 2024-04-26T18:13:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_items": 0, 2024-04-26T18:13:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_bytes": 0, 2024-04-26T18:13:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_items": 0, 2024-04-26T18:13:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_bytes": 8352, 2024-04-26T18:13:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_items": 189, 2024-04-26T18:13:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T18:13:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_items": 0, 2024-04-26T18:13:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_bytes": 0, 2024-04-26T18:13:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_items": 0, 2024-04-26T18:13:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_bytes": 979517, 2024-04-26T18:13:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_items": 31103, 2024-04-26T18:13:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_bytes": 0, 2024-04-26T18:13:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_items": 0, 2024-04-26T18:13:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_bytes": 0, 2024-04-26T18:13:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_items": 0 2024-04-26T18:13:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter": { 2024-04-26T18:13:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "op_active": 0, 2024-04-26T18:13:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "op_laggy": 0, 2024-04-26T18:13:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "op_send": 3729, 2024-04-26T18:13:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "op_send_bytes": 21, 2024-04-26T18:13:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "op_resend": 3, 2024-04-26T18:13:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "op_reply": 3729, 2024-04-26T18:13:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "op_latency": { 2024-04-26T18:13:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 3729, 2024-04-26T18:13:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 6.802887684, 2024-04-26T18:13:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.001824319 2024-04-26T18:13:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "op_inflight": 0, 2024-04-26T18:13:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "oplen_avg": { 2024-04-26T18:13:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 3729, 2024-04-26T18:13:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 3730 2024-04-26T18:13:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "op": 3729, 2024-04-26T18:13:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "op_r": 3729, 2024-04-26T18:13:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "op_w": 0, 2024-04-26T18:13:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "op_rmw": 0, 2024-04-26T18:13:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "op_pg": 0, 2024-04-26T18:13:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_stat": 2157, 2024-04-26T18:13:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_create": 0, 2024-04-26T18:13:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_read": 1570, 2024-04-26T18:13:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_write": 0, 2024-04-26T18:13:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_writefull": 0, 2024-04-26T18:13:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_writesame": 0, 2024-04-26T18:13:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_append": 0, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_zero": 0, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_truncate": 0, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_delete": 0, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_mapext": 0, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_sparse_read": 0, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_clonerange": 0, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_getxattr": 1, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_setxattr": 0, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_cmpxattr": 0, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_rmxattr": 0, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_resetxattrs": 0, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_call": 0, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_watch": 0, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_notify": 0, 2024-04-26T18:13:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T18:13:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_pgls": 0, 2024-04-26T18:13:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_pgls_filter": 0, 2024-04-26T18:13:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_other": 0, 2024-04-26T18:13:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_active": 0, 2024-04-26T18:13:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_send": 0, 2024-04-26T18:13:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_resend": 0, 2024-04-26T18:13:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_ping": 0, 2024-04-26T18:13:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_active": 0, 2024-04-26T18:13:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_send": 0, 2024-04-26T18:13:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_resend": 0, 2024-04-26T18:13:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_active": 0, 2024-04-26T18:13:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_send": 0, 2024-04-26T18:13:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_resend": 0, 2024-04-26T18:13:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_active": 0, 2024-04-26T18:13:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_send": 0, 2024-04-26T18:13:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_resend": 0, 2024-04-26T18:13:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "command_active": 0, 2024-04-26T18:13:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "command_send": 0, 2024-04-26T18:13:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "command_resend": 0, 2024-04-26T18:13:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "map_epoch": 111, 2024-04-26T18:13:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "map_full": 0, 2024-04-26T18:13:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "map_inc": 23, 2024-04-26T18:13:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_sessions": 8, 2024-04-26T18:13:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_session_open": 9, 2024-04-26T18:13:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_session_close": 1, 2024-04-26T18:13:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_laggy": 0, 2024-04-26T18:13:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_wr": 0, 2024-04-26T18:13:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_rd": 2, 2024-04-26T18:13:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_del": 0 2024-04-26T18:13:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "oft": { 2024-04-26T18:13:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_objs": 0, 2024-04-26T18:13:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_kv_pairs": 0, 2024-04-26T18:13:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_updates": 0, 2024-04-26T18:13:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_removes": 0 2024-04-26T18:13:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "osd-slow-ops": { 2024-04-26T18:13:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "slow_ops_count": 0 2024-04-26T18:13:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "purge_queue": { 2024-04-26T18:13:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_ops": 0, 2024-04-26T18:13:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T18:13:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing": 0, 2024-04-26T18:13:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_high_water": 0, 2024-04-26T18:13:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executed_ops": 0, 2024-04-26T18:13:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executed": 0, 2024-04-26T18:13:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_item_in_journal": 0 2024-04-26T18:13:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T18:13:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:13:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 104857600, 2024-04-26T18:13:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:13:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 4130, 2024-04-26T18:13:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 32480999, 2024-04-26T18:13:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 4130, 2024-04-26T18:13:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-04-26T18:13:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-04-26T18:13:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 4130, 2024-04-26T18:13:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 32480999, 2024-04-26T18:13:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:13:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-objecter_bytes": { 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 104857600, 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 3729, 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 30046674, 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 3729, 2024-04-26T18:13:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 30046674, 2024-04-26T18:13:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:13:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-objecter_ops": { 2024-04-26T18:13:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:13:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 1024, 2024-04-26T18:13:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:13:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-04-26T18:13:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-04-26T18:13:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 3729, 2024-04-26T18:13:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 3729, 2024-04-26T18:13:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 3729, 2024-04-26T18:13:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 3729, 2024-04-26T18:13:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:13:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-write_buf_throttle": { 2024-04-26T18:13:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:13:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 3758096384, 2024-04-26T18:13:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:13:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-04-26T18:13:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-04-26T18:13:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-04-26T18:13:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-04-26T18:13:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 0, 2024-04-26T18:13:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 0, 2024-04-26T18:13:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:13:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-write_buf_throttle-0x555fd525e6e0": { 2024-04-26T18:13:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:13:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 3758096384, 2024-04-26T18:13:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:13:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-04-26T18:13:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-04-26T18:13:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-04-26T18:13:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-04-26T18:13:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 0, 2024-04-26T18:13:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 0, 2024-04-26T18:13:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:13:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:00.547 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-26T18:13:01.284 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-26T18:13:01.284 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-26T18:13:01.284 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-26T18:13:01.284 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-26T18:13:01.284 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T18:13:01.284 DEBUG:tasks.check_counter:Getting stats from h 2024-04-26T18:13:01.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-26T18:13:01.523 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T18:13:01.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:01 smithi097 ceph-mon[32419]: pgmap v518: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 69 KiB/s wr, 9 op/s 2024-04-26T18:13:01.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:01 smithi097 ceph-mon[32419]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:13:01.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:01 smithi186 ceph-mon[32797]: pgmap v518: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 69 KiB/s wr, 9 op/s 2024-04-26T18:13:01.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:01 smithi186 ceph-mon[32797]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:13:01.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:01 smithi031 ceph-mon[29164]: pgmap v518: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 69 KiB/s wr, 9 op/s 2024-04-26T18:13:01.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:01 smithi031 ceph-mon[29164]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T18:13:03.206 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-26T18:13:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T18:13:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 2627, 2024-04-26T18:13:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 2924, 2024-04-26T18:13:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 709794, 2024-04-26T18:13:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 14568111, 2024-04-26T18:13:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 10, 2024-04-26T18:13:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 3, 2024-04-26T18:13:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.225308949, 2024-04-26T18:13:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.107731724, 2024-04-26T18:13:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.099625750, 2024-04-26T18:13:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.024897137, 2024-04-26T18:13:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 2913, 2024-04-26T18:13:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.106944235, 2024-04-26T18:13:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000036712 2024-04-26T18:13:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 3893, 2024-04-26T18:13:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000730468, 2024-04-26T18:13:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000187 2024-04-26T18:13:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T18:13:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T18:13:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T18:13:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 1917, 2024-04-26T18:13:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 2079, 2024-04-26T18:13:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 2231243, 2024-04-26T18:13:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 14048192, 2024-04-26T18:13:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 11, 2024-04-26T18:13:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 9, 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.195660879, 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.076954894, 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.071531518, 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.046872952, 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 2076, 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.103472204, 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000049842 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 2, 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000250, 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000125 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 1795056, 2024-04-26T18:13:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 657248 2024-04-26T18:13:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T18:13:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 13068, 2024-04-26T18:13:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 18046, 2024-04-26T18:13:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 4339585, 2024-04-26T18:13:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 13502440, 2024-04-26T18:13:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 17, 2024-04-26T18:13:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 7, 2024-04-26T18:13:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.785588770, 2024-04-26T18:13:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.361595851, 2024-04-26T18:13:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.577616729, 2024-04-26T18:13:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.011670957, 2024-04-26T18:13:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 18037, 2024-04-26T18:13:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.521054391, 2024-04-26T18:13:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000028888 2024-04-26T18:13:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 28627, 2024-04-26T18:13:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.004281954, 2024-04-26T18:13:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000149 2024-04-26T18:13:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 1890192, 2024-04-26T18:13:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 5347744 2024-04-26T18:13:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "cct": { 2024-04-26T18:13:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "total_workers": 1, 2024-04-26T18:13:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "unhealthy_workers": 0 2024-04-26T18:13:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "finisher-MDSRank": { 2024-04-26T18:13:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "queue_len": 0, 2024-04-26T18:13:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "complete_latency": { 2024-04-26T18:13:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 4223, 2024-04-26T18:13:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 2.514341473, 2024-04-26T18:13:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000595392 2024-04-26T18:13:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "finisher-PurgeQueue": { 2024-04-26T18:13:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "queue_len": 0, 2024-04-26T18:13:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "complete_latency": { 2024-04-26T18:13:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 81, 2024-04-26T18:13:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.002582090, 2024-04-26T18:13:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000031877 2024-04-26T18:13:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": { 2024-04-26T18:13:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "request": 2855, 2024-04-26T18:13:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "reply": 2684, 2024-04-26T18:13:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "reply_latency": { 2024-04-26T18:13:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 2684, 2024-04-26T18:13:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 12.911248540, 2024-04-26T18:13:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.004810450 2024-04-26T18:13:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "slow_reply": 0, 2024-04-26T18:13:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "forward": 171, 2024-04-26T18:13:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_fetch_complete": 8, 2024-04-26T18:13:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_fetch_keys": 0, 2024-04-26T18:13:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_commit": 47, 2024-04-26T18:13:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_split": 4, 2024-04-26T18:13:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_merge": 0, 2024-04-26T18:13:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes": 434, 2024-04-26T18:13:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_top": 35, 2024-04-26T18:13:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_bottom": 312, 2024-04-26T18:13:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_pin_tail": 87, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_pinned": 384, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_expired": 1221, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_with_caps": 56, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": 56, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "subtrees": 19, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse": 5155, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_hit": 3802, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_forward": 163, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_discover": 45, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_dir_fetch": 2, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_remote_ino": 0, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_lock": 277, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "load_cent": 1106, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "q": 0, 2024-04-26T18:13:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "exported": 30, 2024-04-26T18:13:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "exported_inodes": 1112, 2024-04-26T18:13:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "imported": 2, 2024-04-26T18:13:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "imported_inodes": 7, 2024-04-26T18:13:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_dir_fetch": 0, 2024-04-26T18:13:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_backtrace_fetch": 0, 2024-04-26T18:13:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_peer_discover": 0, 2024-04-26T18:13:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rfiles": 1025, 2024-04-26T18:13:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rbytes": 1599855916, 2024-04-26T18:13:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rsnaps": 6, 2024-04-26T18:13:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T18:13:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_set_tag": 0, 2024-04-26T18:13:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T18:13:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_inotable_repaired": 0, 2024-04-26T18:13:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dir_inodes": 0, 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_file_inodes": 0, 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_inode_file_caps": 63, 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_revoke": 861, 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_grant": 2147, 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_caps": 2441, 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_caps_dirty": 1434, 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_cap_release": 77, 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "process_request_cap_release": 558 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache": { 2024-04-26T18:13:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_update": 0, 2024-04-26T18:13:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_update_receipt": 0, 2024-04-26T18:13:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_try_discover": 0, 2024-04-26T18:13:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_send_discover": 46, 2024-04-26T18:13:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_handle_discover": 84, 2024-04-26T18:13:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays": 10, 2024-04-26T18:13:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays_delayed": 0, 2024-04-26T18:13:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays_enqueuing": 0, 2024-04-26T18:13:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_created": 182, 2024-04-26T18:13:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_enqueued": 18, 2024-04-26T18:13:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_reintegrated": 153, 2024-04-26T18:13:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_migrated": 19, 2024-04-26T18:13:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_processing": 0, 2024-04-26T18:13:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_enqueued": 0, 2024-04-26T18:13:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_prioritized": 0, 2024-04-26T18:13:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_started": 0, 2024-04-26T18:13:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_completed": 0, 2024-04-26T18:13:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_quiesce_path": 5, 2024-04-26T18:13:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_quiesce_inode": 1096, 2024-04-26T18:13:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T18:13:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_exportdir": 30, 2024-04-26T18:13:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_flush": 0, 2024-04-26T18:13:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_fragmentdir": 4, 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_fragstats": 0, 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_inodestats": 0 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log": { 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "evlrg": 0, 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "evadd": 4663, 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "evex": 0, 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "evtrm": 0, 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "ev": 4663, 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "evexg": 0, 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "evexd": 0, 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "segadd": 10, 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "segex": 0, 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "segtrm": 0, 2024-04-26T18:13:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "seg": 10, 2024-04-26T18:13:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "segmjr": 2, 2024-04-26T18:13:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "segexg": 0, 2024-04-26T18:13:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "segexd": 0, 2024-04-26T18:13:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "expos": 4194304, 2024-04-26T18:13:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "wrpos": 34314614, 2024-04-26T18:13:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "rdpos": 0, 2024-04-26T18:13:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "jlat": { 2024-04-26T18:13:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 2223, 2024-04-26T18:13:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 9.703792909, 2024-04-26T18:13:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.004365178 2024-04-26T18:13:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "replayed": 0 2024-04-26T18:13:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_mem": { 2024-04-26T18:13:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "ino": 151, 2024-04-26T18:13:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "ino+": 1247, 2024-04-26T18:13:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "ino-": 1096, 2024-04-26T18:13:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "dir": 50, 2024-04-26T18:13:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "dir+": 88, 2024-04-26T18:13:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "dir-": 38, 2024-04-26T18:13:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "dn": 443, 2024-04-26T18:13:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "dn+": 1975, 2024-04-26T18:13:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "dn-": 1532, 2024-04-26T18:13:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "cap": 56, 2024-04-26T18:13:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "cap+": 1260, 2024-04-26T18:13:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "cap-": 1204, 2024-04-26T18:13:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "rss": 69756, 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "heap": 223608 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_server": { 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "dispatch_client_request": 4106, 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "dispatch_server_request": 1480, 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_request": 2855, 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_session": 416, 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_peer_request": 3493, 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "req_create_latency": { 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 1034, 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 2.325542474, 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.002249073 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getattr_latency": { 2024-04-26T18:13:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 48, 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.071345598, 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.001486366 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getfilelock_latency": { 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "req_link_latency": { 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 167, 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.569683689, 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.003411279 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookup_latency": { 2024-04-26T18:13:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 520, 2024-04-26T18:13:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 2.816358261, 2024-04-26T18:13:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.005416073 2024-04-26T18:13:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookuphash_latency": { 2024-04-26T18:13:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupino_latency": { 2024-04-26T18:13:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupname_latency": { 2024-04-26T18:13:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupparent_latency": { 2024-04-26T18:13:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupsnap_latency": { 2024-04-26T18:13:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lssnap_latency": { 2024-04-26T18:13:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mkdir_latency": { 2024-04-26T18:13:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 31, 2024-04-26T18:13:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.146371532, 2024-04-26T18:13:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.004721662 2024-04-26T18:13:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mknod_latency": { 2024-04-26T18:13:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mksnap_latency": { 2024-04-26T18:13:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: "req_open_latency": { 2024-04-26T18:13:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 460, 2024-04-26T18:13:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.168759708, 2024-04-26T18:13:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000366868 2024-04-26T18:13:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "req_readdir_latency": { 2024-04-26T18:13:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 61, 2024-04-26T18:13:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.087716175, 2024-04-26T18:13:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.001437970 2024-04-26T18:13:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rename_latency": { 2024-04-26T18:13:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 186, 2024-04-26T18:13:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 6.278215905, 2024-04-26T18:13:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.033753848 2024-04-26T18:13:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "req_renamesnap_latency": { 2024-04-26T18:13:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "req_snapdiff_latency": { 2024-04-26T18:13:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmdir_latency": { 2024-04-26T18:13:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 2, 2024-04-26T18:13:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.011112236, 2024-04-26T18:13:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.005556118 2024-04-26T18:13:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmsnap_latency": { 2024-04-26T18:13:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmxattr_latency": { 2024-04-26T18:13:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setattr_latency": { 2024-04-26T18:13:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setdirlayout_latency": { 2024-04-26T18:13:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setfilelock_latency": { 2024-04-26T18:13:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setlayout_latency": { 2024-04-26T18:13:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setxattr_latency": { 2024-04-26T18:13:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "req_symlink_latency": { 2024-04-26T18:13:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "req_unlink_latency": { 2024-04-26T18:13:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 175, 2024-04-26T18:13:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.436142962, 2024-04-26T18:13:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.002492245 2024-04-26T18:13:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "cap_revoke_eviction": 0, 2024-04-26T18:13:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "cap_acquisition_throttle": 0, 2024-04-26T18:13:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getvxattr_latency": { 2024-04-26T18:13:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_sessions": { 2024-04-26T18:13:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "session_count": 2, 2024-04-26T18:13:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "session_add": 3, 2024-04-26T18:13:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "session_remove": 1, 2024-04-26T18:13:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "sessions_open": 2, 2024-04-26T18:13:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "sessions_stale": 0, 2024-04-26T18:13:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "total_load": 158, 2024-04-26T18:13:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "average_load": 79, 2024-04-26T18:13:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "avg_session_uptime": 977, 2024-04-26T18:13:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "mdthresh_evicted": 0 2024-04-26T18:13:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "mempool": { 2024-04-26T18:13:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_bytes": 240, 2024-04-26T18:13:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_items": 240, 2024-04-26T18:13:03.238 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T18:13:03.238 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_items": 0, 2024-04-26T18:13:03.238 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T18:13:03.238 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_items": 0, 2024-04-26T18:13:03.238 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T18:13:03.238 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T18:13:03.238 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T18:13:03.238 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T18:13:03.238 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T18:13:03.238 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_items": 0, 2024-04-26T18:13:03.238 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T18:13:03.239 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T18:13:03.239 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_bytes": 0, 2024-04-26T18:13:03.239 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_items": 0, 2024-04-26T18:13:03.239 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_bytes": 0, 2024-04-26T18:13:03.239 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_items": 0, 2024-04-26T18:13:03.239 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T18:13:03.239 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T18:13:03.239 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T18:13:03.239 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T18:13:03.239 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T18:13:03.239 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_items": 0, 2024-04-26T18:13:03.239 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_bytes": 0, 2024-04-26T18:13:03.239 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_items": 0, 2024-04-26T18:13:03.239 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T18:13:03.240 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T18:13:03.240 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_bytes": 0, 2024-04-26T18:13:03.240 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_items": 0, 2024-04-26T18:13:03.240 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_bytes": 0, 2024-04-26T18:13:03.240 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_items": 0, 2024-04-26T18:13:03.240 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T18:13:03.240 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_items": 0, 2024-04-26T18:13:03.240 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T18:13:03.240 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_items": 0, 2024-04-26T18:13:03.240 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_bytes": 768144, 2024-04-26T18:13:03.241 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_items": 420, 2024-04-26T18:13:03.241 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_bytes": 0, 2024-04-26T18:13:03.241 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_items": 0, 2024-04-26T18:13:03.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bytes": 0, 2024-04-26T18:13:03.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_items": 0, 2024-04-26T18:13:03.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_bytes": 0, 2024-04-26T18:13:03.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_items": 0, 2024-04-26T18:13:03.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_bytes": 0, 2024-04-26T18:13:03.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_items": 0, 2024-04-26T18:13:03.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_bytes": 8352, 2024-04-26T18:13:03.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_items": 189, 2024-04-26T18:13:03.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T18:13:03.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_items": 0, 2024-04-26T18:13:03.241 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_bytes": 0, 2024-04-26T18:13:03.242 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_items": 0, 2024-04-26T18:13:03.242 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_bytes": 1174485, 2024-04-26T18:13:03.242 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_items": 46138, 2024-04-26T18:13:03.242 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_bytes": 0, 2024-04-26T18:13:03.242 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_items": 0, 2024-04-26T18:13:03.242 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_bytes": 0, 2024-04-26T18:13:03.242 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_items": 0 2024-04-26T18:13:03.242 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.242 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter": { 2024-04-26T18:13:03.242 INFO:teuthology.orchestra.run.smithi097.stdout: "op_active": 0, 2024-04-26T18:13:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: "op_laggy": 0, 2024-04-26T18:13:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: "op_send": 2620, 2024-04-26T18:13:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: "op_send_bytes": 30734504, 2024-04-26T18:13:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: "op_resend": 33, 2024-04-26T18:13:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: "op_reply": 2620, 2024-04-26T18:13:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: "op_latency": { 2024-04-26T18:13:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 2620, 2024-04-26T18:13:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 10.463824854, 2024-04-26T18:13:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.003993826 2024-04-26T18:13:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: "op_inflight": 0, 2024-04-26T18:13:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: "oplen_avg": { 2024-04-26T18:13:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 2620, 2024-04-26T18:13:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 2893 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: "op": 2620, 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: "op_r": 15, 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: "op_w": 2605, 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: "op_rmw": 0, 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: "op_pg": 0, 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_stat": 0, 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_create": 10, 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_read": 7, 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_write": 2230, 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_writefull": 74, 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_writesame": 0, 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_append": 0, 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_zero": 0, 2024-04-26T18:13:03.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_truncate": 4, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_delete": 51, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_mapext": 0, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_sparse_read": 0, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_clonerange": 0, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_getxattr": 0, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_setxattr": 20, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_cmpxattr": 0, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_rmxattr": 0, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_resetxattrs": 0, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_call": 0, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_watch": 0, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_notify": 0, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_pgls": 0, 2024-04-26T18:13:03.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_pgls_filter": 0, 2024-04-26T18:13:03.246 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_other": 0, 2024-04-26T18:13:03.246 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_active": 0, 2024-04-26T18:13:03.246 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_send": 0, 2024-04-26T18:13:03.246 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_resend": 0, 2024-04-26T18:13:03.246 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_ping": 0, 2024-04-26T18:13:03.246 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_active": 0, 2024-04-26T18:13:03.246 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_send": 0, 2024-04-26T18:13:03.246 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_resend": 0, 2024-04-26T18:13:03.246 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_active": 0, 2024-04-26T18:13:03.246 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_send": 0, 2024-04-26T18:13:03.247 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_resend": 0, 2024-04-26T18:13:03.247 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_active": 0, 2024-04-26T18:13:03.247 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_send": 0, 2024-04-26T18:13:03.247 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_resend": 0, 2024-04-26T18:13:03.247 INFO:teuthology.orchestra.run.smithi097.stdout: "command_active": 0, 2024-04-26T18:13:03.247 INFO:teuthology.orchestra.run.smithi097.stdout: "command_send": 0, 2024-04-26T18:13:03.247 INFO:teuthology.orchestra.run.smithi097.stdout: "command_resend": 0, 2024-04-26T18:13:03.247 INFO:teuthology.orchestra.run.smithi097.stdout: "map_epoch": 111, 2024-04-26T18:13:03.247 INFO:teuthology.orchestra.run.smithi097.stdout: "map_full": 0, 2024-04-26T18:13:03.247 INFO:teuthology.orchestra.run.smithi097.stdout: "map_inc": 23, 2024-04-26T18:13:03.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_sessions": 12, 2024-04-26T18:13:03.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_session_open": 12, 2024-04-26T18:13:03.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_session_close": 0, 2024-04-26T18:13:03.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_laggy": 0, 2024-04-26T18:13:03.248 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_wr": 402, 2024-04-26T18:13:03.248 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_rd": 16, 2024-04-26T18:13:03.248 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_del": 79 2024-04-26T18:13:03.248 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.248 INFO:teuthology.orchestra.run.smithi097.stdout: "oft": { 2024-04-26T18:13:03.248 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_objs": 1, 2024-04-26T18:13:03.248 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_kv_pairs": 0, 2024-04-26T18:13:03.248 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_updates": 0, 2024-04-26T18:13:03.248 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_removes": 0 2024-04-26T18:13:03.248 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.248 INFO:teuthology.orchestra.run.smithi097.stdout: "osd-slow-ops": { 2024-04-26T18:13:03.248 INFO:teuthology.orchestra.run.smithi097.stdout: "slow_ops_count": 1203 2024-04-26T18:13:03.248 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.248 INFO:teuthology.orchestra.run.smithi097.stdout: "purge_queue": { 2024-04-26T18:13:03.249 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_ops": 0, 2024-04-26T18:13:03.249 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_ops_high_water": 17, 2024-04-26T18:13:03.249 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing": 0, 2024-04-26T18:13:03.249 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_high_water": 10, 2024-04-26T18:13:03.249 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executed_ops": 32, 2024-04-26T18:13:03.249 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executed": 18, 2024-04-26T18:13:03.249 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_item_in_journal": 0 2024-04-26T18:13:03.249 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.249 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T18:13:03.249 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:13:03.249 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 104857600, 2024-04-26T18:13:03.249 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:13:03.249 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 17608, 2024-04-26T18:13:03.249 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 5406772, 2024-04-26T18:13:03.250 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:03.250 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 17608, 2024-04-26T18:13:03.250 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-04-26T18:13:03.250 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-04-26T18:13:03.250 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 17608, 2024-04-26T18:13:03.250 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 5406772, 2024-04-26T18:13:03.250 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:13:03.250 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.250 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.250 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.251 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:03.251 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.251 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-objecter_bytes": { 2024-04-26T18:13:03.251 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:13:03.251 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 104857600, 2024-04-26T18:13:03.251 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:13:03.251 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-04-26T18:13:03.252 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-04-26T18:13:03.252 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:03.252 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:03.252 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 2620, 2024-04-26T18:13:03.252 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 30736250, 2024-04-26T18:13:03.252 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 2620, 2024-04-26T18:13:03.252 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 30736250, 2024-04-26T18:13:03.252 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:13:03.252 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.252 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.252 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.252 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:03.253 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.253 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-objecter_ops": { 2024-04-26T18:13:03.253 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:13:03.253 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 1024, 2024-04-26T18:13:03.253 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:13:03.253 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-04-26T18:13:03.253 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-04-26T18:13:03.253 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:03.253 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:03.253 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 2620, 2024-04-26T18:13:03.253 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 2620, 2024-04-26T18:13:03.253 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 2620, 2024-04-26T18:13:03.253 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 2620, 2024-04-26T18:13:03.253 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:13:03.254 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.254 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.254 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.254 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:03.254 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.254 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-write_buf_throttle": { 2024-04-26T18:13:03.254 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:13:03.254 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 3758096384, 2024-04-26T18:13:03.254 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:13:03.254 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 18, 2024-04-26T18:13:03.254 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 1874, 2024-04-26T18:13:03.254 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:03.255 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 18, 2024-04-26T18:13:03.255 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-04-26T18:13:03.255 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-04-26T18:13:03.255 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 7, 2024-04-26T18:13:03.255 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 1874, 2024-04-26T18:13:03.255 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:13:03.255 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.255 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.255 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.255 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:03.255 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:03.256 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-write_buf_throttle-0x556a0e7bc500": { 2024-04-26T18:13:03.256 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:13:03.256 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 3758096384, 2024-04-26T18:13:03.256 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:13:03.256 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 4663, 2024-04-26T18:13:03.256 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 30120310, 2024-04-26T18:13:03.256 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:03.256 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 4663, 2024-04-26T18:13:03.256 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-04-26T18:13:03.256 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-04-26T18:13:03.256 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 2223, 2024-04-26T18:13:03.256 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 30120310, 2024-04-26T18:13:03.256 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:13:03.257 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:03.257 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:03.257 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:03.257 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:03.257 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:03.257 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-26T18:13:03.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:03 smithi097 ceph-mon[32419]: pgmap v519: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 121 KiB/s wr, 30 op/s 2024-04-26T18:13:03.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:03 smithi186 ceph-mon[32797]: pgmap v519: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 121 KiB/s wr, 30 op/s 2024-04-26T18:13:03.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:03 smithi031 ceph-mon[29164]: pgmap v519: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 121 KiB/s wr, 30 op/s 2024-04-26T18:13:03.889 INFO:tasks.check_counter:Daemon mds.h mds.exported=30 2024-04-26T18:13:03.889 INFO:tasks.check_counter:Daemon mds.h mds.imported=2 2024-04-26T18:13:03.889 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-26T18:13:03.889 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-26T18:13:03.889 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T18:13:03.889 DEBUG:tasks.check_counter:Getting stats from k 2024-04-26T18:13:03.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-26T18:13:04.130 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T18:13:05.526 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:05 smithi097 ceph-mon[32419]: pgmap v520: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 96 KiB/s wr, 25 op/s 2024-04-26T18:13:05.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:05 smithi186 ceph-mon[32797]: pgmap v520: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 96 KiB/s wr, 25 op/s 2024-04-26T18:13:05.788 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-26T18:13:05.788 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T18:13:05.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 0, 2024-04-26T18:13:05.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 2, 2024-04-26T18:13:05.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 0, 2024-04-26T18:13:05.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 0, 2024-04-26T18:13:05.788 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 4, 2024-04-26T18:13:05.789 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 0, 2024-04-26T18:13:05.789 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.001186651, 2024-04-26T18:13:05.789 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T18:13:05.789 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.000220307, 2024-04-26T18:13:05.789 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T18:13:05.789 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:05.789 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:05.789 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:05.789 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:05.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:05.789 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:05.789 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:05.789 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:05.790 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:05.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:05.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T18:13:05.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T18:13:05.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:05.790 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T18:13:05.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 7, 2024-04-26T18:13:05.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 257, 2024-04-26T18:13:05.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 3424, 2024-04-26T18:13:05.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 281056, 2024-04-26T18:13:05.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 6, 2024-04-26T18:13:05.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 3, 2024-04-26T18:13:05.790 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.016186006, 2024-04-26T18:13:05.791 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.009338312, 2024-04-26T18:13:05.791 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.004185185, 2024-04-26T18:13:05.791 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T18:13:05.791 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:05.791 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 255, 2024-04-26T18:13:05.791 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.031007918, 2024-04-26T18:13:05.791 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000121599 2024-04-26T18:13:05.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:05.791 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:05.791 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:05.791 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:05.791 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:05.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:05.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 3424, 2024-04-26T18:13:05.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 281056 2024-04-26T18:13:05.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:05.792 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T18:13:05.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 343, 2024-04-26T18:13:05.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 332, 2024-04-26T18:13:05.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 1624032, 2024-04-26T18:13:05.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 205360, 2024-04-26T18:13:05.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 2, 2024-04-26T18:13:05.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 1, 2024-04-26T18:13:05.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.056727224, 2024-04-26T18:13:05.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.017627128, 2024-04-26T18:13:05.792 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.039654040, 2024-04-26T18:13:05.793 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.007529334, 2024-04-26T18:13:05.793 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:05.793 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 331, 2024-04-26T18:13:05.793 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.014998307, 2024-04-26T18:13:05.793 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000045312 2024-04-26T18:13:05.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:05.793 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:05.793 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:05.793 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:05.793 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:05.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:05.793 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 1624032, 2024-04-26T18:13:05.793 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 205360 2024-04-26T18:13:05.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:05.794 INFO:teuthology.orchestra.run.smithi097.stdout: "cct": { 2024-04-26T18:13:05.794 INFO:teuthology.orchestra.run.smithi097.stdout: "total_workers": 0, 2024-04-26T18:13:05.794 INFO:teuthology.orchestra.run.smithi097.stdout: "unhealthy_workers": 0 2024-04-26T18:13:05.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:05.794 INFO:teuthology.orchestra.run.smithi097.stdout: "mempool": { 2024-04-26T18:13:05.794 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_bytes": 0, 2024-04-26T18:13:05.794 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_items": 0, 2024-04-26T18:13:05.794 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T18:13:05.794 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_items": 0, 2024-04-26T18:13:05.794 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T18:13:05.794 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_items": 0, 2024-04-26T18:13:05.794 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T18:13:05.795 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T18:13:05.795 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T18:13:05.795 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T18:13:05.795 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T18:13:05.795 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_items": 0, 2024-04-26T18:13:05.795 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T18:13:05.795 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T18:13:05.795 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_bytes": 0, 2024-04-26T18:13:05.795 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_items": 0, 2024-04-26T18:13:05.795 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_bytes": 0, 2024-04-26T18:13:05.795 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_items": 0, 2024-04-26T18:13:05.795 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T18:13:05.795 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T18:13:05.795 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T18:13:05.796 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T18:13:05.796 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T18:13:05.796 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_items": 0, 2024-04-26T18:13:05.796 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_bytes": 0, 2024-04-26T18:13:05.796 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_items": 0, 2024-04-26T18:13:05.796 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T18:13:05.796 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T18:13:05.796 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_bytes": 0, 2024-04-26T18:13:05.796 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_items": 0, 2024-04-26T18:13:05.796 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_bytes": 0, 2024-04-26T18:13:05.796 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_items": 0, 2024-04-26T18:13:05.796 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T18:13:05.796 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_items": 0, 2024-04-26T18:13:05.796 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T18:13:05.797 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_items": 0, 2024-04-26T18:13:05.797 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_bytes": 147930, 2024-04-26T18:13:05.797 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_items": 28, 2024-04-26T18:13:05.797 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_bytes": 0, 2024-04-26T18:13:05.797 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_items": 0, 2024-04-26T18:13:05.797 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bytes": 0, 2024-04-26T18:13:05.797 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_items": 0, 2024-04-26T18:13:05.797 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_bytes": 0, 2024-04-26T18:13:05.797 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_items": 0, 2024-04-26T18:13:05.797 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_bytes": 0, 2024-04-26T18:13:05.797 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_items": 0, 2024-04-26T18:13:05.797 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_bytes": 0, 2024-04-26T18:13:05.797 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_items": 0, 2024-04-26T18:13:05.798 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T18:13:05.798 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_items": 0, 2024-04-26T18:13:05.798 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_bytes": 0, 2024-04-26T18:13:05.798 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_items": 0, 2024-04-26T18:13:05.798 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_bytes": 928, 2024-04-26T18:13:05.798 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_items": 2, 2024-04-26T18:13:05.798 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_bytes": 0, 2024-04-26T18:13:05.798 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_items": 0, 2024-04-26T18:13:05.798 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_bytes": 0, 2024-04-26T18:13:05.798 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_items": 0 2024-04-26T18:13:05.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-26T18:13:05.798 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T18:13:05.798 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-04-26T18:13:05.799 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 104857600, 2024-04-26T18:13:05.799 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-04-26T18:13:05.799 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 346, 2024-04-26T18:13:05.799 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 1575622, 2024-04-26T18:13:05.799 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:05.799 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 346, 2024-04-26T18:13:05.799 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-04-26T18:13:05.799 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-04-26T18:13:05.799 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 346, 2024-04-26T18:13:05.799 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 1575622, 2024-04-26T18:13:05.799 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-04-26T18:13:05.799 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-04-26T18:13:05.799 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-04-26T18:13:05.799 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-04-26T18:13:05.800 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:05.800 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-26T18:13:05.800 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-26T18:13:05.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:05 smithi031 ceph-mon[29164]: pgmap v520: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 96 KiB/s wr, 25 op/s 2024-04-26T18:13:06.477 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-26T18:13:06.477 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-26T18:13:06.477 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-26T18:13:06.477 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-26T18:13:06.477 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-26T18:13:06.477 DEBUG:tasks.check_counter:Getting stats from c 2024-04-26T18:13:06.477 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-26T18:13:06.729 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T18:13:07.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:07 smithi186 ceph-mon[32797]: pgmap v521: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 95 KiB/s wr, 25 op/s 2024-04-26T18:13:07.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:07 smithi097 ceph-mon[32419]: pgmap v521: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 95 KiB/s wr, 25 op/s 2024-04-26T18:13:07.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:07 smithi031 ceph-mon[29164]: pgmap v521: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 95 KiB/s wr, 25 op/s 2024-04-26T18:13:08.425 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:08 smithi186 ceph-mon[32797]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T18:13:08.551 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-26T18:13:08.551 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T18:13:08.551 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 588, 2024-04-26T18:13:08.551 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 590, 2024-04-26T18:13:08.551 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 845094, 2024-04-26T18:13:08.551 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 186994, 2024-04-26T18:13:08.551 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 6, 2024-04-26T18:13:08.552 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 1, 2024-04-26T18:13:08.552 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.055299353, 2024-04-26T18:13:08.552 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.015770722, 2024-04-26T18:13:08.552 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.015013737, 2024-04-26T18:13:08.552 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.018313135, 2024-04-26T18:13:08.552 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:08.552 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 588, 2024-04-26T18:13:08.552 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.027291492, 2024-04-26T18:13:08.552 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000046414 2024-04-26T18:13:08.552 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.553 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:08.553 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.553 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.553 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.553 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.553 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T18:13:08.553 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T18:13:08.553 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.553 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T18:13:08.553 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 572, 2024-04-26T18:13:08.554 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 563, 2024-04-26T18:13:08.554 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 333851, 2024-04-26T18:13:08.554 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 215854, 2024-04-26T18:13:08.554 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 4, 2024-04-26T18:13:08.554 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 3, 2024-04-26T18:13:08.554 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.051472328, 2024-04-26T18:13:08.554 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.017473036, 2024-04-26T18:13:08.554 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.015123422, 2024-04-26T18:13:08.554 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.015802842, 2024-04-26T18:13:08.554 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:08.554 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 563, 2024-04-26T18:13:08.554 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.025999203, 2024-04-26T18:13:08.555 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000046179 2024-04-26T18:13:08.555 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.555 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:08.555 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.555 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.555 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.555 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.555 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 231552, 2024-04-26T18:13:08.555 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 85408 2024-04-26T18:13:08.555 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.555 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T18:13:08.555 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 638, 2024-04-26T18:13:08.555 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 739, 2024-04-26T18:13:08.556 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 2962936, 2024-04-26T18:13:08.556 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 466567, 2024-04-26T18:13:08.556 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 9, 2024-04-26T18:13:08.556 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 18446744073709551615, 2024-04-26T18:13:08.556 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.053000807, 2024-04-26T18:13:08.556 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.019894353, 2024-04-26T18:13:08.556 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.014434116, 2024-04-26T18:13:08.556 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.014159710, 2024-04-26T18:13:08.556 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:08.556 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 737, 2024-04-26T18:13:08.556 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.027701916, 2024-04-26T18:13:08.556 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000037587 2024-04-26T18:13:08.557 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.557 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:08.557 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.557 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.557 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.557 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.558 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-26T18:13:08.558 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 264000 2024-04-26T18:13:08.558 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.558 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-04-26T18:13:08.558 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 1, 2024-04-26T18:13:08.558 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-04-26T18:13:08.559 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.559 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-MDSRank": { 2024-04-26T18:13:08.559 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-26T18:13:08.559 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-26T18:13:08.559 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 2748, 2024-04-26T18:13:08.559 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.064028084, 2024-04-26T18:13:08.559 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000023299 2024-04-26T18:13:08.559 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:08.559 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.559 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-PurgeQueue": { 2024-04-26T18:13:08.559 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-26T18:13:08.559 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-26T18:13:08.559 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.560 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.560 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.560 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:08.560 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.560 INFO:teuthology.orchestra.run.smithi186.stdout: "mds": { 2024-04-26T18:13:08.560 INFO:teuthology.orchestra.run.smithi186.stdout: "request": 0, 2024-04-26T18:13:08.560 INFO:teuthology.orchestra.run.smithi186.stdout: "reply": 0, 2024-04-26T18:13:08.560 INFO:teuthology.orchestra.run.smithi186.stdout: "reply_latency": { 2024-04-26T18:13:08.560 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.560 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.560 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.560 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.560 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_reply": 0, 2024-04-26T18:13:08.560 INFO:teuthology.orchestra.run.smithi186.stdout: "forward": 0, 2024-04-26T18:13:08.561 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_complete": 0, 2024-04-26T18:13:08.561 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_keys": 0, 2024-04-26T18:13:08.561 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_commit": 0, 2024-04-26T18:13:08.561 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_split": 0, 2024-04-26T18:13:08.561 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_merge": 0, 2024-04-26T18:13:08.561 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes": 436, 2024-04-26T18:13:08.561 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_top": 13, 2024-04-26T18:13:08.561 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_bottom": 423, 2024-04-26T18:13:08.561 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pin_tail": 0, 2024-04-26T18:13:08.561 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pinned": 418, 2024-04-26T18:13:08.561 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_expired": 0, 2024-04-26T18:13:08.561 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_with_caps": 0, 2024-04-26T18:13:08.561 INFO:teuthology.orchestra.run.smithi186.stdout: "caps": 0, 2024-04-26T18:13:08.562 INFO:teuthology.orchestra.run.smithi186.stdout: "subtrees": 17, 2024-04-26T18:13:08.562 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse": 0, 2024-04-26T18:13:08.562 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_hit": 0, 2024-04-26T18:13:08.562 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_forward": 0, 2024-04-26T18:13:08.562 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_discover": 0, 2024-04-26T18:13:08.562 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_dir_fetch": 0, 2024-04-26T18:13:08.562 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_remote_ino": 0, 2024-04-26T18:13:08.562 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_lock": 0, 2024-04-26T18:13:08.562 INFO:teuthology.orchestra.run.smithi186.stdout: "load_cent": 0, 2024-04-26T18:13:08.562 INFO:teuthology.orchestra.run.smithi186.stdout: "q": 0, 2024-04-26T18:13:08.562 INFO:teuthology.orchestra.run.smithi186.stdout: "exported": 0, 2024-04-26T18:13:08.562 INFO:teuthology.orchestra.run.smithi186.stdout: "exported_inodes": 0, 2024-04-26T18:13:08.562 INFO:teuthology.orchestra.run.smithi186.stdout: "imported": 0, 2024-04-26T18:13:08.563 INFO:teuthology.orchestra.run.smithi186.stdout: "imported_inodes": 0, 2024-04-26T18:13:08.563 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_dir_fetch": 0, 2024-04-26T18:13:08.563 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_backtrace_fetch": 0, 2024-04-26T18:13:08.563 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_peer_discover": 0, 2024-04-26T18:13:08.563 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rfiles": 1008, 2024-04-26T18:13:08.563 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rbytes": 1353473565, 2024-04-26T18:13:08.563 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rsnaps": 7, 2024-04-26T18:13:08.563 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T18:13:08.563 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_set_tag": 0, 2024-04-26T18:13:08.563 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T18:13:08.563 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_inotable_repaired": 0, 2024-04-26T18:13:08.563 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_inodes": 0, 2024-04-26T18:13:08.563 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T18:13:08.564 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T18:13:08.564 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_file_inodes": 0, 2024-04-26T18:13:08.564 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_inode_file_caps": 0, 2024-04-26T18:13:08.564 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T18:13:08.564 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_grant": 0, 2024-04-26T18:13:08.564 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T18:13:08.564 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T18:13:08.564 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T18:13:08.564 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps": 0, 2024-04-26T18:13:08.564 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps_dirty": 0, 2024-04-26T18:13:08.564 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_cap_release": 0, 2024-04-26T18:13:08.564 INFO:teuthology.orchestra.run.smithi186.stdout: "process_request_cap_release": 0 2024-04-26T18:13:08.564 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.565 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_cache": { 2024-04-26T18:13:08.565 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update": 0, 2024-04-26T18:13:08.565 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update_receipt": 0, 2024-04-26T18:13:08.565 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_try_discover": 0, 2024-04-26T18:13:08.565 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_send_discover": 0, 2024-04-26T18:13:08.565 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_handle_discover": 0, 2024-04-26T18:13:08.565 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays": 0, 2024-04-26T18:13:08.565 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_delayed": 0, 2024-04-26T18:13:08.565 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_enqueuing": 0, 2024-04-26T18:13:08.565 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_created": 0, 2024-04-26T18:13:08.565 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_enqueued": 0, 2024-04-26T18:13:08.565 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_reintegrated": 0, 2024-04-26T18:13:08.565 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_migrated": 0, 2024-04-26T18:13:08.565 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_processing": 0, 2024-04-26T18:13:08.566 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_enqueued": 0, 2024-04-26T18:13:08.566 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_prioritized": 0, 2024-04-26T18:13:08.566 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_started": 0, 2024-04-26T18:13:08.566 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_completed": 0, 2024-04-26T18:13:08.566 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_path": 0, 2024-04-26T18:13:08.566 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_inode": 0, 2024-04-26T18:13:08.566 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T18:13:08.566 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_exportdir": 0, 2024-04-26T18:13:08.566 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_flush": 0, 2024-04-26T18:13:08.566 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragmentdir": 0, 2024-04-26T18:13:08.566 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragstats": 0, 2024-04-26T18:13:08.566 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_inodestats": 0 2024-04-26T18:13:08.566 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_log": { 2024-04-26T18:13:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: "evlrg": 0, 2024-04-26T18:13:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: "evadd": 0, 2024-04-26T18:13:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: "evex": 0, 2024-04-26T18:13:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: "evtrm": 0, 2024-04-26T18:13:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: "ev": 1609, 2024-04-26T18:13:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: "evexg": 0, 2024-04-26T18:13:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: "evexd": 0, 2024-04-26T18:13:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: "segadd": 0, 2024-04-26T18:13:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: "segex": 0, 2024-04-26T18:13:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: "segtrm": 0, 2024-04-26T18:13:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: "seg": 3, 2024-04-26T18:13:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: "segmjr": 1, 2024-04-26T18:13:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: "segexg": 0, 2024-04-26T18:13:08.568 INFO:teuthology.orchestra.run.smithi186.stdout: "segexd": 0, 2024-04-26T18:13:08.568 INFO:teuthology.orchestra.run.smithi186.stdout: "expos": 7738070, 2024-04-26T18:13:08.568 INFO:teuthology.orchestra.run.smithi186.stdout: "wrpos": 11187369, 2024-04-26T18:13:08.568 INFO:teuthology.orchestra.run.smithi186.stdout: "rdpos": 11174249, 2024-04-26T18:13:08.568 INFO:teuthology.orchestra.run.smithi186.stdout: "jlat": { 2024-04-26T18:13:08.568 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.568 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.568 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.568 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.569 INFO:teuthology.orchestra.run.smithi186.stdout: "replayed": 1609 2024-04-26T18:13:08.569 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.569 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_mem": { 2024-04-26T18:13:08.569 INFO:teuthology.orchestra.run.smithi186.stdout: "ino": 222, 2024-04-26T18:13:08.569 INFO:teuthology.orchestra.run.smithi186.stdout: "ino+": 382, 2024-04-26T18:13:08.569 INFO:teuthology.orchestra.run.smithi186.stdout: "ino-": 160, 2024-04-26T18:13:08.569 INFO:teuthology.orchestra.run.smithi186.stdout: "dir": 28, 2024-04-26T18:13:08.570 INFO:teuthology.orchestra.run.smithi186.stdout: "dir+": 28, 2024-04-26T18:13:08.570 INFO:teuthology.orchestra.run.smithi186.stdout: "dir-": 0, 2024-04-26T18:13:08.570 INFO:teuthology.orchestra.run.smithi186.stdout: "dn": 436, 2024-04-26T18:13:08.570 INFO:teuthology.orchestra.run.smithi186.stdout: "dn+": 436, 2024-04-26T18:13:08.570 INFO:teuthology.orchestra.run.smithi186.stdout: "dn-": 0, 2024-04-26T18:13:08.570 INFO:teuthology.orchestra.run.smithi186.stdout: "cap": 0, 2024-04-26T18:13:08.570 INFO:teuthology.orchestra.run.smithi186.stdout: "cap+": 0, 2024-04-26T18:13:08.570 INFO:teuthology.orchestra.run.smithi186.stdout: "cap-": 0, 2024-04-26T18:13:08.570 INFO:teuthology.orchestra.run.smithi186.stdout: "rss": 51452, 2024-04-26T18:13:08.570 INFO:teuthology.orchestra.run.smithi186.stdout: "heap": 215416 2024-04-26T18:13:08.570 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.570 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_server": { 2024-04-26T18:13:08.571 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_client_request": 0, 2024-04-26T18:13:08.571 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_server_request": 0, 2024-04-26T18:13:08.571 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_request": 0, 2024-04-26T18:13:08.571 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_session": 0, 2024-04-26T18:13:08.571 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_peer_request": 0, 2024-04-26T18:13:08.571 INFO:teuthology.orchestra.run.smithi186.stdout: "req_create_latency": { 2024-04-26T18:13:08.571 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.571 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.571 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.571 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.571 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getattr_latency": { 2024-04-26T18:13:08.571 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.572 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.572 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.572 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.572 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getfilelock_latency": { 2024-04-26T18:13:08.572 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.572 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.572 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.572 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.572 INFO:teuthology.orchestra.run.smithi186.stdout: "req_link_latency": { 2024-04-26T18:13:08.572 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.572 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.572 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.572 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.572 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookup_latency": { 2024-04-26T18:13:08.573 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.573 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.573 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.573 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.573 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookuphash_latency": { 2024-04-26T18:13:08.573 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.573 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.573 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.573 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.573 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupino_latency": { 2024-04-26T18:13:08.573 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.573 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.574 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.574 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.574 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupname_latency": { 2024-04-26T18:13:08.574 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.574 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.574 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.574 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.574 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupparent_latency": { 2024-04-26T18:13:08.574 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.574 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.574 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.574 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.575 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupsnap_latency": { 2024-04-26T18:13:08.575 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.575 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.575 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.575 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.575 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lssnap_latency": { 2024-04-26T18:13:08.575 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.575 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.575 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.575 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.576 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mkdir_latency": { 2024-04-26T18:13:08.576 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.576 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.576 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.576 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.576 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mknod_latency": { 2024-04-26T18:13:08.576 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.576 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.576 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.577 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.577 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mksnap_latency": { 2024-04-26T18:13:08.577 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.577 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.577 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.577 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.577 INFO:teuthology.orchestra.run.smithi186.stdout: "req_open_latency": { 2024-04-26T18:13:08.577 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.577 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.577 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.577 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.577 INFO:teuthology.orchestra.run.smithi186.stdout: "req_readdir_latency": { 2024-04-26T18:13:08.577 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.578 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.578 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.578 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.578 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rename_latency": { 2024-04-26T18:13:08.578 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.578 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.578 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.578 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.578 INFO:teuthology.orchestra.run.smithi186.stdout: "req_renamesnap_latency": { 2024-04-26T18:13:08.578 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.578 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.578 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.579 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.579 INFO:teuthology.orchestra.run.smithi186.stdout: "req_snapdiff_latency": { 2024-04-26T18:13:08.579 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.579 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.579 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.579 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.580 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmdir_latency": { 2024-04-26T18:13:08.580 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.580 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.580 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.580 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.580 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmsnap_latency": { 2024-04-26T18:13:08.580 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.580 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.580 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.580 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.580 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmxattr_latency": { 2024-04-26T18:13:08.580 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.580 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.581 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.581 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.581 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setattr_latency": { 2024-04-26T18:13:08.581 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.581 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.581 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.581 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.581 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setdirlayout_latency": { 2024-04-26T18:13:08.581 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.581 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.581 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.581 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.581 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setfilelock_latency": { 2024-04-26T18:13:08.582 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.582 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.582 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.582 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.582 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setlayout_latency": { 2024-04-26T18:13:08.582 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.582 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.582 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.582 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.582 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setxattr_latency": { 2024-04-26T18:13:08.582 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.582 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.583 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.583 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.583 INFO:teuthology.orchestra.run.smithi186.stdout: "req_symlink_latency": { 2024-04-26T18:13:08.583 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.583 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.583 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.583 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.583 INFO:teuthology.orchestra.run.smithi186.stdout: "req_unlink_latency": { 2024-04-26T18:13:08.583 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.583 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.584 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.584 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.584 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_revoke_eviction": 0, 2024-04-26T18:13:08.584 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_acquisition_throttle": 0, 2024-04-26T18:13:08.584 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getvxattr_latency": { 2024-04-26T18:13:08.584 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.584 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.584 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.584 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:08.584 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.584 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_sessions": { 2024-04-26T18:13:08.584 INFO:teuthology.orchestra.run.smithi186.stdout: "session_count": 2, 2024-04-26T18:13:08.584 INFO:teuthology.orchestra.run.smithi186.stdout: "session_add": 2, 2024-04-26T18:13:08.585 INFO:teuthology.orchestra.run.smithi186.stdout: "session_remove": 0, 2024-04-26T18:13:08.585 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_open": 0, 2024-04-26T18:13:08.585 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_stale": 0, 2024-04-26T18:13:08.585 INFO:teuthology.orchestra.run.smithi186.stdout: "total_load": 0, 2024-04-26T18:13:08.585 INFO:teuthology.orchestra.run.smithi186.stdout: "average_load": 0, 2024-04-26T18:13:08.585 INFO:teuthology.orchestra.run.smithi186.stdout: "avg_session_uptime": 2196, 2024-04-26T18:13:08.585 INFO:teuthology.orchestra.run.smithi186.stdout: "mdthresh_evicted": 0 2024-04-26T18:13:08.585 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.585 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-04-26T18:13:08.585 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-04-26T18:13:08.585 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-04-26T18:13:08.585 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T18:13:08.585 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-04-26T18:13:08.586 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T18:13:08.586 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-04-26T18:13:08.586 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T18:13:08.586 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T18:13:08.586 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T18:13:08.586 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T18:13:08.586 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T18:13:08.586 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-04-26T18:13:08.586 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T18:13:08.586 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T18:13:08.586 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-04-26T18:13:08.586 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-04-26T18:13:08.587 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-04-26T18:13:08.587 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-04-26T18:13:08.587 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T18:13:08.587 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T18:13:08.587 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T18:13:08.587 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T18:13:08.587 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T18:13:08.587 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-04-26T18:13:08.587 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-04-26T18:13:08.587 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-04-26T18:13:08.587 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T18:13:08.587 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T18:13:08.587 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-04-26T18:13:08.588 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-04-26T18:13:08.588 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-04-26T18:13:08.588 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-04-26T18:13:08.588 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T18:13:08.588 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-04-26T18:13:08.588 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T18:13:08.588 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-04-26T18:13:08.588 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 200040, 2024-04-26T18:13:08.588 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 331, 2024-04-26T18:13:08.589 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-04-26T18:13:08.589 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-04-26T18:13:08.589 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-04-26T18:13:08.589 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-04-26T18:13:08.589 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-04-26T18:13:08.589 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-04-26T18:13:08.590 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-04-26T18:13:08.590 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-04-26T18:13:08.590 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 8352, 2024-04-26T18:13:08.590 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 189, 2024-04-26T18:13:08.590 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T18:13:08.591 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-04-26T18:13:08.591 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-04-26T18:13:08.591 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-04-26T18:13:08.591 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 977938, 2024-04-26T18:13:08.591 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 14646, 2024-04-26T18:13:08.591 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-04-26T18:13:08.592 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-04-26T18:13:08.592 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-04-26T18:13:08.592 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-04-26T18:13:08.592 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.592 INFO:teuthology.orchestra.run.smithi186.stdout: "objecter": { 2024-04-26T18:13:08.592 INFO:teuthology.orchestra.run.smithi186.stdout: "op_active": 0, 2024-04-26T18:13:08.593 INFO:teuthology.orchestra.run.smithi186.stdout: "op_laggy": 0, 2024-04-26T18:13:08.593 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send": 1635, 2024-04-26T18:13:08.593 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send_bytes": 26, 2024-04-26T18:13:08.593 INFO:teuthology.orchestra.run.smithi186.stdout: "op_resend": 0, 2024-04-26T18:13:08.593 INFO:teuthology.orchestra.run.smithi186.stdout: "op_reply": 1635, 2024-04-26T18:13:08.593 INFO:teuthology.orchestra.run.smithi186.stdout: "op_latency": { 2024-04-26T18:13:08.594 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1635, 2024-04-26T18:13:08.594 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 5.927895582, 2024-04-26T18:13:08.594 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.003625624 2024-04-26T18:13:08.594 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.594 INFO:teuthology.orchestra.run.smithi186.stdout: "op_inflight": 0, 2024-04-26T18:13:08.594 INFO:teuthology.orchestra.run.smithi186.stdout: "oplen_avg": { 2024-04-26T18:13:08.594 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1635, 2024-04-26T18:13:08.595 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 1636 2024-04-26T18:13:08.595 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.595 INFO:teuthology.orchestra.run.smithi186.stdout: "op": 1635, 2024-04-26T18:13:08.595 INFO:teuthology.orchestra.run.smithi186.stdout: "op_r": 1635, 2024-04-26T18:13:08.595 INFO:teuthology.orchestra.run.smithi186.stdout: "op_w": 0, 2024-04-26T18:13:08.595 INFO:teuthology.orchestra.run.smithi186.stdout: "op_rmw": 0, 2024-04-26T18:13:08.595 INFO:teuthology.orchestra.run.smithi186.stdout: "op_pg": 0, 2024-04-26T18:13:08.595 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_stat": 966, 2024-04-26T18:13:08.595 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_create": 0, 2024-04-26T18:13:08.595 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_read": 666, 2024-04-26T18:13:08.595 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_write": 0, 2024-04-26T18:13:08.596 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writefull": 0, 2024-04-26T18:13:08.596 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writesame": 0, 2024-04-26T18:13:08.596 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_append": 0, 2024-04-26T18:13:08.596 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_zero": 0, 2024-04-26T18:13:08.596 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_truncate": 0, 2024-04-26T18:13:08.596 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_delete": 0, 2024-04-26T18:13:08.596 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_mapext": 0, 2024-04-26T18:13:08.596 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_sparse_read": 0, 2024-04-26T18:13:08.596 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_clonerange": 0, 2024-04-26T18:13:08.597 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_getxattr": 2, 2024-04-26T18:13:08.597 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_setxattr": 0, 2024-04-26T18:13:08.597 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_cmpxattr": 0, 2024-04-26T18:13:08.597 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_rmxattr": 0, 2024-04-26T18:13:08.597 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_resetxattrs": 0, 2024-04-26T18:13:08.597 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_call": 0, 2024-04-26T18:13:08.597 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_watch": 0, 2024-04-26T18:13:08.597 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_notify": 0, 2024-04-26T18:13:08.597 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T18:13:08.598 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls": 0, 2024-04-26T18:13:08.598 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls_filter": 0, 2024-04-26T18:13:08.598 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_other": 0, 2024-04-26T18:13:08.598 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_active": 0, 2024-04-26T18:13:08.598 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_send": 0, 2024-04-26T18:13:08.598 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_resend": 0, 2024-04-26T18:13:08.598 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_ping": 0, 2024-04-26T18:13:08.598 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_active": 0, 2024-04-26T18:13:08.598 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_send": 0, 2024-04-26T18:13:08.598 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_resend": 0, 2024-04-26T18:13:08.598 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_active": 0, 2024-04-26T18:13:08.598 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_send": 0, 2024-04-26T18:13:08.598 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_resend": 0, 2024-04-26T18:13:08.599 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_active": 0, 2024-04-26T18:13:08.599 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_send": 0, 2024-04-26T18:13:08.599 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_resend": 0, 2024-04-26T18:13:08.599 INFO:teuthology.orchestra.run.smithi186.stdout: "command_active": 0, 2024-04-26T18:13:08.599 INFO:teuthology.orchestra.run.smithi186.stdout: "command_send": 0, 2024-04-26T18:13:08.599 INFO:teuthology.orchestra.run.smithi186.stdout: "command_resend": 0, 2024-04-26T18:13:08.599 INFO:teuthology.orchestra.run.smithi186.stdout: "map_epoch": 112, 2024-04-26T18:13:08.599 INFO:teuthology.orchestra.run.smithi186.stdout: "map_full": 0, 2024-04-26T18:13:08.599 INFO:teuthology.orchestra.run.smithi186.stdout: "map_inc": 15, 2024-04-26T18:13:08.599 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_sessions": 5, 2024-04-26T18:13:08.599 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_open": 5, 2024-04-26T18:13:08.599 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_close": 0, 2024-04-26T18:13:08.599 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_laggy": 0, 2024-04-26T18:13:08.600 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_wr": 0, 2024-04-26T18:13:08.600 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_rd": 2, 2024-04-26T18:13:08.600 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_del": 0 2024-04-26T18:13:08.600 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.600 INFO:teuthology.orchestra.run.smithi186.stdout: "oft": { 2024-04-26T18:13:08.600 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_objs": 0, 2024-04-26T18:13:08.600 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_kv_pairs": 0, 2024-04-26T18:13:08.601 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_updates": 0, 2024-04-26T18:13:08.601 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_removes": 0 2024-04-26T18:13:08.601 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.601 INFO:teuthology.orchestra.run.smithi186.stdout: "osd-slow-ops": { 2024-04-26T18:13:08.601 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_ops_count": 0 2024-04-26T18:13:08.601 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.601 INFO:teuthology.orchestra.run.smithi186.stdout: "purge_queue": { 2024-04-26T18:13:08.601 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops": 0, 2024-04-26T18:13:08.601 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T18:13:08.601 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing": 0, 2024-04-26T18:13:08.601 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_high_water": 0, 2024-04-26T18:13:08.601 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed_ops": 0, 2024-04-26T18:13:08.602 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed": 0, 2024-04-26T18:13:08.602 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_item_in_journal": 0 2024-04-26T18:13:08.602 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.602 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T18:13:08.602 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:08.602 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-26T18:13:08.602 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:08.602 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 1794, 2024-04-26T18:13:08.603 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 3969423, 2024-04-26T18:13:08.603 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:08.603 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 1794, 2024-04-26T18:13:08.603 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T18:13:08.603 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T18:13:08.603 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 1794, 2024-04-26T18:13:08.604 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 3969423, 2024-04-26T18:13:08.604 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:08.604 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.604 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.604 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.604 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:08.605 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.605 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_bytes": { 2024-04-26T18:13:08.605 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:08.605 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-26T18:13:08.605 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:08.605 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T18:13:08.606 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T18:13:08.606 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:08.606 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:08.606 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 1635, 2024-04-26T18:13:08.606 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 3451031, 2024-04-26T18:13:08.607 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 1635, 2024-04-26T18:13:08.607 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 3451031, 2024-04-26T18:13:08.607 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:08.607 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.607 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.607 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.608 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:08.608 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.608 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_ops": { 2024-04-26T18:13:08.608 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:08.608 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 1024, 2024-04-26T18:13:08.609 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:08.609 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T18:13:08.609 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T18:13:08.609 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:08.609 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:08.609 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 1635, 2024-04-26T18:13:08.610 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 1635, 2024-04-26T18:13:08.610 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 1635, 2024-04-26T18:13:08.610 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 1635, 2024-04-26T18:13:08.610 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:08.610 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.610 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.611 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.611 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:08.611 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.611 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle": { 2024-04-26T18:13:08.611 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:08.612 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-26T18:13:08.612 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:08.612 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T18:13:08.612 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T18:13:08.612 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:08.612 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:08.612 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T18:13:08.612 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T18:13:08.612 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-04-26T18:13:08.612 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-04-26T18:13:08.612 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:08.613 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.613 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.613 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.613 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:08.613 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:08.613 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle-0x561cf017c780": { 2024-04-26T18:13:08.613 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:08.613 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-26T18:13:08.613 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:08.613 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T18:13:08.613 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T18:13:08.613 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:08.613 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:08.613 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T18:13:08.614 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T18:13:08.614 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-04-26T18:13:08.614 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-04-26T18:13:08.614 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:08.614 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:08.614 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:08.614 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:08.614 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:08.614 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:08.614 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-26T18:13:08.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:08 smithi097 ceph-mon[32419]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T18:13:08.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:08 smithi031 ceph-mon[29164]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T18:13:09.405 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-26T18:13:09.405 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-26T18:13:09.405 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=7 2024-04-26T18:13:09.405 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-26T18:13:09.405 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T18:13:09.405 DEBUG:tasks.check_counter:Getting stats from f 2024-04-26T18:13:09.405 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-26T18:13:09.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:09 smithi097 ceph-mon[32419]: pgmap v523: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 29 op/s 2024-04-26T18:13:09.648 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T18:13:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:09 smithi186 ceph-mon[32797]: pgmap v523: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 29 op/s 2024-04-26T18:13:09.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:09 smithi031 ceph-mon[29164]: pgmap v523: 97 pgs: 97 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 29 op/s 2024-04-26T18:13:11.556 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:11 smithi186 ceph-mon[32797]: pgmap v524: 97 pgs: 97 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 29 op/s 2024-04-26T18:13:11.600 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-26T18:13:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T18:13:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 3768, 2024-04-26T18:13:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 3768, 2024-04-26T18:13:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 8471245, 2024-04-26T18:13:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 1197407, 2024-04-26T18:13:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 6, 2024-04-26T18:13:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 2, 2024-04-26T18:13:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.284354662, 2024-04-26T18:13:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.086475544, 2024-04-26T18:13:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.073227886, 2024-04-26T18:13:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.105421265, 2024-04-26T18:13:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 3767, 2024-04-26T18:13:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.112175538, 2024-04-26T18:13:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000029778 2024-04-26T18:13:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-26T18:13:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T18:13:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T18:13:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 5, 2024-04-26T18:13:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 256, 2024-04-26T18:13:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 1036, 2024-04-26T18:13:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 691962, 2024-04-26T18:13:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 7, 2024-04-26T18:13:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 4, 2024-04-26T18:13:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.014370388, 2024-04-26T18:13:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.008937736, 2024-04-26T18:13:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.002889869, 2024-04-26T18:13:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000100303, 2024-04-26T18:13:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 253, 2024-04-26T18:13:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.023382064, 2024-04-26T18:13:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000092419 2024-04-26T18:13:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T18:13:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 691264 2024-04-26T18:13:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T18:13:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 403, 2024-04-26T18:13:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 360, 2024-04-26T18:13:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 1784508, 2024-04-26T18:13:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 209454, 2024-04-26T18:13:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 4, 2024-04-26T18:13:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 2, 2024-04-26T18:13:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.059132572, 2024-04-26T18:13:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.018781877, 2024-04-26T18:13:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.042483349, 2024-04-26T18:13:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.007448961, 2024-04-26T18:13:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 359, 2024-04-26T18:13:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.024985631, 2024-04-26T18:13:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000069597 2024-04-26T18:13:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 1784240, 2024-04-26T18:13:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 209136 2024-04-26T18:13:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-04-26T18:13:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 1, 2024-04-26T18:13:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-04-26T18:13:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-MDSRank": { 2024-04-26T18:13:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-26T18:13:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-26T18:13:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 6404, 2024-04-26T18:13:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.151734102, 2024-04-26T18:13:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000023693 2024-04-26T18:13:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-PurgeQueue": { 2024-04-26T18:13:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-26T18:13:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-26T18:13:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "mds": { 2024-04-26T18:13:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "request": 0, 2024-04-26T18:13:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "reply": 0, 2024-04-26T18:13:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "reply_latency": { 2024-04-26T18:13:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_reply": 0, 2024-04-26T18:13:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "forward": 0, 2024-04-26T18:13:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_complete": 0, 2024-04-26T18:13:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_keys": 0, 2024-04-26T18:13:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_commit": 0, 2024-04-26T18:13:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_split": 0, 2024-04-26T18:13:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_merge": 0, 2024-04-26T18:13:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes": 1142, 2024-04-26T18:13:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_top": 28, 2024-04-26T18:13:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_bottom": 1114, 2024-04-26T18:13:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pin_tail": 0, 2024-04-26T18:13:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pinned": 1103, 2024-04-26T18:13:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_expired": 0, 2024-04-26T18:13:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_with_caps": 0, 2024-04-26T18:13:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "caps": 0, 2024-04-26T18:13:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "subtrees": 15, 2024-04-26T18:13:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse": 0, 2024-04-26T18:13:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_hit": 0, 2024-04-26T18:13:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_forward": 0, 2024-04-26T18:13:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_discover": 0, 2024-04-26T18:13:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_dir_fetch": 0, 2024-04-26T18:13:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_remote_ino": 0, 2024-04-26T18:13:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_lock": 0, 2024-04-26T18:13:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "load_cent": 0, 2024-04-26T18:13:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "q": 0, 2024-04-26T18:13:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "exported": 0, 2024-04-26T18:13:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "exported_inodes": 0, 2024-04-26T18:13:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "imported": 0, 2024-04-26T18:13:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "imported_inodes": 0, 2024-04-26T18:13:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_dir_fetch": 0, 2024-04-26T18:13:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_backtrace_fetch": 0, 2024-04-26T18:13:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_peer_discover": 0, 2024-04-26T18:13:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rfiles": 0, 2024-04-26T18:13:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rbytes": 0, 2024-04-26T18:13:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rsnaps": 0, 2024-04-26T18:13:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T18:13:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_set_tag": 0, 2024-04-26T18:13:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T18:13:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_inotable_repaired": 0, 2024-04-26T18:13:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_inodes": 0, 2024-04-26T18:13:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T18:13:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T18:13:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_file_inodes": 0, 2024-04-26T18:13:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_inode_file_caps": 0, 2024-04-26T18:13:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T18:13:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_grant": 0, 2024-04-26T18:13:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T18:13:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T18:13:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T18:13:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps": 0, 2024-04-26T18:13:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps_dirty": 0, 2024-04-26T18:13:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_cap_release": 0, 2024-04-26T18:13:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "process_request_cap_release": 0 2024-04-26T18:13:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_cache": { 2024-04-26T18:13:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update": 0, 2024-04-26T18:13:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update_receipt": 0, 2024-04-26T18:13:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_try_discover": 0, 2024-04-26T18:13:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_send_discover": 0, 2024-04-26T18:13:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_handle_discover": 0, 2024-04-26T18:13:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays": 0, 2024-04-26T18:13:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_delayed": 0, 2024-04-26T18:13:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_enqueuing": 0, 2024-04-26T18:13:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_created": 0, 2024-04-26T18:13:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_enqueued": 0, 2024-04-26T18:13:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_reintegrated": 0, 2024-04-26T18:13:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_migrated": 0, 2024-04-26T18:13:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_processing": 0, 2024-04-26T18:13:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_enqueued": 0, 2024-04-26T18:13:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_prioritized": 0, 2024-04-26T18:13:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_started": 0, 2024-04-26T18:13:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_completed": 0, 2024-04-26T18:13:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_path": 0, 2024-04-26T18:13:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_inode": 0, 2024-04-26T18:13:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T18:13:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_exportdir": 0, 2024-04-26T18:13:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_flush": 0, 2024-04-26T18:13:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragmentdir": 0, 2024-04-26T18:13:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragstats": 0, 2024-04-26T18:13:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_inodestats": 0 2024-04-26T18:13:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_log": { 2024-04-26T18:13:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "evlrg": 0, 2024-04-26T18:13:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "evadd": 0, 2024-04-26T18:13:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "evex": 0, 2024-04-26T18:13:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "evtrm": 0, 2024-04-26T18:13:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "ev": 2067, 2024-04-26T18:13:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "evexg": 0, 2024-04-26T18:13:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "evexd": 0, 2024-04-26T18:13:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "segadd": 0, 2024-04-26T18:13:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "segex": 0, 2024-04-26T18:13:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "segtrm": 0, 2024-04-26T18:13:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "seg": 4, 2024-04-26T18:13:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "segmjr": 2, 2024-04-26T18:13:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "segexg": 0, 2024-04-26T18:13:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "segexd": 0, 2024-04-26T18:13:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "expos": 4194304, 2024-04-26T18:13:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "wrpos": 11610432, 2024-04-26T18:13:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "rdpos": 11592355, 2024-04-26T18:13:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "jlat": { 2024-04-26T18:13:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "replayed": 2067 2024-04-26T18:13:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_mem": { 2024-04-26T18:13:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "ino": 967, 2024-04-26T18:13:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "ino+": 989, 2024-04-26T18:13:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "ino-": 22, 2024-04-26T18:13:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "dir": 56, 2024-04-26T18:13:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "dir+": 58, 2024-04-26T18:13:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "dir-": 2, 2024-04-26T18:13:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "dn": 1142, 2024-04-26T18:13:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "dn+": 1142, 2024-04-26T18:13:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "dn-": 0, 2024-04-26T18:13:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "cap": 0, 2024-04-26T18:13:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "cap+": 0, 2024-04-26T18:13:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "cap-": 0, 2024-04-26T18:13:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "rss": 56596, 2024-04-26T18:13:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "heap": 215416 2024-04-26T18:13:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_server": { 2024-04-26T18:13:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_client_request": 0, 2024-04-26T18:13:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_server_request": 0, 2024-04-26T18:13:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_request": 0, 2024-04-26T18:13:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_session": 0, 2024-04-26T18:13:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_peer_request": 0, 2024-04-26T18:13:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "req_create_latency": { 2024-04-26T18:13:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getattr_latency": { 2024-04-26T18:13:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getfilelock_latency": { 2024-04-26T18:13:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "req_link_latency": { 2024-04-26T18:13:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookup_latency": { 2024-04-26T18:13:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookuphash_latency": { 2024-04-26T18:13:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupino_latency": { 2024-04-26T18:13:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupname_latency": { 2024-04-26T18:13:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupparent_latency": { 2024-04-26T18:13:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupsnap_latency": { 2024-04-26T18:13:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lssnap_latency": { 2024-04-26T18:13:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mkdir_latency": { 2024-04-26T18:13:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mknod_latency": { 2024-04-26T18:13:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mksnap_latency": { 2024-04-26T18:13:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "req_open_latency": { 2024-04-26T18:13:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "req_readdir_latency": { 2024-04-26T18:13:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rename_latency": { 2024-04-26T18:13:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "req_renamesnap_latency": { 2024-04-26T18:13:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "req_snapdiff_latency": { 2024-04-26T18:13:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmdir_latency": { 2024-04-26T18:13:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmsnap_latency": { 2024-04-26T18:13:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmxattr_latency": { 2024-04-26T18:13:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setattr_latency": { 2024-04-26T18:13:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setdirlayout_latency": { 2024-04-26T18:13:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setfilelock_latency": { 2024-04-26T18:13:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setlayout_latency": { 2024-04-26T18:13:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setxattr_latency": { 2024-04-26T18:13:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "req_symlink_latency": { 2024-04-26T18:13:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "req_unlink_latency": { 2024-04-26T18:13:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_revoke_eviction": 0, 2024-04-26T18:13:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_acquisition_throttle": 0, 2024-04-26T18:13:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getvxattr_latency": { 2024-04-26T18:13:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_sessions": { 2024-04-26T18:13:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "session_count": 2, 2024-04-26T18:13:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "session_add": 4, 2024-04-26T18:13:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "session_remove": 2, 2024-04-26T18:13:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_open": 3, 2024-04-26T18:13:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_stale": 0, 2024-04-26T18:13:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "total_load": 0, 2024-04-26T18:13:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "average_load": 0, 2024-04-26T18:13:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "avg_session_uptime": 2802, 2024-04-26T18:13:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "mdthresh_evicted": 0 2024-04-26T18:13:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-04-26T18:13:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-04-26T18:13:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-04-26T18:13:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T18:13:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-04-26T18:13:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T18:13:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-04-26T18:13:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T18:13:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T18:13:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T18:13:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T18:13:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T18:13:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-04-26T18:13:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T18:13:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T18:13:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-04-26T18:13:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-04-26T18:13:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-04-26T18:13:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-04-26T18:13:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T18:13:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T18:13:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T18:13:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T18:13:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T18:13:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-04-26T18:13:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-04-26T18:13:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-04-26T18:13:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T18:13:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T18:13:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-04-26T18:13:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-04-26T18:13:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-04-26T18:13:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-04-26T18:13:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T18:13:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-04-26T18:13:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T18:13:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-04-26T18:13:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 221338, 2024-04-26T18:13:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 1109, 2024-04-26T18:13:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-04-26T18:13:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-04-26T18:13:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-04-26T18:13:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-04-26T18:13:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-04-26T18:13:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-04-26T18:13:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-04-26T18:13:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-04-26T18:13:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 8472, 2024-04-26T18:13:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 192, 2024-04-26T18:13:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T18:13:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-04-26T18:13:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-04-26T18:13:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-04-26T18:13:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 3242370, 2024-04-26T18:13:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 45665, 2024-04-26T18:13:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-04-26T18:13:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-04-26T18:13:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-04-26T18:13:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-04-26T18:13:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "objecter": { 2024-04-26T18:13:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "op_active": 0, 2024-04-26T18:13:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "op_laggy": 0, 2024-04-26T18:13:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send": 3767, 2024-04-26T18:13:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send_bytes": 21, 2024-04-26T18:13:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "op_resend": 3, 2024-04-26T18:13:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "op_reply": 3767, 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "op_latency": { 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 3767, 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 6.299888008, 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.001672388 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "op_inflight": 0, 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "oplen_avg": { 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 3767, 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 3768 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "op": 3767, 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "op_r": 3767, 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "op_w": 0, 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "op_rmw": 0, 2024-04-26T18:13:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "op_pg": 0, 2024-04-26T18:13:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_stat": 2280, 2024-04-26T18:13:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_create": 0, 2024-04-26T18:13:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_read": 1485, 2024-04-26T18:13:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_write": 0, 2024-04-26T18:13:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writefull": 0, 2024-04-26T18:13:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writesame": 0, 2024-04-26T18:13:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_append": 0, 2024-04-26T18:13:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_zero": 0, 2024-04-26T18:13:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_truncate": 0, 2024-04-26T18:13:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_delete": 0, 2024-04-26T18:13:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_mapext": 0, 2024-04-26T18:13:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_sparse_read": 0, 2024-04-26T18:13:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_clonerange": 0, 2024-04-26T18:13:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_getxattr": 1, 2024-04-26T18:13:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_setxattr": 0, 2024-04-26T18:13:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_cmpxattr": 0, 2024-04-26T18:13:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_rmxattr": 0, 2024-04-26T18:13:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_resetxattrs": 0, 2024-04-26T18:13:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_call": 0, 2024-04-26T18:13:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_watch": 0, 2024-04-26T18:13:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_notify": 0, 2024-04-26T18:13:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T18:13:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls": 0, 2024-04-26T18:13:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls_filter": 0, 2024-04-26T18:13:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_other": 0, 2024-04-26T18:13:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_active": 0, 2024-04-26T18:13:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_send": 0, 2024-04-26T18:13:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_resend": 0, 2024-04-26T18:13:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_ping": 0, 2024-04-26T18:13:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_active": 0, 2024-04-26T18:13:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_send": 0, 2024-04-26T18:13:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_resend": 0, 2024-04-26T18:13:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_active": 0, 2024-04-26T18:13:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_send": 0, 2024-04-26T18:13:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_resend": 0, 2024-04-26T18:13:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_active": 0, 2024-04-26T18:13:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_send": 0, 2024-04-26T18:13:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_resend": 0, 2024-04-26T18:13:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "command_active": 0, 2024-04-26T18:13:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "command_send": 0, 2024-04-26T18:13:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "command_resend": 0, 2024-04-26T18:13:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "map_epoch": 112, 2024-04-26T18:13:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "map_full": 0, 2024-04-26T18:13:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "map_inc": 24, 2024-04-26T18:13:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_sessions": 4, 2024-04-26T18:13:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_open": 5, 2024-04-26T18:13:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_close": 1, 2024-04-26T18:13:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_laggy": 0, 2024-04-26T18:13:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_wr": 0, 2024-04-26T18:13:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_rd": 2, 2024-04-26T18:13:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_del": 0 2024-04-26T18:13:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "oft": { 2024-04-26T18:13:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_objs": 0, 2024-04-26T18:13:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_kv_pairs": 0, 2024-04-26T18:13:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_updates": 0, 2024-04-26T18:13:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_removes": 0 2024-04-26T18:13:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "osd-slow-ops": { 2024-04-26T18:13:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_ops_count": 0 2024-04-26T18:13:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "purge_queue": { 2024-04-26T18:13:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops": 0, 2024-04-26T18:13:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T18:13:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing": 0, 2024-04-26T18:13:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_high_water": 0, 2024-04-26T18:13:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed_ops": 0, 2024-04-26T18:13:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed": 0, 2024-04-26T18:13:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_item_in_journal": 0 2024-04-26T18:13:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T18:13:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-26T18:13:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 4172, 2024-04-26T18:13:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 9857916, 2024-04-26T18:13:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 4172, 2024-04-26T18:13:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T18:13:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T18:13:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 4172, 2024-04-26T18:13:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 9857916, 2024-04-26T18:13:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_bytes": { 2024-04-26T18:13:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-26T18:13:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T18:13:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T18:13:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 3767, 2024-04-26T18:13:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 7416133, 2024-04-26T18:13:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 3767, 2024-04-26T18:13:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 7416133, 2024-04-26T18:13:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_ops": { 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 1024, 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 3767, 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 3767, 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 3767, 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 3767, 2024-04-26T18:13:11.649 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:11.650 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.650 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.650 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.650 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:11.650 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.650 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle": { 2024-04-26T18:13:11.650 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:11.650 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-26T18:13:11.650 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:11.650 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T18:13:11.650 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T18:13:11.650 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:11.650 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:11.650 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T18:13:11.651 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T18:13:11.651 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-04-26T18:13:11.651 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-04-26T18:13:11.651 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:11.651 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.651 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.651 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.651 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:11.651 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:11.651 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle-0x559acb68a6e0": { 2024-04-26T18:13:11.651 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:11.651 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-26T18:13:11.652 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:11.652 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T18:13:11.652 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T18:13:11.652 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:11.652 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:11.652 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T18:13:11.652 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T18:13:11.652 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-04-26T18:13:11.652 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-04-26T18:13:11.652 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:11.652 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:11.652 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:11.652 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:11.653 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:11.653 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:11.653 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-26T18:13:11.653 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:11 smithi097 ceph-mon[32419]: pgmap v524: 97 pgs: 97 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 29 op/s 2024-04-26T18:13:11.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:11 smithi031 ceph-mon[29164]: pgmap v524: 97 pgs: 97 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 29 op/s 2024-04-26T18:13:12.495 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-26T18:13:12.496 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-26T18:13:12.496 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-04-26T18:13:12.496 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-26T18:13:12.496 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T18:13:12.496 DEBUG:tasks.check_counter:Getting stats from i 2024-04-26T18:13:12.496 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-26T18:13:12.737 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T18:13:13.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:13 smithi186 ceph-mon[32797]: pgmap v525: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-04-26T18:13:13.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:13 smithi097 ceph-mon[32419]: pgmap v525: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-04-26T18:13:13.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:13 smithi031 ceph-mon[29164]: pgmap v525: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-04-26T18:13:14.372 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-26T18:13:14.372 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T18:13:14.373 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 4292, 2024-04-26T18:13:14.373 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 5076, 2024-04-26T18:13:14.373 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 1397941, 2024-04-26T18:13:14.373 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 10489017, 2024-04-26T18:13:14.373 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 17, 2024-04-26T18:13:14.373 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 3, 2024-04-26T18:13:14.373 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.358828582, 2024-04-26T18:13:14.373 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.139028340, 2024-04-26T18:13:14.373 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.164045263, 2024-04-26T18:13:14.373 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.070292179, 2024-04-26T18:13:14.373 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:14.373 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 5074, 2024-04-26T18:13:14.373 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.175808007, 2024-04-26T18:13:14.373 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000034648 2024-04-26T18:13:14.374 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.374 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:14.374 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 5101, 2024-04-26T18:13:14.374 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000820304, 2024-04-26T18:13:14.374 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000160 2024-04-26T18:13:14.374 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.374 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 1047440, 2024-04-26T18:13:14.374 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 2529440 2024-04-26T18:13:14.374 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.374 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T18:13:14.374 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 4824, 2024-04-26T18:13:14.374 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 3857, 2024-04-26T18:13:14.375 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 3072224, 2024-04-26T18:13:14.375 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 1645594, 2024-04-26T18:13:14.375 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 10, 2024-04-26T18:13:14.375 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 6, 2024-04-26T18:13:14.375 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.290467284, 2024-04-26T18:13:14.375 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.105338051, 2024-04-26T18:13:14.375 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.227483112, 2024-04-26T18:13:14.375 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.014937919, 2024-04-26T18:13:14.375 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:14.375 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 3834, 2024-04-26T18:13:14.375 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.146255781, 2024-04-26T18:13:14.375 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000038147 2024-04-26T18:13:14.375 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.376 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:14.376 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 7385, 2024-04-26T18:13:14.376 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.001473340, 2024-04-26T18:13:14.376 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000199 2024-04-26T18:13:14.376 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.376 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 1804000, 2024-04-26T18:13:14.376 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 231680 2024-04-26T18:13:14.376 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.376 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T18:13:14.376 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 5844, 2024-04-26T18:13:14.376 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 4880, 2024-04-26T18:13:14.376 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 2755364, 2024-04-26T18:13:14.376 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 1870039, 2024-04-26T18:13:14.376 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 13, 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 9, 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.291925781, 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.109513251, 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.232707396, 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.004617501, 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 4878, 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.123331982, 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000025283 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 10034, 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.001421624, 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000141 2024-04-26T18:13:14.377 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.378 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 4032, 2024-04-26T18:13:14.378 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 157344 2024-04-26T18:13:14.378 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.378 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-04-26T18:13:14.378 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 1, 2024-04-26T18:13:14.378 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-04-26T18:13:14.378 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.378 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-MDSRank": { 2024-04-26T18:13:14.378 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-26T18:13:14.378 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-26T18:13:14.378 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 2383, 2024-04-26T18:13:14.378 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 1.139318688, 2024-04-26T18:13:14.379 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000478102 2024-04-26T18:13:14.379 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:14.379 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.379 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-PurgeQueue": { 2024-04-26T18:13:14.379 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-26T18:13:14.379 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-26T18:13:14.379 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 17, 2024-04-26T18:13:14.379 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000470265, 2024-04-26T18:13:14.379 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000027662 2024-04-26T18:13:14.379 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:14.379 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.379 INFO:teuthology.orchestra.run.smithi186.stdout: "mds": { 2024-04-26T18:13:14.379 INFO:teuthology.orchestra.run.smithi186.stdout: "request": 769, 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: "reply": 678, 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: "reply_latency": { 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 678, 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 8.459332181, 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.012476891 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_reply": 0, 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: "forward": 91, 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_complete": 0, 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_keys": 0, 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_commit": 27, 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_split": 2, 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_merge": 0, 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes": 1162, 2024-04-26T18:13:14.380 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_top": 121, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_bottom": 1041, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pin_tail": 0, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pinned": 988, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_expired": 59, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_with_caps": 672, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "caps": 672, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "subtrees": 18, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse": 2212, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_hit": 1778, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_forward": 78, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_discover": 57, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_dir_fetch": 0, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_remote_ino": 3, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_lock": 67, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "load_cent": 1872, 2024-04-26T18:13:14.381 INFO:teuthology.orchestra.run.smithi186.stdout: "q": 1, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "exported": 0, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "exported_inodes": 0, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "imported": 17, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "imported_inodes": 768, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_dir_fetch": 0, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_backtrace_fetch": 0, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_peer_discover": 7, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rfiles": 1025, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rbytes": 1599855916, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rsnaps": 6, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_set_tag": 0, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_inotable_repaired": 0, 2024-04-26T18:13:14.382 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_inodes": 0, 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_file_inodes": 0, 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_inode_file_caps": 14, 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_revoke": 874, 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_grant": 772, 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flush_ack": 1, 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps": 1353, 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps_dirty": 290, 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_cap_release": 65, 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: "process_request_cap_release": 187 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.383 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_cache": { 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update": 0, 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update_receipt": 0, 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_try_discover": 0, 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_send_discover": 74, 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_handle_discover": 53, 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays": 6, 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_delayed": 0, 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_enqueuing": 0, 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_created": 47, 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_enqueued": 2, 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_reintegrated": 38, 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_migrated": 20, 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_processing": 0, 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_enqueued": 0, 2024-04-26T18:13:14.384 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_prioritized": 0, 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_started": 0, 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_completed": 0, 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_path": 5, 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_inode": 2550, 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_exportdir": 0, 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_flush": 0, 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragmentdir": 2, 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragstats": 0, 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_inodestats": 0 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_log": { 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: "evlrg": 0, 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: "evadd": 2067, 2024-04-26T18:13:14.385 INFO:teuthology.orchestra.run.smithi186.stdout: "evex": 0, 2024-04-26T18:13:14.386 INFO:teuthology.orchestra.run.smithi186.stdout: "evtrm": 0, 2024-04-26T18:13:14.386 INFO:teuthology.orchestra.run.smithi186.stdout: "ev": 2067, 2024-04-26T18:13:14.386 INFO:teuthology.orchestra.run.smithi186.stdout: "evexg": 0, 2024-04-26T18:13:14.386 INFO:teuthology.orchestra.run.smithi186.stdout: "evexd": 0, 2024-04-26T18:13:14.386 INFO:teuthology.orchestra.run.smithi186.stdout: "segadd": 4, 2024-04-26T18:13:14.386 INFO:teuthology.orchestra.run.smithi186.stdout: "segex": 0, 2024-04-26T18:13:14.386 INFO:teuthology.orchestra.run.smithi186.stdout: "segtrm": 0, 2024-04-26T18:13:14.386 INFO:teuthology.orchestra.run.smithi186.stdout: "seg": 4, 2024-04-26T18:13:14.386 INFO:teuthology.orchestra.run.smithi186.stdout: "segmjr": 2, 2024-04-26T18:13:14.386 INFO:teuthology.orchestra.run.smithi186.stdout: "segexg": 0, 2024-04-26T18:13:14.386 INFO:teuthology.orchestra.run.smithi186.stdout: "segexd": 0, 2024-04-26T18:13:14.386 INFO:teuthology.orchestra.run.smithi186.stdout: "expos": 4194304, 2024-04-26T18:13:14.386 INFO:teuthology.orchestra.run.smithi186.stdout: "wrpos": 11610432, 2024-04-26T18:13:14.386 INFO:teuthology.orchestra.run.smithi186.stdout: "rdpos": 0, 2024-04-26T18:13:14.387 INFO:teuthology.orchestra.run.smithi186.stdout: "jlat": { 2024-04-26T18:13:14.387 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1335, 2024-04-26T18:13:14.387 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 13.469533440, 2024-04-26T18:13:14.387 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.010089538 2024-04-26T18:13:14.387 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.387 INFO:teuthology.orchestra.run.smithi186.stdout: "replayed": 0 2024-04-26T18:13:14.387 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.387 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_mem": { 2024-04-26T18:13:14.387 INFO:teuthology.orchestra.run.smithi186.stdout: "ino": 1003, 2024-04-26T18:13:14.387 INFO:teuthology.orchestra.run.smithi186.stdout: "ino+": 1007, 2024-04-26T18:13:14.387 INFO:teuthology.orchestra.run.smithi186.stdout: "ino-": 4, 2024-04-26T18:13:14.387 INFO:teuthology.orchestra.run.smithi186.stdout: "dir": 75, 2024-04-26T18:13:14.387 INFO:teuthology.orchestra.run.smithi186.stdout: "dir+": 78, 2024-04-26T18:13:14.387 INFO:teuthology.orchestra.run.smithi186.stdout: "dir-": 3, 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: "dn": 1172, 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: "dn+": 1262, 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: "dn-": 90, 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: "cap": 672, 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: "cap+": 1274, 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: "cap-": 602, 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: "rss": 68536, 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: "heap": 223608 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_server": { 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_client_request": 1675, 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_server_request": 2656, 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_request": 769, 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_session": 104, 2024-04-26T18:13:14.388 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_peer_request": 5205, 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: "req_create_latency": { 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 4, 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.008855256, 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.002213814 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getattr_latency": { 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 50, 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 6.368854033, 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.127377080 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getfilelock_latency": { 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.389 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.390 INFO:teuthology.orchestra.run.smithi186.stdout: "req_link_latency": { 2024-04-26T18:13:14.390 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 47, 2024-04-26T18:13:14.390 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.294347348, 2024-04-26T18:13:14.390 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.006262709 2024-04-26T18:13:14.390 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.390 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookup_latency": { 2024-04-26T18:13:14.390 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 329, 2024-04-26T18:13:14.390 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.406978213, 2024-04-26T18:13:14.390 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.001237015 2024-04-26T18:13:14.390 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.390 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookuphash_latency": { 2024-04-26T18:13:14.390 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.390 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.390 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupino_latency": { 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupname_latency": { 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupparent_latency": { 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.391 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.392 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.392 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupsnap_latency": { 2024-04-26T18:13:14.392 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.392 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.392 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.392 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.392 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lssnap_latency": { 2024-04-26T18:13:14.392 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.392 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.393 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.393 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.393 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mkdir_latency": { 2024-04-26T18:13:14.393 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.393 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.393 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.393 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.393 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mknod_latency": { 2024-04-26T18:13:14.393 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.393 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.393 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.393 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.393 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mksnap_latency": { 2024-04-26T18:13:14.394 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.394 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.394 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.394 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.394 INFO:teuthology.orchestra.run.smithi186.stdout: "req_open_latency": { 2024-04-26T18:13:14.394 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 100, 2024-04-26T18:13:14.394 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.123442663, 2024-04-26T18:13:14.394 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.001234426 2024-04-26T18:13:14.394 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.394 INFO:teuthology.orchestra.run.smithi186.stdout: "req_readdir_latency": { 2024-04-26T18:13:14.394 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 44, 2024-04-26T18:13:14.394 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.038654867, 2024-04-26T18:13:14.394 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000878519 2024-04-26T18:13:14.394 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.395 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rename_latency": { 2024-04-26T18:13:14.395 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 57, 2024-04-26T18:13:14.395 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 1.041680187, 2024-04-26T18:13:14.395 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.018275091 2024-04-26T18:13:14.395 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.395 INFO:teuthology.orchestra.run.smithi186.stdout: "req_renamesnap_latency": { 2024-04-26T18:13:14.395 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.395 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.395 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.395 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.395 INFO:teuthology.orchestra.run.smithi186.stdout: "req_snapdiff_latency": { 2024-04-26T18:13:14.395 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.395 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.395 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.396 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.396 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmdir_latency": { 2024-04-26T18:13:14.396 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.396 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.396 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.396 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.396 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmsnap_latency": { 2024-04-26T18:13:14.396 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.396 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.396 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.396 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.396 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmxattr_latency": { 2024-04-26T18:13:14.396 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.397 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.397 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.397 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.397 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setattr_latency": { 2024-04-26T18:13:14.397 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.397 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.397 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.397 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.397 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setdirlayout_latency": { 2024-04-26T18:13:14.397 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.397 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.397 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.397 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.397 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setfilelock_latency": { 2024-04-26T18:13:14.398 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.398 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.398 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.398 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.398 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setlayout_latency": { 2024-04-26T18:13:14.398 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.398 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.398 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.398 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.398 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setxattr_latency": { 2024-04-26T18:13:14.398 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.398 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.398 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.399 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.399 INFO:teuthology.orchestra.run.smithi186.stdout: "req_symlink_latency": { 2024-04-26T18:13:14.399 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.399 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.399 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.399 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.399 INFO:teuthology.orchestra.run.smithi186.stdout: "req_unlink_latency": { 2024-04-26T18:13:14.399 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 47, 2024-04-26T18:13:14.399 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.176519614, 2024-04-26T18:13:14.399 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.003755736 2024-04-26T18:13:14.399 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.399 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_revoke_eviction": 0, 2024-04-26T18:13:14.399 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_acquisition_throttle": 0, 2024-04-26T18:13:14.400 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getvxattr_latency": { 2024-04-26T18:13:14.400 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.400 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.400 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.400 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:14.400 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.400 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_sessions": { 2024-04-26T18:13:14.400 INFO:teuthology.orchestra.run.smithi186.stdout: "session_count": 2, 2024-04-26T18:13:14.400 INFO:teuthology.orchestra.run.smithi186.stdout: "session_add": 4, 2024-04-26T18:13:14.400 INFO:teuthology.orchestra.run.smithi186.stdout: "session_remove": 2, 2024-04-26T18:13:14.400 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_open": 2, 2024-04-26T18:13:14.400 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_stale": 0, 2024-04-26T18:13:14.400 INFO:teuthology.orchestra.run.smithi186.stdout: "total_load": 200, 2024-04-26T18:13:14.400 INFO:teuthology.orchestra.run.smithi186.stdout: "average_load": 100, 2024-04-26T18:13:14.401 INFO:teuthology.orchestra.run.smithi186.stdout: "avg_session_uptime": 973, 2024-04-26T18:13:14.401 INFO:teuthology.orchestra.run.smithi186.stdout: "mdthresh_evicted": 0 2024-04-26T18:13:14.401 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.401 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-04-26T18:13:14.401 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-04-26T18:13:14.401 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-04-26T18:13:14.401 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T18:13:14.401 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-04-26T18:13:14.401 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T18:13:14.401 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-04-26T18:13:14.401 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T18:13:14.401 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T18:13:14.401 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T18:13:14.401 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T18:13:14.402 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T18:13:14.402 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-04-26T18:13:14.402 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T18:13:14.402 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T18:13:14.402 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-04-26T18:13:14.402 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-04-26T18:13:14.402 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-04-26T18:13:14.402 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-04-26T18:13:14.402 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T18:13:14.402 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T18:13:14.402 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T18:13:14.402 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T18:13:14.402 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T18:13:14.402 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-04-26T18:13:14.403 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-04-26T18:13:14.403 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-04-26T18:13:14.403 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T18:13:14.403 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T18:13:14.403 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-04-26T18:13:14.403 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-04-26T18:13:14.403 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-04-26T18:13:14.403 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-04-26T18:13:14.403 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T18:13:14.403 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-04-26T18:13:14.403 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T18:13:14.403 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-04-26T18:13:14.403 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 738608, 2024-04-26T18:13:14.403 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 1084, 2024-04-26T18:13:14.404 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-04-26T18:13:14.404 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-04-26T18:13:14.404 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-04-26T18:13:14.404 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-04-26T18:13:14.404 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-04-26T18:13:14.404 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-04-26T18:13:14.404 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-04-26T18:13:14.404 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-04-26T18:13:14.404 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 8472, 2024-04-26T18:13:14.404 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 192, 2024-04-26T18:13:14.404 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T18:13:14.404 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-04-26T18:13:14.404 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-04-26T18:13:14.405 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-04-26T18:13:14.405 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 3764466, 2024-04-26T18:13:14.405 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 64559, 2024-04-26T18:13:14.405 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-04-26T18:13:14.405 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-04-26T18:13:14.405 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-04-26T18:13:14.405 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-04-26T18:13:14.405 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.405 INFO:teuthology.orchestra.run.smithi186.stdout: "objecter": { 2024-04-26T18:13:14.405 INFO:teuthology.orchestra.run.smithi186.stdout: "op_active": 0, 2024-04-26T18:13:14.405 INFO:teuthology.orchestra.run.smithi186.stdout: "op_laggy": 0, 2024-04-26T18:13:14.405 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send": 1588, 2024-04-26T18:13:14.405 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send_bytes": 7613239, 2024-04-26T18:13:14.405 INFO:teuthology.orchestra.run.smithi186.stdout: "op_resend": 28, 2024-04-26T18:13:14.406 INFO:teuthology.orchestra.run.smithi186.stdout: "op_reply": 1588, 2024-04-26T18:13:14.406 INFO:teuthology.orchestra.run.smithi186.stdout: "op_latency": { 2024-04-26T18:13:14.406 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1588, 2024-04-26T18:13:14.406 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 16.123714629, 2024-04-26T18:13:14.406 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.010153472 2024-04-26T18:13:14.406 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.406 INFO:teuthology.orchestra.run.smithi186.stdout: "op_inflight": 0, 2024-04-26T18:13:14.406 INFO:teuthology.orchestra.run.smithi186.stdout: "oplen_avg": { 2024-04-26T18:13:14.406 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1588, 2024-04-26T18:13:14.406 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 1765 2024-04-26T18:13:14.406 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.406 INFO:teuthology.orchestra.run.smithi186.stdout: "op": 1588, 2024-04-26T18:13:14.406 INFO:teuthology.orchestra.run.smithi186.stdout: "op_r": 1, 2024-04-26T18:13:14.406 INFO:teuthology.orchestra.run.smithi186.stdout: "op_w": 1587, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "op_rmw": 0, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "op_pg": 0, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_stat": 0, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_create": 10, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_read": 1, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_write": 1336, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writefull": 66, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writesame": 0, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_append": 0, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_zero": 0, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_truncate": 0, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_delete": 17, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_mapext": 0, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_sparse_read": 0, 2024-04-26T18:13:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_clonerange": 0, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_getxattr": 0, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_setxattr": 20, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_cmpxattr": 0, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_rmxattr": 0, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_resetxattrs": 0, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_call": 0, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_watch": 0, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_notify": 0, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls": 0, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls_filter": 0, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_other": 0, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_active": 0, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_send": 0, 2024-04-26T18:13:14.408 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_resend": 0, 2024-04-26T18:13:14.409 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_ping": 0, 2024-04-26T18:13:14.409 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_active": 0, 2024-04-26T18:13:14.409 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_send": 0, 2024-04-26T18:13:14.409 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_resend": 0, 2024-04-26T18:13:14.409 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_active": 0, 2024-04-26T18:13:14.409 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_send": 0, 2024-04-26T18:13:14.409 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_resend": 0, 2024-04-26T18:13:14.409 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_active": 0, 2024-04-26T18:13:14.409 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_send": 0, 2024-04-26T18:13:14.409 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_resend": 0, 2024-04-26T18:13:14.409 INFO:teuthology.orchestra.run.smithi186.stdout: "command_active": 0, 2024-04-26T18:13:14.409 INFO:teuthology.orchestra.run.smithi186.stdout: "command_send": 0, 2024-04-26T18:13:14.409 INFO:teuthology.orchestra.run.smithi186.stdout: "command_resend": 0, 2024-04-26T18:13:14.410 INFO:teuthology.orchestra.run.smithi186.stdout: "map_epoch": 112, 2024-04-26T18:13:14.410 INFO:teuthology.orchestra.run.smithi186.stdout: "map_full": 0, 2024-04-26T18:13:14.410 INFO:teuthology.orchestra.run.smithi186.stdout: "map_inc": 24, 2024-04-26T18:13:14.410 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_sessions": 12, 2024-04-26T18:13:14.410 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_open": 13, 2024-04-26T18:13:14.410 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_close": 1, 2024-04-26T18:13:14.410 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_laggy": 0, 2024-04-26T18:13:14.410 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_wr": 259, 2024-04-26T18:13:14.410 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_rd": 0, 2024-04-26T18:13:14.410 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_del": 56 2024-04-26T18:13:14.410 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.410 INFO:teuthology.orchestra.run.smithi186.stdout: "oft": { 2024-04-26T18:13:14.410 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_objs": 1, 2024-04-26T18:13:14.410 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_kv_pairs": 0, 2024-04-26T18:13:14.411 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_updates": 0, 2024-04-26T18:13:14.411 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_removes": 0 2024-04-26T18:13:14.411 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.411 INFO:teuthology.orchestra.run.smithi186.stdout: "osd-slow-ops": { 2024-04-26T18:13:14.411 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_ops_count": 2602 2024-04-26T18:13:14.411 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.411 INFO:teuthology.orchestra.run.smithi186.stdout: "purge_queue": { 2024-04-26T18:13:14.411 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops": 0, 2024-04-26T18:13:14.411 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops_high_water": 2, 2024-04-26T18:13:14.411 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing": 0, 2024-04-26T18:13:14.411 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_high_water": 2, 2024-04-26T18:13:14.411 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed_ops": 2, 2024-04-26T18:13:14.411 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed": 2, 2024-04-26T18:13:14.411 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_item_in_journal": 0 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 14956, 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 5728339, 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 14956, 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 14956, 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 5728339, 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:14.412 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.413 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.413 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.413 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:14.413 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.413 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_bytes": { 2024-04-26T18:13:14.413 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:14.413 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-26T18:13:14.413 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:14.413 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T18:13:14.413 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T18:13:14.413 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:14.413 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:14.413 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 1588, 2024-04-26T18:13:14.413 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 7613537, 2024-04-26T18:13:14.414 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 1588, 2024-04-26T18:13:14.414 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 7613537, 2024-04-26T18:13:14.414 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:14.414 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.414 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.414 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.414 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:14.414 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.414 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_ops": { 2024-04-26T18:13:14.414 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:14.414 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 1024, 2024-04-26T18:13:14.414 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:14.414 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T18:13:14.414 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T18:13:14.415 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:14.415 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T18:13:14.415 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 1588, 2024-04-26T18:13:14.415 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 1588, 2024-04-26T18:13:14.415 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 1588, 2024-04-26T18:13:14.415 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 1588, 2024-04-26T18:13:14.415 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:14.415 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.415 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.415 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.415 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:14.415 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.415 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle": { 2024-04-26T18:13:14.415 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 2, 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 298, 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 2, 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 1, 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 298, 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.416 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:14.417 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:14.417 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle-0x5558ff318500": { 2024-04-26T18:13:14.417 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:14.417 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-26T18:13:14.417 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:14.417 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 2067, 2024-04-26T18:13:14.417 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 7416128, 2024-04-26T18:13:14.417 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:14.417 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 2067, 2024-04-26T18:13:14.417 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T18:13:14.417 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T18:13:14.417 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 1335, 2024-04-26T18:13:14.417 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 7416128, 2024-04-26T18:13:14.417 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:14.418 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:14.418 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:14.418 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:14.418 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:14.418 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:14.418 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-26T18:13:15.113 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-26T18:13:15.113 INFO:tasks.check_counter:Daemon mds.i mds.imported=17 2024-04-26T18:13:15.113 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-26T18:13:15.113 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-26T18:13:15.113 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T18:13:15.113 DEBUG:tasks.check_counter:Getting stats from l 2024-04-26T18:13:15.114 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-26T18:13:15.354 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T18:13:15.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:15 smithi097 ceph-mon[32419]: pgmap v526: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-04-26T18:13:15.736 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:15 smithi186 ceph-mon[32797]: pgmap v526: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-04-26T18:13:15.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:15 smithi031 ceph-mon[29164]: pgmap v526: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-04-26T18:13:17.147 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-26T18:13:17.148 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T18:13:17.148 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 0, 2024-04-26T18:13:17.148 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 2, 2024-04-26T18:13:17.148 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 0, 2024-04-26T18:13:17.148 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 0, 2024-04-26T18:13:17.148 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 4, 2024-04-26T18:13:17.148 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 0, 2024-04-26T18:13:17.149 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.001121094, 2024-04-26T18:13:17.149 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T18:13:17.149 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.000272883, 2024-04-26T18:13:17.149 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T18:13:17.149 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:17.149 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:17.149 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:17.150 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:17.150 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:17.150 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:17.150 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:17.150 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:17.151 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:17.151 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:17.151 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T18:13:17.151 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T18:13:17.151 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:17.151 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T18:13:17.151 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 351, 2024-04-26T18:13:17.151 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 338, 2024-04-26T18:13:17.151 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 1628688, 2024-04-26T18:13:17.152 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 208064, 2024-04-26T18:13:17.152 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 2, 2024-04-26T18:13:17.152 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 1, 2024-04-26T18:13:17.152 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.054965992, 2024-04-26T18:13:17.152 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.017795713, 2024-04-26T18:13:17.152 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.037867023, 2024-04-26T18:13:17.152 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.007439995, 2024-04-26T18:13:17.152 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:17.152 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 338, 2024-04-26T18:13:17.152 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.024004997, 2024-04-26T18:13:17.153 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000071020 2024-04-26T18:13:17.153 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:17.153 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:17.153 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:17.153 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:17.153 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:17.153 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:17.153 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 1628688, 2024-04-26T18:13:17.153 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 208064 2024-04-26T18:13:17.154 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:17.154 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T18:13:17.154 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 3, 2024-04-26T18:13:17.154 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 258, 2024-04-26T18:13:17.154 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 480, 2024-04-26T18:13:17.155 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 281136, 2024-04-26T18:13:17.155 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 6, 2024-04-26T18:13:17.155 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 3, 2024-04-26T18:13:17.155 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.014885973, 2024-04-26T18:13:17.155 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.010339717, 2024-04-26T18:13:17.155 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.001928028, 2024-04-26T18:13:17.156 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T18:13:17.156 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T18:13:17.156 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 255, 2024-04-26T18:13:17.156 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.032836205, 2024-04-26T18:13:17.156 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000128769 2024-04-26T18:13:17.156 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:17.156 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T18:13:17.156 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:17.156 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:17.156 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:17.156 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:17.156 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T18:13:17.156 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 281136 2024-04-26T18:13:17.157 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:17.157 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-04-26T18:13:17.157 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 0, 2024-04-26T18:13:17.157 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-04-26T18:13:17.157 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:17.157 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-04-26T18:13:17.158 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-04-26T18:13:17.158 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-04-26T18:13:17.158 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T18:13:17.158 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-04-26T18:13:17.158 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T18:13:17.159 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-04-26T18:13:17.159 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T18:13:17.159 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T18:13:17.159 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T18:13:17.159 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T18:13:17.160 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T18:13:17.160 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-04-26T18:13:17.160 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T18:13:17.160 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T18:13:17.160 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-04-26T18:13:17.160 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-04-26T18:13:17.160 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-04-26T18:13:17.160 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-04-26T18:13:17.160 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T18:13:17.160 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T18:13:17.161 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T18:13:17.161 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T18:13:17.161 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T18:13:17.161 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-04-26T18:13:17.161 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-04-26T18:13:17.161 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-04-26T18:13:17.161 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T18:13:17.161 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T18:13:17.162 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-04-26T18:13:17.162 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-04-26T18:13:17.162 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-04-26T18:13:17.162 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-04-26T18:13:17.162 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T18:13:17.163 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-04-26T18:13:17.163 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T18:13:17.163 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-04-26T18:13:17.163 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 148010, 2024-04-26T18:13:17.164 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 30, 2024-04-26T18:13:17.164 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-04-26T18:13:17.164 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-04-26T18:13:17.164 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-04-26T18:13:17.164 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-04-26T18:13:17.164 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-04-26T18:13:17.164 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-04-26T18:13:17.164 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-04-26T18:13:17.164 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-04-26T18:13:17.164 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 0, 2024-04-26T18:13:17.165 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 0, 2024-04-26T18:13:17.165 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T18:13:17.165 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-04-26T18:13:17.165 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-04-26T18:13:17.165 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-04-26T18:13:17.165 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 928, 2024-04-26T18:13:17.165 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 2, 2024-04-26T18:13:17.165 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-04-26T18:13:17.166 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-04-26T18:13:17.166 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-04-26T18:13:17.166 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-04-26T18:13:17.166 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T18:13:17.166 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T18:13:17.166 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T18:13:17.166 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-26T18:13:17.167 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T18:13:17.167 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 350, 2024-04-26T18:13:17.167 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 1576779, 2024-04-26T18:13:17.167 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T18:13:17.167 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 350, 2024-04-26T18:13:17.168 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T18:13:17.168 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T18:13:17.168 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 350, 2024-04-26T18:13:17.168 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 1576779, 2024-04-26T18:13:17.168 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T18:13:17.168 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T18:13:17.168 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T18:13:17.169 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T18:13:17.169 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:17.169 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T18:13:17.169 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-26T18:13:17.512 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:17 smithi186 ceph-mon[32797]: pgmap v527: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-04-26T18:13:17.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:17 smithi031 ceph-mon[29164]: pgmap v527: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-04-26T18:13:17.838 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-26T18:13:17.839 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-26T18:13:17.839 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-26T18:13:17.839 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-26T18:13:17.839 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-26T18:13:17.839 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-26T18:13:17.849 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-26T18:13:17.850 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T18:13:17.850 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T18:13:17.871 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T18:13:17.871 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T18:13:17.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:17 smithi097 ceph-mon[32419]: pgmap v527: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-04-26T18:13:17.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd blocklist ls 2024-04-26T18:13:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:13:19.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:19 smithi031 ceph-mon[29164]: pgmap v528: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.8 KiB/s wr, 6 op/s 2024-04-26T18:13:19.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:19 smithi031 ceph-mon[29164]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T18:13:19.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:19 smithi186 ceph-mon[32797]: pgmap v528: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.8 KiB/s wr, 6 op/s 2024-04-26T18:13:19.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:19 smithi186 ceph-mon[32797]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T18:13:19.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:19 smithi097 ceph-mon[32419]: pgmap v528: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.8 KiB/s wr, 6 op/s 2024-04-26T18:13:19.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:19 smithi097 ceph-mon[32419]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T18:13:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.902+0000 7fae8b59e700 1 -- 172.21.15.31:0/836704689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae8c102860 msgr2=0x7fae8c108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.902+0000 7fae8b59e700 1 --2- 172.21.15.31:0/836704689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae8c102860 0x7fae8c108040 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fae80009990 tx=0x7fae80009ca0 comp rx=0 tx=0).stop 2024-04-26T18:13:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.902+0000 7fae8b59e700 1 -- 172.21.15.31:0/836704689 shutdown_connections 2024-04-26T18:13:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.902+0000 7fae8b59e700 1 --2- 172.21.15.31:0/836704689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae8c108720 0x7fae8c1102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.902+0000 7fae8b59e700 1 --2- 172.21.15.31:0/836704689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae8c101eb0 0x7fae8c102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.902+0000 7fae8b59e700 1 --2- 172.21.15.31:0/836704689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae8c102860 0x7fae8c108040 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.902+0000 7fae8b59e700 1 -- 172.21.15.31:0/836704689 >> 172.21.15.31:0/836704689 conn(0x7fae8c0775d0 msgr2=0x7fae8c0779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.902+0000 7fae8b59e700 1 -- 172.21.15.31:0/836704689 shutdown_connections 2024-04-26T18:13:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.902+0000 7fae8b59e700 1 -- 172.21.15.31:0/836704689 wait complete. 2024-04-26T18:13:19.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.903+0000 7fae8b59e700 1 Processor -- start 2024-04-26T18:13:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.903+0000 7fae8b59e700 1 -- start start 2024-04-26T18:13:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.903+0000 7fae8b59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae8c101eb0 0x7fae8c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.903+0000 7fae8b59e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae8c102860 0x7fae8c1a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.903+0000 7fae8b59e700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae8c108720 0x7fae8c10e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.903+0000 7fae8b59e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c113550 con 0x7fae8c102860 2024-04-26T18:13:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae8b59e700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c1133d0 con 0x7fae8c108720 2024-04-26T18:13:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae89d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae8c102860 0x7fae8c1a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae8b59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c1136d0 con 0x7fae8c101eb0 2024-04-26T18:13:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae89d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae8c102860 0x7fae8c1a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59494/0 (socket says 172.21.15.31:59494) 2024-04-26T18:13:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae89d9b700 1 -- 172.21.15.31:0/3297387193 learned_addr learned my addr 172.21.15.31:0/3297387193 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:13:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae8a59c700 1 --2- 172.21.15.31:0/3297387193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae8c101eb0 0x7fae8c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae8ad9d700 1 --2- 172.21.15.31:0/3297387193 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae8c108720 0x7fae8c10e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae89d9b700 1 -- 172.21.15.31:0/3297387193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae8c101eb0 msgr2=0x7fae8c1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae89d9b700 1 --2- 172.21.15.31:0/3297387193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae8c101eb0 0x7fae8c1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae89d9b700 1 -- 172.21.15.31:0/3297387193 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae8c108720 msgr2=0x7fae8c10e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae89d9b700 1 --2- 172.21.15.31:0/3297387193 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae8c108720 0x7fae8c10e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae89d9b700 1 -- 172.21.15.31:0/3297387193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae8c10ec40 con 0x7fae8c102860 2024-04-26T18:13:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae89d9b700 1 --2- 172.21.15.31:0/3297387193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae8c102860 0x7fae8c1a0b90 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fae80000c00 tx=0x7fae800047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae7b7fe700 1 -- 172.21.15.31:0/3297387193 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80027070 con 0x7fae8c102860 2024-04-26T18:13:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae8b59e700 1 -- 172.21.15.31:0/3297387193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae8c10eed0 con 0x7fae8c102860 2024-04-26T18:13:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.904+0000 7fae7b7fe700 1 -- 172.21.15.31:0/3297387193 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae800040e0 con 0x7fae8c102860 2024-04-26T18:13:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.905+0000 7fae7b7fe700 1 -- 172.21.15.31:0/3297387193 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8000f460 con 0x7fae8c102860 2024-04-26T18:13:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.905+0000 7fae8b59e700 1 -- 172.21.15.31:0/3297387193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae8c19c270 con 0x7fae8c102860 2024-04-26T18:13:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.906+0000 7fae8b59e700 1 -- 172.21.15.31:0/3297387193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae8c06b520 con 0x7fae8c102860 2024-04-26T18:13:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.907+0000 7fae7b7fe700 1 -- 172.21.15.31:0/3297387193 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fae8002a070 con 0x7fae8c102860 2024-04-26T18:13:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.907+0000 7fae7b7fe700 1 --2- 172.21.15.31:0/3297387193 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fae740a3e30 0x7fae740a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.908+0000 7fae8a59c700 1 --2- 172.21.15.31:0/3297387193 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fae740a3e30 0x7fae740a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.908+0000 7fae7b7fe700 1 -- 172.21.15.31:0/3297387193 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8656+0+0 (secure 0 0 0) 0x7fae800d1880 con 0x7fae8c102860 2024-04-26T18:13:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.908+0000 7fae8a59c700 1 --2- 172.21.15.31:0/3297387193 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fae740a3e30 0x7fae740a62f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00ac50 tx=0x7fae7c00a380 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:19.912+0000 7fae7b7fe700 1 -- 172.21.15.31:0/3297387193 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fae8005eff0 con 0x7fae8c102860 2024-04-26T18:13:20.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:20.140+0000 7fae8b59e700 1 -- 172.21.15.31:0/3297387193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fae8c198b40 con 0x7fae8c102860 2024-04-26T18:13:20.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:20.141+0000 7fae7b7fe700 1 -- 172.21.15.31:0/3297387193 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 20 entries v113) ==== 81+0+1183 (secure 0 0 0) 0x7fae80013070 con 0x7fae8c102860 2024-04-26T18:13:20.141 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.186:6837/1510454985 2024-04-27T18:04:55.814793+0000 2024-04-26T18:13:20.141 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.97:6839/3765855684 2024-04-27T18:04:17.185934+0000 2024-04-26T18:13:20.141 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.97:6838/3765855684 2024-04-27T18:04:17.185934+0000 2024-04-26T18:13:20.141 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3101596726 2024-04-27T17:45:21.129062+0000 2024-04-26T18:13:20.141 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/288292084 2024-04-27T17:45:48.218816+0000 2024-04-26T18:13:20.142 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/545710545 2024-04-27T17:45:21.129062+0000 2024-04-26T18:13:20.142 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3148082955 2024-04-27T17:45:21.129062+0000 2024-04-26T18:13:20.142 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/810461143 2024-04-27T17:45:21.129062+0000 2024-04-26T18:13:20.142 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.186:6836/1510454985 2024-04-27T18:04:55.814793+0000 2024-04-26T18:13:20.142 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/153341213 2024-04-27T17:45:48.218816+0000 2024-04-26T18:13:20.142 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/153341213 2024-04-27T17:45:48.218816+0000 2024-04-26T18:13:20.142 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/614760140 2024-04-27T17:56:29.776588+0000 2024-04-26T18:13:20.142 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/718966776 2024-04-27T17:45:48.218816+0000 2024-04-26T18:13:20.142 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3429421652 2024-04-27T17:56:29.776588+0000 2024-04-26T18:13:20.142 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/2035551671 2024-04-27T17:45:48.218816+0000 2024-04-26T18:13:20.142 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/2095760002 2024-04-27T17:56:29.776588+0000 2024-04-26T18:13:20.142 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/545710545 2024-04-27T17:45:21.129062+0000 2024-04-26T18:13:20.143 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/540352687 2024-04-27T17:56:29.776588+0000 2024-04-26T18:13:20.143 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/2410008001 2024-04-27T17:56:29.776588+0000 2024-04-26T18:13:20.143 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/2410008001 2024-04-27T17:56:29.776588+0000 2024-04-26T18:13:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:20.143+0000 7fae8b59e700 1 -- 172.21.15.31:0/3297387193 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fae740a3e30 msgr2=0x7fae740a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:20.143+0000 7fae8b59e700 1 --2- 172.21.15.31:0/3297387193 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fae740a3e30 0x7fae740a62f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00ac50 tx=0x7fae7c00a380 comp rx=0 tx=0).stop 2024-04-26T18:13:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:20.143+0000 7fae8b59e700 1 -- 172.21.15.31:0/3297387193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae8c102860 msgr2=0x7fae8c1a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:20.143+0000 7fae8b59e700 1 --2- 172.21.15.31:0/3297387193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae8c102860 0x7fae8c1a0b90 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fae80000c00 tx=0x7fae800047b0 comp rx=0 tx=0).stop 2024-04-26T18:13:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:20.143+0000 7fae8b59e700 1 -- 172.21.15.31:0/3297387193 shutdown_connections 2024-04-26T18:13:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:20.143+0000 7fae8b59e700 1 --2- 172.21.15.31:0/3297387193 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fae740a3e30 0x7fae740a62f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:20.143+0000 7fae8b59e700 1 --2- 172.21.15.31:0/3297387193 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae8c108720 0x7fae8c10e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:20.143+0000 7fae8b59e700 1 --2- 172.21.15.31:0/3297387193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae8c101eb0 0x7fae8c1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:20.143+0000 7fae8b59e700 1 --2- 172.21.15.31:0/3297387193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae8c102860 0x7fae8c1a0b90 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:20.143+0000 7fae8b59e700 1 -- 172.21.15.31:0/3297387193 >> 172.21.15.31:0/3297387193 conn(0x7fae8c0775d0 msgr2=0x7fae8c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:20.143+0000 7fae8b59e700 1 -- 172.21.15.31:0/3297387193 shutdown_connections 2024-04-26T18:13:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:20.143+0000 7fae8b59e700 1 -- 172.21.15.31:0/3297387193 wait complete. 2024-04-26T18:13:20.145 INFO:teuthology.orchestra.run.smithi031.stderr:listed 20 entries 2024-04-26T18:13:20.519 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:20 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3297387193' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T18:13:20.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:20 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3297387193' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T18:13:20.816 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T18:13:20.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T18:13:20.817 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /tmp/cephfs 2024-04-26T18:13:20.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:20 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3297387193' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T18:13:21.054 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-04-26T18:13:21.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T18:13:21.054 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /tmp/cephfs) 2024-04-26T18:13:21.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T18:13:21.100 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-04-26T18:13:21.157 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--tmp-cephfs (id: 0) 2024-04-26T18:13:21.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T18:13:21.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns delete ceph-ns--tmp-cephfs 2024-04-26T18:13:21.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T18:13:21.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-04-26T18:13:21.344 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-26T18:13:21.344 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-26T18:13:21.344 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T18:13:21.383 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056882,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1390 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/64a19b96bb23b04153432808710ab1229eac3d485c26bb020baefc38fb6aa525/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ITK5S3LEIXMZ7W52LCHNMDHUJZ:/var/lib/containers/storage/overlay/l/GH7UOZ4HHT5VANLADRQF6OROJN,upperdir=/var/lib/containers/storage/overlay/64a19b96bb23b04153432808710ab1229eac3d485c26bb020baefc38fb6aa525/diff,workdir=/var/lib/containers/storage/overlay/64a19b96bb23b04153432808710ab1229eac3d485c26bb020baefc38fb6aa525/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/3986cf343e807534c0d6794ba2fcfddea794c7309c88cc95601428d9773ea882/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ITK5S3LEIXMZ7W52LCHNMDHUJZ:/var/lib/containers/storage/overlay/l/GH7UOZ4HHT5VANLADRQF6OROJN,upperdir=/var/lib/containers/storage/overlay/3986cf343e807534c0d6794ba2fcfddea794c7309c88cc95601428d9773ea882/diff,workdir=/var/lib/containers/storage/overlay/3986cf343e807534c0d6794ba2fcfddea794c7309c88cc95601428d9773ea882/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/4adaf1546562432d2eaa915074f5773e102a1a0d96590648b33a3b28c728545f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ITK5S3LEIXMZ7W52LCHNMDHUJZ:/var/lib/containers/storage/overlay/l/GH7UOZ4HHT5VANLADRQF6OROJN,upperdir=/var/lib/containers/storage/overlay/4adaf1546562432d2eaa915074f5773e102a1a0d96590648b33a3b28c728545f/diff,workdir=/var/lib/containers/storage/overlay/4adaf1546562432d2eaa915074f5773e102a1a0d96590648b33a3b28c728545f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b234eb93a8b4e7d5489be248aa96ef1d2223b3b3a7bed9e1b11f70c596dfea01/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ITK5S3LEIXMZ7W52LCHNMDHUJZ:/var/lib/containers/storage/overlay/l/GH7UOZ4HHT5VANLADRQF6OROJN,upperdir=/var/lib/containers/storage/overlay/b234eb93a8b4e7d5489be248aa96ef1d2223b3b3a7bed9e1b11f70c596dfea01/diff,workdir=/var/lib/containers/storage/overlay/b234eb93a8b4e7d5489be248aa96ef1d2223b3b3a7bed9e1b11f70c596dfea01/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/dd83d08f694af5ca56417c6533d1d83f274f3aa2fec6c31c15e0341d00e8f88c/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ITK5S3LEIXMZ7W52LCHNMDHUJZ:/var/lib/containers/storage/overlay/l/GH7UOZ4HHT5VANLADRQF6OROJN,upperdir=/var/lib/containers/storage/overlay/dd83d08f694af5ca56417c6533d1d83f274f3aa2fec6c31c15e0341d00e8f88c/diff,workdir=/var/lib/containers/storage/overlay/dd83d08f694af5ca56417c6533d1d83f274f3aa2fec6c31c15e0341d00e8f88c/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/55fef7135b3d1f232879629bbbf274e91b4b606c82fbbd7edbda9681885a1f2a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ITK5S3LEIXMZ7W52LCHNMDHUJZ:/var/lib/containers/storage/overlay/l/GH7UOZ4HHT5VANLADRQF6OROJN,upperdir=/var/lib/containers/storage/overlay/55fef7135b3d1f232879629bbbf274e91b4b606c82fbbd7edbda9681885a1f2a/diff,workdir=/var/lib/containers/storage/overlay/55fef7135b3d1f232879629bbbf274e91b4b606c82fbbd7edbda9681885a1f2a/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/335e7df99a47d84467d0859a57e609d97ed8223da305411ae8d9b90176ab2eec/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ITK5S3LEIXMZ7W52LCHNMDHUJZ:/var/lib/containers/storage/overlay/l/GH7UOZ4HHT5VANLADRQF6OROJN,upperdir=/var/lib/containers/storage/overlay/335e7df99a47d84467d0859a57e609d97ed8223da305411ae8d9b90176ab2eec/diff,workdir=/var/lib/containers/storage/overlay/335e7df99a47d84467d0859a57e609d97ed8223da305411ae8d9b90176ab2eec/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4924340dc62b51753d28873274012305bebb68113b199d4d8c65dfa0ad9b3c7e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ITK5S3LEIXMZ7W52LCHNMDHUJZ:/var/lib/containers/storage/overlay/l/GH7UOZ4HHT5VANLADRQF6OROJN,upperdir=/var/lib/containers/storage/overlay/4924340dc62b51753d28873274012305bebb68113b199d4d8c65dfa0ad9b3c7e/diff,workdir=/var/lib/containers/storage/overlay/4924340dc62b51753d28873274012305bebb68113b199d4d8c65dfa0ad9b3c7e/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d868ee8ae8c134bfd42a8f850c1ed5763405803150116eabec212b6799bc7861/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ITK5S3LEIXMZ7W52LCHNMDHUJZ:/var/lib/containers/storage/overlay/l/GH7UOZ4HHT5VANLADRQF6OROJN,upperdir=/var/lib/containers/storage/overlay/d868ee8ae8c134bfd42a8f850c1ed5763405803150116eabec212b6799bc7861/diff,workdir=/var/lib/containers/storage/overlay/d868ee8ae8c134bfd42a8f850c1ed5763405803150116eabec212b6799bc7861/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/758b4b930ad0be1fec658cece77e6ea03f3a837520af5d7faa37c6cc3c409225/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ITK5S3LEIXMZ7W52LCHNMDHUJZ:/var/lib/containers/storage/overlay/l/GH7UOZ4HHT5VANLADRQF6OROJN,upperdir=/var/lib/containers/storage/overlay/758b4b930ad0be1fec658cece77e6ea03f3a837520af5d7faa37c6cc3c409225/diff,workdir=/var/lib/containers/storage/overlay/758b4b930ad0be1fec658cece77e6ea03f3a837520af5d7faa37c6cc3c409225/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 2024-04-26T18:13:21.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T18:13:21.383 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-04-26T18:13:21.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T18:13:21.472 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-04-26T18:13:21.538 INFO:teuthology.orchestra.run.smithi031.stderr:Cannot find device "ceph-brx" 2024-04-26T18:13:21.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T18:13:21.541 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-26T18:13:21.554 INFO:tasks.cephadm:Teardown begin 2024-04-26T18:13:21.554 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T18:13:21.555 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T18:13:21.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:21 smithi031 ceph-mon[29164]: pgmap v530: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-04-26T18:13:21.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:21 smithi186 ceph-mon[32797]: pgmap v530: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-04-26T18:13:21.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:21 smithi097 ceph-mon[32419]: pgmap v530: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-04-26T18:13:21.894 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:13:23.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:23 smithi031 ceph-mon[29164]: pgmap v531: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 28 KiB/s wr, 7 op/s 2024-04-26T18:13:23.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:23 smithi186 ceph-mon[32797]: pgmap v531: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 28 KiB/s wr, 7 op/s 2024-04-26T18:13:23.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.790+0000 7f1a007a1700 1 -- 172.21.15.31:0/316469756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f8103e30 msgr2=0x7f19f8110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:23.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.790+0000 7f1a007a1700 1 --2- 172.21.15.31:0/316469756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f8103e30 0x7f19f8110290 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f19f000b0a0 tx=0x7f19f000b3b0 comp rx=0 tx=0).stop 2024-04-26T18:13:23.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.790+0000 7f1a007a1700 1 -- 172.21.15.31:0/316469756 shutdown_connections 2024-04-26T18:13:23.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.790+0000 7f1a007a1700 1 --2- 172.21.15.31:0/316469756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19f81094c0 0x7f19f81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:23.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.790+0000 7f1a007a1700 1 --2- 172.21.15.31:0/316469756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f8103e30 0x7f19f8110290 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:23.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.790+0000 7f1a007a1700 1 --2- 172.21.15.31:0/316469756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f8103470 0x7f19f81038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:23.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.790+0000 7f1a007a1700 1 -- 172.21.15.31:0/316469756 >> 172.21.15.31:0/316469756 conn(0x7f19f807a580 msgr2=0x7f19f807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:23.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.790+0000 7f1a007a1700 1 -- 172.21.15.31:0/316469756 shutdown_connections 2024-04-26T18:13:23.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.790+0000 7f1a007a1700 1 -- 172.21.15.31:0/316469756 wait complete. 2024-04-26T18:13:23.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f1a007a1700 1 Processor -- start 2024-04-26T18:13:23.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f1a007a1700 1 -- start start 2024-04-26T18:13:23.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f1a007a1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f8103470 0x7f19f81a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:23.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f1a007a1700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19f8103e30 0x7f19f81a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:23.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f1a007a1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f81094c0 0x7f19f819bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:23.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f1a007a1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f19f81135c0 con 0x7f19f8103470 2024-04-26T18:13:23.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f1a007a1700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f19f8113440 con 0x7f19f8103e30 2024-04-26T18:13:23.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f1a007a1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f19f8113740 con 0x7f19f81094c0 2024-04-26T18:13:23.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f19fe53d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f8103470 0x7f19f81a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:23.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f19fe53d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f8103470 0x7f19f81a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59512/0 (socket says 172.21.15.31:59512) 2024-04-26T18:13:23.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f19fe53d700 1 -- 172.21.15.31:0/363648309 learned_addr learned my addr 172.21.15.31:0/363648309 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:13:23.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f19fdd3c700 1 --2- 172.21.15.31:0/363648309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19f8103e30 0x7f19f81a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:23.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f19fed3e700 1 --2- 172.21.15.31:0/363648309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f81094c0 0x7f19f819bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:23.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f19fe53d700 1 -- 172.21.15.31:0/363648309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f81094c0 msgr2=0x7f19f819bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:23.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f19fe53d700 1 --2- 172.21.15.31:0/363648309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f81094c0 0x7f19f819bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:23.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f19fe53d700 1 -- 172.21.15.31:0/363648309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19f8103e30 msgr2=0x7f19f81a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:23.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f19fe53d700 1 --2- 172.21.15.31:0/363648309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19f8103e30 0x7f19f81a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:23.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.791+0000 7f19fe53d700 1 -- 172.21.15.31:0/363648309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19f819c220 con 0x7f19f8103470 2024-04-26T18:13:23.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.792+0000 7f19fe53d700 1 --2- 172.21.15.31:0/363648309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f8103470 0x7f19f81a1900 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f19f400d950 tx=0x7f19f400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:23.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.792+0000 7f19ef7fe700 1 -- 172.21.15.31:0/363648309 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f4016070 con 0x7f19f8103470 2024-04-26T18:13:23.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.792+0000 7f1a007a1700 1 -- 172.21.15.31:0/363648309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19f819c510 con 0x7f19f8103470 2024-04-26T18:13:23.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.792+0000 7f19ef7fe700 1 -- 172.21.15.31:0/363648309 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19f400fd40 con 0x7f19f8103470 2024-04-26T18:13:23.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.792+0000 7f19ef7fe700 1 -- 172.21.15.31:0/363648309 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f400e3b0 con 0x7f19f8103470 2024-04-26T18:13:23.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.792+0000 7f1a007a1700 1 -- 172.21.15.31:0/363648309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19f81a87d0 con 0x7f19f8103470 2024-04-26T18:13:23.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.793+0000 7f1a007a1700 1 -- 172.21.15.31:0/363648309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19f806a110 con 0x7f19f8103470 2024-04-26T18:13:23.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.794+0000 7f19ef7fe700 1 -- 172.21.15.31:0/363648309 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f19f4019070 con 0x7f19f8103470 2024-04-26T18:13:23.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.795+0000 7f19ef7fe700 1 --2- 172.21.15.31:0/363648309 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f19e40a3db0 0x7f19e40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:23.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.795+0000 7f19fdd3c700 1 --2- 172.21.15.31:0/363648309 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f19e40a3db0 0x7f19e40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:23.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.795+0000 7f19ef7fe700 1 -- 172.21.15.31:0/363648309 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8656+0+0 (secure 0 0 0) 0x7f19f40d0270 con 0x7f19f8103470 2024-04-26T18:13:23.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.795+0000 7f19fdd3c700 1 --2- 172.21.15.31:0/363648309 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f19e40a3db0 0x7f19e40a6270 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f19f807ae80 tx=0x7f19e800b410 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:23.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:23.799+0000 7f19ef7fe700 1 -- 172.21.15.31:0/363648309 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f19f409c890 con 0x7f19f8103470 2024-04-26T18:13:23.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:23 smithi097 ceph-mon[32419]: pgmap v531: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 28 KiB/s wr, 7 op/s 2024-04-26T18:13:24.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:24.056+0000 7f1a007a1700 1 -- 172.21.15.31:0/363648309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19f81a8a00 con 0x7f19f8103470 2024-04-26T18:13:24.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:24.057+0000 7f19ef7fe700 1 -- 172.21.15.31:0/363648309 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11878 (secure 0 0 0) 0x7f19f409c190 con 0x7f19f8103470 2024-04-26T18:13:24.057 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:13:24.058 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T18:12:59.935009+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":25799,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25747":{"gid":25747,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1162255559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1162255559},{"type":"v1","addr":"172.21.15.186:6837","nonce":1162255559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25799":{"gid":25799,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":14,"addr":"172.21.15.97:6839/2506986520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2506986520},{"type":"v1","addr":"172.21.15.97:6839","nonce":2506986520}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":25799,"qdb_cluster":[24446,24457,25799]},"id":1}]} 2024-04-26T18:13:24.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:24.059+0000 7f1a007a1700 1 -- 172.21.15.31:0/363648309 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f19e40a3db0 msgr2=0x7f19e40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:24.059+0000 7f1a007a1700 1 --2- 172.21.15.31:0/363648309 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f19e40a3db0 0x7f19e40a6270 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f19f807ae80 tx=0x7f19e800b410 comp rx=0 tx=0).stop 2024-04-26T18:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:24.059+0000 7f1a007a1700 1 -- 172.21.15.31:0/363648309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f8103470 msgr2=0x7f19f81a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:24.059+0000 7f1a007a1700 1 --2- 172.21.15.31:0/363648309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f8103470 0x7f19f81a1900 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f19f400d950 tx=0x7f19f400dc60 comp rx=0 tx=0).stop 2024-04-26T18:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:24.059+0000 7f1a007a1700 1 -- 172.21.15.31:0/363648309 shutdown_connections 2024-04-26T18:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:24.059+0000 7f1a007a1700 1 --2- 172.21.15.31:0/363648309 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f19e40a3db0 0x7f19e40a6270 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:24.059+0000 7f1a007a1700 1 --2- 172.21.15.31:0/363648309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f81094c0 0x7f19f819bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:24.059+0000 7f1a007a1700 1 --2- 172.21.15.31:0/363648309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f8103470 0x7f19f81a1900 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:24.059+0000 7f1a007a1700 1 --2- 172.21.15.31:0/363648309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19f8103e30 0x7f19f81a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:24.059+0000 7f1a007a1700 1 -- 172.21.15.31:0/363648309 >> 172.21.15.31:0/363648309 conn(0x7f19f807a580 msgr2=0x7f19f810e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:24.059+0000 7f1a007a1700 1 -- 172.21.15.31:0/363648309 shutdown_connections 2024-04-26T18:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:24.059+0000 7f1a007a1700 1 -- 172.21.15.31:0/363648309 wait complete. 2024-04-26T18:13:24.061 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 56 2024-04-26T18:13:24.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:24 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/363648309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:13:24.687 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T18:13:24.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:24 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/363648309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:13:24.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:24 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/363648309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:13:24.983 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:13:25.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:25 smithi186 ceph-mon[32797]: pgmap v532: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 28 KiB/s wr, 7 op/s 2024-04-26T18:13:25.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:25 smithi031 ceph-mon[29164]: pgmap v532: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 28 KiB/s wr, 7 op/s 2024-04-26T18:13:25.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:25 smithi097 ceph-mon[32419]: pgmap v532: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 28 KiB/s wr, 7 op/s 2024-04-26T18:13:26.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.848+0000 7f148b002700 1 -- 172.21.15.31:0/3531248432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1484103470 msgr2=0x7f14841038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:26.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.848+0000 7f148b002700 1 --2- 172.21.15.31:0/3531248432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1484103470 0x7f14841038f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f1478009990 tx=0x7f1478009ca0 comp rx=0 tx=0).stop 2024-04-26T18:13:26.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.848+0000 7f148b002700 1 -- 172.21.15.31:0/3531248432 shutdown_connections 2024-04-26T18:13:26.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.848+0000 7f148b002700 1 --2- 172.21.15.31:0/3531248432 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14841094c0 0x7f14841098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:26.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.848+0000 7f148b002700 1 --2- 172.21.15.31:0/3531248432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1484103e30 0x7f1484110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:26.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.848+0000 7f148b002700 1 --2- 172.21.15.31:0/3531248432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1484103470 0x7f14841038f0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:26.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.848+0000 7f148b002700 1 -- 172.21.15.31:0/3531248432 >> 172.21.15.31:0/3531248432 conn(0x7f148407a580 msgr2=0x7f148407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:26.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.848+0000 7f148b002700 1 -- 172.21.15.31:0/3531248432 shutdown_connections 2024-04-26T18:13:26.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.848+0000 7f148b002700 1 -- 172.21.15.31:0/3531248432 wait complete. 2024-04-26T18:13:26.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.849+0000 7f148b002700 1 Processor -- start 2024-04-26T18:13:26.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.849+0000 7f148b002700 1 -- start start 2024-04-26T18:13:26.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.849+0000 7f148b002700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1484103470 0x7f1484075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:26.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.849+0000 7f148b002700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1484103e30 0x7f1484075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:26.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.849+0000 7f148b002700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14841094c0 0x7f14840761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:26.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.849+0000 7f148b002700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f14841135c0 con 0x7f1484103e30 2024-04-26T18:13:26.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.849+0000 7f148b002700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f1484113440 con 0x7f14841094c0 2024-04-26T18:13:26.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.849+0000 7f148b002700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1484113740 con 0x7f1484103470 2024-04-26T18:13:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.849+0000 7f1483fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1484103e30 0x7f1484075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.849+0000 7f1483fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1484103e30 0x7f1484075c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38474/0 (socket says 172.21.15.31:38474) 2024-04-26T18:13:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.849+0000 7f1483fff700 1 -- 172.21.15.31:0/359109312 learned_addr learned my addr 172.21.15.31:0/359109312 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:13:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.850+0000 7f1488d9e700 1 --2- 172.21.15.31:0/359109312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1484103470 0x7f1484075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.850+0000 7f148959f700 1 --2- 172.21.15.31:0/359109312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14841094c0 0x7f14840761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.850+0000 7f1483fff700 1 -- 172.21.15.31:0/359109312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1484103470 msgr2=0x7f1484075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.850+0000 7f1483fff700 1 --2- 172.21.15.31:0/359109312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1484103470 0x7f1484075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.850+0000 7f1483fff700 1 -- 172.21.15.31:0/359109312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14841094c0 msgr2=0x7f14840761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.850+0000 7f1483fff700 1 --2- 172.21.15.31:0/359109312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14841094c0 0x7f14840761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.850+0000 7f1483fff700 1 -- 172.21.15.31:0/359109312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14841a7f60 con 0x7f1484103e30 2024-04-26T18:13:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.850+0000 7f1483fff700 1 --2- 172.21.15.31:0/359109312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1484103e30 0x7f1484075c60 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f14780057f0 tx=0x7f1478003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.850+0000 7f1481ffb700 1 -- 172.21.15.31:0/359109312 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1478027070 con 0x7f1484103e30 2024-04-26T18:13:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.850+0000 7f148b002700 1 -- 172.21.15.31:0/359109312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14841a81f0 con 0x7f1484103e30 2024-04-26T18:13:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.850+0000 7f1481ffb700 1 -- 172.21.15.31:0/359109312 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14780041c0 con 0x7f1484103e30 2024-04-26T18:13:26.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.850+0000 7f1481ffb700 1 -- 172.21.15.31:0/359109312 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147800f460 con 0x7f1484103e30 2024-04-26T18:13:26.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.850+0000 7f148b002700 1 -- 172.21.15.31:0/359109312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14841a86d0 con 0x7f1484103e30 2024-04-26T18:13:26.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.851+0000 7f148b002700 1 -- 172.21.15.31:0/359109312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f148406a110 con 0x7f1484103e30 2024-04-26T18:13:26.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.853+0000 7f1481ffb700 1 -- 172.21.15.31:0/359109312 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f147802a070 con 0x7f1484103e30 2024-04-26T18:13:26.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.853+0000 7f1481ffb700 1 --2- 172.21.15.31:0/359109312 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f146c0a3db0 0x7f146c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:26.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.853+0000 7f1481ffb700 1 -- 172.21.15.31:0/359109312 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8656+0+0 (secure 0 0 0) 0x7f14780d2000 con 0x7f1484103e30 2024-04-26T18:13:26.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.853+0000 7f1488d9e700 1 --2- 172.21.15.31:0/359109312 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f146c0a3db0 0x7f146c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:26.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.854+0000 7f1488d9e700 1 --2- 172.21.15.31:0/359109312 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f146c0a3db0 0x7f146c0a6270 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1474005fd0 tx=0x7f147400af20 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:26.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:26.857+0000 7f1481ffb700 1 -- 172.21.15.31:0/359109312 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1478031020 con 0x7f1484103e30 2024-04-26T18:13:27.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:27.119+0000 7f148b002700 1 -- 172.21.15.31:0/359109312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1484076c50 con 0x7f1484103e30 2024-04-26T18:13:27.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:27.120+0000 7f1481ffb700 1 -- 172.21.15.31:0/359109312 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11878 (secure 0 0 0) 0x7f1478013070 con 0x7f1484103e30 2024-04-26T18:13:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:13:27.121 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14508,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3371658851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3371658851},{"type":"v1","addr":"172.21.15.31:6835","nonce":3371658851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14520,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1425597980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1425597980},{"type":"v1","addr":"172.21.15.31:6837","nonce":1425597980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/2783481787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":2783481787},{"type":"v1","addr":"172.21.15.31:6839","nonce":2783481787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2984904862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2984904862},{"type":"v1","addr":"172.21.15.31:6841","nonce":2984904862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/2883739522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2883739522},{"type":"v1","addr":"172.21.15.97:6833","nonce":2883739522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1597494912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1597494912},{"type":"v1","addr":"172.21.15.186:6833","nonce":1597494912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T17:53:05.425897+0000","modified":"2024-04-26T18:12:59.935009+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":25799,"mds_1":24457,"mds_2":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24428":{"gid":24428,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6835/1721162755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1721162755},{"type":"v1","addr":"172.21.15.97:6835","nonce":1721162755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/580433615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":580433615},{"type":"v1","addr":"172.21.15.186:6835","nonce":580433615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.97:6837/3334702590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":3334702590},{"type":"v1","addr":"172.21.15.97:6837","nonce":3334702590}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.186:6839/1587952500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":1587952500},{"type":"v1","addr":"172.21.15.186:6839","nonce":1587952500}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25747":{"gid":25747,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/1162255559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":1162255559},{"type":"v1","addr":"172.21.15.186:6837","nonce":1162255559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25799":{"gid":25799,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":14,"addr":"172.21.15.97:6839/2506986520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2506986520},{"type":"v1","addr":"172.21.15.97:6839","nonce":2506986520}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":25799,"qdb_cluster":[24446,24457,25799]},"id":1}]} 2024-04-26T18:13:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:27.122+0000 7f148b002700 1 -- 172.21.15.31:0/359109312 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f146c0a3db0 msgr2=0x7f146c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:27.122+0000 7f148b002700 1 --2- 172.21.15.31:0/359109312 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f146c0a3db0 0x7f146c0a6270 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1474005fd0 tx=0x7f147400af20 comp rx=0 tx=0).stop 2024-04-26T18:13:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:27.122+0000 7f148b002700 1 -- 172.21.15.31:0/359109312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1484103e30 msgr2=0x7f1484075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:27.122+0000 7f148b002700 1 --2- 172.21.15.31:0/359109312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1484103e30 0x7f1484075c60 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f14780057f0 tx=0x7f1478003730 comp rx=0 tx=0).stop 2024-04-26T18:13:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:27.122+0000 7f148b002700 1 -- 172.21.15.31:0/359109312 shutdown_connections 2024-04-26T18:13:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:27.122+0000 7f148b002700 1 --2- 172.21.15.31:0/359109312 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f146c0a3db0 0x7f146c0a6270 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:27.122+0000 7f148b002700 1 --2- 172.21.15.31:0/359109312 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14841094c0 0x7f14840761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:27.122+0000 7f148b002700 1 --2- 172.21.15.31:0/359109312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1484103470 0x7f1484075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:27.122+0000 7f148b002700 1 --2- 172.21.15.31:0/359109312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1484103e30 0x7f1484075c60 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:27.122+0000 7f148b002700 1 -- 172.21.15.31:0/359109312 >> 172.21.15.31:0/359109312 conn(0x7f148407a580 msgr2=0x7f14841020d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:27.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:27.123+0000 7f148b002700 1 -- 172.21.15.31:0/359109312 shutdown_connections 2024-04-26T18:13:27.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:27.123+0000 7f148b002700 1 -- 172.21.15.31:0/359109312 wait complete. 2024-04-26T18:13:27.124 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 56 2024-04-26T18:13:27.713 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:27 smithi031 ceph-mon[29164]: pgmap v533: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 28 KiB/s wr, 7 op/s 2024-04-26T18:13:27.713 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:27 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/359109312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:13:27.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T18:13:27.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:27 smithi186 ceph-mon[32797]: pgmap v533: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 28 KiB/s wr, 7 op/s 2024-04-26T18:13:27.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:27 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/359109312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:13:27.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:27 smithi097 ceph-mon[32419]: pgmap v533: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 28 KiB/s wr, 7 op/s 2024-04-26T18:13:27.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:27 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/359109312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T18:13:28.002 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:13:29.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:29 smithi031 ceph-mon[29164]: pgmap v534: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 78 KiB/s wr, 8 op/s 2024-04-26T18:13:29.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:29 smithi186 ceph-mon[32797]: pgmap v534: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 78 KiB/s wr, 8 op/s 2024-04-26T18:13:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.850+0000 7ffbb1767700 1 -- 172.21.15.31:0/1170527285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbac109500 msgr2=0x7ffbac1098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.850+0000 7ffbb1767700 1 --2- 172.21.15.31:0/1170527285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbac109500 0x7ffbac1098e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c009990 tx=0x7ffb9c009ca0 comp rx=0 tx=0).stop 2024-04-26T18:13:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.850+0000 7ffbb1767700 1 -- 172.21.15.31:0/1170527285 shutdown_connections 2024-04-26T18:13:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.850+0000 7ffbb1767700 1 --2- 172.21.15.31:0/1170527285 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffbac06c4f0 0x7ffbac1102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.850+0000 7ffbb1767700 1 --2- 172.21.15.31:0/1170527285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbac109500 0x7ffbac1098e0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.850+0000 7ffbb1767700 1 --2- 172.21.15.31:0/1170527285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbac06bb30 0x7ffbac06bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.850+0000 7ffbb1767700 1 -- 172.21.15.31:0/1170527285 >> 172.21.15.31:0/1170527285 conn(0x7ffbac079080 msgr2=0x7ffbac079490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.850+0000 7ffbb1767700 1 -- 172.21.15.31:0/1170527285 shutdown_connections 2024-04-26T18:13:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.850+0000 7ffbb1767700 1 -- 172.21.15.31:0/1170527285 wait complete. 2024-04-26T18:13:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.850+0000 7ffbb1767700 1 Processor -- start 2024-04-26T18:13:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbb1767700 1 -- start start 2024-04-26T18:13:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbb1767700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffbac06bb30 0x7ffbac19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbb1767700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbac06c4f0 0x7ffbac19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbb1767700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbac109500 0x7ffbac199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbb1767700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffbac1135e0 con 0x7ffbac06c4f0 2024-04-26T18:13:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbb1767700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7ffbac113460 con 0x7ffbac06bb30 2024-04-26T18:13:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbb1767700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffbac113760 con 0x7ffbac109500 2024-04-26T18:13:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbaa7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbac06c4f0 0x7ffbac19fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbaa7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbac06c4f0 0x7ffbac19fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38502/0 (socket says 172.21.15.31:38502) 2024-04-26T18:13:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbaa7fc700 1 -- 172.21.15.31:0/600929087 learned_addr learned my addr 172.21.15.31:0/600929087 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:13:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbab7fe700 1 --2- 172.21.15.31:0/600929087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbac109500 0x7ffbac199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbaaffd700 1 --2- 172.21.15.31:0/600929087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffbac06bb30 0x7ffbac19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbaa7fc700 1 -- 172.21.15.31:0/600929087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbac109500 msgr2=0x7ffbac199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbaa7fc700 1 --2- 172.21.15.31:0/600929087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbac109500 0x7ffbac199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbaa7fc700 1 -- 172.21.15.31:0/600929087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffbac06bb30 msgr2=0x7ffbac19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbaa7fc700 1 --2- 172.21.15.31:0/600929087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffbac06bb30 0x7ffbac19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbaa7fc700 1 -- 172.21.15.31:0/600929087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbac19a080 con 0x7ffbac06c4f0 2024-04-26T18:13:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.851+0000 7ffbaa7fc700 1 --2- 172.21.15.31:0/600929087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbac06c4f0 0x7ffbac19fca0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ffba000d950 tx=0x7ffba000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.852+0000 7ffb93fff700 1 -- 172.21.15.31:0/600929087 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba0016070 con 0x7ffbac06c4f0 2024-04-26T18:13:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.852+0000 7ffbb1767700 1 -- 172.21.15.31:0/600929087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbac19a370 con 0x7ffbac06c4f0 2024-04-26T18:13:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.852+0000 7ffb93fff700 1 -- 172.21.15.31:0/600929087 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffba000fd40 con 0x7ffbac06c4f0 2024-04-26T18:13:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.852+0000 7ffb93fff700 1 -- 172.21.15.31:0/600929087 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba000e3b0 con 0x7ffbac06c4f0 2024-04-26T18:13:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.852+0000 7ffbb1767700 1 -- 172.21.15.31:0/600929087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbac1a6630 con 0x7ffbac06c4f0 2024-04-26T18:13:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.853+0000 7ffbb1767700 1 -- 172.21.15.31:0/600929087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbac06a110 con 0x7ffbac06c4f0 2024-04-26T18:13:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.854+0000 7ffb93fff700 1 -- 172.21.15.31:0/600929087 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffba0019070 con 0x7ffbac06c4f0 2024-04-26T18:13:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.855+0000 7ffb93fff700 1 --2- 172.21.15.31:0/600929087 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ffb940a3db0 0x7ffb940a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.855+0000 7ffbaaffd700 1 --2- 172.21.15.31:0/600929087 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ffb940a3db0 0x7ffb940a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:29.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.855+0000 7ffb93fff700 1 -- 172.21.15.31:0/600929087 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8656+0+0 (secure 0 0 0) 0x7ffba00d0380 con 0x7ffbac06c4f0 2024-04-26T18:13:29.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.855+0000 7ffbaaffd700 1 --2- 172.21.15.31:0/600929087 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ffb940a3db0 0x7ffb940a6270 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c00b5c0 tx=0x7ffb9c005960 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:29.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:29.859+0000 7ffb93fff700 1 -- 172.21.15.31:0/600929087 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffba009c960 con 0x7ffbac06c4f0 2024-04-26T18:13:29.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:29 smithi097 ceph-mon[32419]: pgmap v534: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 78 KiB/s wr, 8 op/s 2024-04-26T18:13:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:30.070+0000 7ffbb1767700 1 -- 172.21.15.31:0/600929087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffbac02e490 con 0x7ffbac06c4f0 2024-04-26T18:13:30.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:30.072+0000 7ffb93fff700 1 -- 172.21.15.31:0/600929087 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+23309 (secure 0 0 0) 0x7ffba009c260 con 0x7ffbac06c4f0 2024-04-26T18:13:30.073 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-26T18:13:30.073 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":113,"fsid":"7e53e050-03f4-11ef-bc93-c7b262605968","created":"2024-04-26T17:44:48.618250+0000","modified":"2024-04-26T18:13:18.855921+0000","last_up_change":"2024-04-26T17:52:00.798886+0000","last_in_change":"2024-04-26T17:51:40.613593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T17:48:12.307250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T17:52:58.650338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T17:53:02.488144+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":28,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f684ee5-eeee-454a-a00b-e5d6d2901572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6803","nonce":1957398328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6805","nonce":1957398328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6809","nonce":1957398328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1957398328},{"type":"v1","addr":"172.21.15.31:6807","nonce":1957398328}]},"public_addr":"172.21.15.31:6803/1957398328","cluster_addr":"172.21.15.31:6805/1957398328","heartbeat_back_addr":"172.21.15.31:6809/1957398328","heartbeat_front_addr":"172.21.15.31:6807/1957398328","state":["exists","up"]},{"osd":1,"uuid":"08be6b26-cb60-41f7-be8b-01a0c93ebafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6801","nonce":4209866427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6803","nonce":4209866427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6807","nonce":4209866427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":4209866427},{"type":"v1","addr":"172.21.15.97:6805","nonce":4209866427}]},"public_addr":"172.21.15.97:6801/4209866427","cluster_addr":"172.21.15.97:6803/4209866427","heartbeat_back_addr":"172.21.15.97:6807/4209866427","heartbeat_front_addr":"172.21.15.97:6805/4209866427","state":["exists","up"]},{"osd":2,"uuid":"498216b1-a0ac-4923-ade8-ce50fb97d472","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6801","nonce":223535147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6803","nonce":223535147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6807","nonce":223535147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":223535147},{"type":"v1","addr":"172.21.15.186:6805","nonce":223535147}]},"public_addr":"172.21.15.186:6801/223535147","cluster_addr":"172.21.15.186:6803/223535147","heartbeat_back_addr":"172.21.15.186:6807/223535147","heartbeat_front_addr":"172.21.15.186:6805/223535147","state":["exists","up"]},{"osd":3,"uuid":"568a352d-a430-4f0f-9a49-6f992e0ceae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6811","nonce":2640112341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6813","nonce":2640112341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6817","nonce":2640112341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2640112341},{"type":"v1","addr":"172.21.15.31:6815","nonce":2640112341}]},"public_addr":"172.21.15.31:6811/2640112341","cluster_addr":"172.21.15.31:6813/2640112341","heartbeat_back_addr":"172.21.15.31:6817/2640112341","heartbeat_front_addr":"172.21.15.31:6815/2640112341","state":["exists","up"]},{"osd":4,"uuid":"112ccbf7-24e8-473c-9852-2e7167dd3108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6809","nonce":1359263143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6811","nonce":1359263143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6815","nonce":1359263143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1359263143},{"type":"v1","addr":"172.21.15.97:6813","nonce":1359263143}]},"public_addr":"172.21.15.97:6809/1359263143","cluster_addr":"172.21.15.97:6811/1359263143","heartbeat_back_addr":"172.21.15.97:6815/1359263143","heartbeat_front_addr":"172.21.15.97:6813/1359263143","state":["exists","up"]},{"osd":5,"uuid":"eb17759e-1ad8-4bda-9f61-54d2a9781535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6809","nonce":2006134626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6811","nonce":2006134626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6815","nonce":2006134626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":2006134626},{"type":"v1","addr":"172.21.15.186:6813","nonce":2006134626}]},"public_addr":"172.21.15.186:6809/2006134626","cluster_addr":"172.21.15.186:6811/2006134626","heartbeat_back_addr":"172.21.15.186:6815/2006134626","heartbeat_front_addr":"172.21.15.186:6813/2006134626","state":["exists","up"]},{"osd":6,"uuid":"f1d11e38-40e9-4db4-868f-145f1ffc0d30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6819","nonce":2955759395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6821","nonce":2955759395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6825","nonce":2955759395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2955759395},{"type":"v1","addr":"172.21.15.31:6823","nonce":2955759395}]},"public_addr":"172.21.15.31:6819/2955759395","cluster_addr":"172.21.15.31:6821/2955759395","heartbeat_back_addr":"172.21.15.31:6825/2955759395","heartbeat_front_addr":"172.21.15.31:6823/2955759395","state":["exists","up"]},{"osd":7,"uuid":"4e95f728-23ff-4bca-9b94-afde2662a5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6817","nonce":3940083075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6819","nonce":3940083075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6823","nonce":3940083075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3940083075},{"type":"v1","addr":"172.21.15.97:6821","nonce":3940083075}]},"public_addr":"172.21.15.97:6817/3940083075","cluster_addr":"172.21.15.97:6819/3940083075","heartbeat_back_addr":"172.21.15.97:6823/3940083075","heartbeat_front_addr":"172.21.15.97:6821/3940083075","state":["exists","up"]},{"osd":8,"uuid":"5f91c915-2568-4d20-b645-00dad2efe952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6817","nonce":3193558170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6819","nonce":3193558170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6823","nonce":3193558170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3193558170},{"type":"v1","addr":"172.21.15.186:6821","nonce":3193558170}]},"public_addr":"172.21.15.186:6817/3193558170","cluster_addr":"172.21.15.186:6819/3193558170","heartbeat_back_addr":"172.21.15.186:6823/3193558170","heartbeat_front_addr":"172.21.15.186:6821/3193558170","state":["exists","up"]},{"osd":9,"uuid":"12fda87d-830a-461f-a6cb-4699c52dc5e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6827","nonce":3660801905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6829","nonce":3660801905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6833","nonce":3660801905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3660801905},{"type":"v1","addr":"172.21.15.31:6831","nonce":3660801905}]},"public_addr":"172.21.15.31:6827/3660801905","cluster_addr":"172.21.15.31:6829/3660801905","heartbeat_back_addr":"172.21.15.31:6833/3660801905","heartbeat_front_addr":"172.21.15.31:6831/3660801905","state":["exists","up"]},{"osd":10,"uuid":"92a639b3-918b-485c-bd30-258b4224993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6825","nonce":4097344903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6827","nonce":4097344903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6831","nonce":4097344903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":4097344903},{"type":"v1","addr":"172.21.15.97:6829","nonce":4097344903}]},"public_addr":"172.21.15.97:6825/4097344903","cluster_addr":"172.21.15.97:6827/4097344903","heartbeat_back_addr":"172.21.15.97:6831/4097344903","heartbeat_front_addr":"172.21.15.97:6829/4097344903","state":["exists","up"]},{"osd":11,"uuid":"842aa599-d21d-4e4a-b872-e9ac20d8c0ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6825","nonce":1063973352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6827","nonce":1063973352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6831","nonce":1063973352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1063973352},{"type":"v1","addr":"172.21.15.186:6829","nonce":1063973352}]},"public_addr":"172.21.15.186:6825/1063973352","cluster_addr":"172.21.15.186:6827/1063973352","heartbeat_back_addr":"172.21.15.186:6831/1063973352","heartbeat_front_addr":"172.21.15.186:6829/1063973352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:21.305331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:47:44.877932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:08.784665+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:34.381762+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:48:58.999358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:23.620237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:49:49.961719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:15.363702+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:50:40.599703+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:07.391645+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:32.800851+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T17:51:59.384148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.186:6837/1510454985":"2024-04-27T18:04:55.814793+0000","172.21.15.97:6839/3765855684":"2024-04-27T18:04:17.185934+0000","172.21.15.97:6838/3765855684":"2024-04-27T18:04:17.185934+0000","172.21.15.31:0/3101596726":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/288292084":"2024-04-27T17:45:48.218816+0000","172.21.15.31:6801/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/3148082955":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/810461143":"2024-04-27T17:45:21.129062+0000","172.21.15.186:6836/1510454985":"2024-04-27T18:04:55.814793+0000","172.21.15.31:6801/153341213":"2024-04-27T17:45:48.218816+0000","172.21.15.31:6800/153341213":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/614760140":"2024-04-27T17:56:29.776588+0000","172.21.15.31:0/718966776":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/3429421652":"2024-04-27T17:56:29.776588+0000","172.21.15.31:0/2035551671":"2024-04-27T17:45:48.218816+0000","172.21.15.31:0/2095760002":"2024-04-27T17:56:29.776588+0000","172.21.15.31:6800/545710545":"2024-04-27T17:45:21.129062+0000","172.21.15.31:0/540352687":"2024-04-27T17:56:29.776588+0000","172.21.15.31:6800/2410008001":"2024-04-27T17:56:29.776588+0000","172.21.15.31:6801/2410008001":"2024-04-27T17:56:29.776588+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":15,"length":1},{"begin":28,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T18:13:30.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:30.074+0000 7ffbb1767700 1 -- 172.21.15.31:0/600929087 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ffb940a3db0 msgr2=0x7ffb940a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:30.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:30.074+0000 7ffbb1767700 1 --2- 172.21.15.31:0/600929087 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ffb940a3db0 0x7ffb940a6270 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c00b5c0 tx=0x7ffb9c005960 comp rx=0 tx=0).stop 2024-04-26T18:13:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:30.074+0000 7ffbb1767700 1 -- 172.21.15.31:0/600929087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbac06c4f0 msgr2=0x7ffbac19fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:30.074+0000 7ffbb1767700 1 --2- 172.21.15.31:0/600929087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbac06c4f0 0x7ffbac19fca0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ffba000d950 tx=0x7ffba000dc60 comp rx=0 tx=0).stop 2024-04-26T18:13:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:30.074+0000 7ffbb1767700 1 -- 172.21.15.31:0/600929087 shutdown_connections 2024-04-26T18:13:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:30.074+0000 7ffbb1767700 1 --2- 172.21.15.31:0/600929087 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7ffb940a3db0 0x7ffb940a6270 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:30.074+0000 7ffbb1767700 1 --2- 172.21.15.31:0/600929087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffbac06bb30 0x7ffbac19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:30.074+0000 7ffbb1767700 1 --2- 172.21.15.31:0/600929087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbac109500 0x7ffbac199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:30.074+0000 7ffbb1767700 1 --2- 172.21.15.31:0/600929087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbac06c4f0 0x7ffbac19fca0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:30.074+0000 7ffbb1767700 1 -- 172.21.15.31:0/600929087 >> 172.21.15.31:0/600929087 conn(0x7ffbac079080 msgr2=0x7ffbac101440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:30.074+0000 7ffbb1767700 1 -- 172.21.15.31:0/600929087 shutdown_connections 2024-04-26T18:13:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:30.074+0000 7ffbb1767700 1 -- 172.21.15.31:0/600929087 wait complete. 2024-04-26T18:13:30.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:30 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/600929087' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T18:13:30.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs fail cephfs 2024-04-26T18:13:30.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:30 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/600929087' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T18:13:30.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:30 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/600929087' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T18:13:31.015 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:13:31.683 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:31 smithi186 ceph-mon[32797]: pgmap v535: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 69 KiB/s wr, 7 op/s 2024-04-26T18:13:31.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:31 smithi031 ceph-mon[29164]: pgmap v535: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 69 KiB/s wr, 7 op/s 2024-04-26T18:13:31.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:31 smithi097 ceph-mon[32419]: pgmap v535: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 69 KiB/s wr, 7 op/s 2024-04-26T18:13:32.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.722+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2502383420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de406b130 msgr2=0x7f3de406b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:32.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.722+0000 7f3deb3fb700 1 --2- 172.21.15.31:0/2502383420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de406b130 0x7f3de406b510 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f3de0009990 tx=0x7f3de0009ca0 comp rx=0 tx=0).stop 2024-04-26T18:13:32.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.722+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2502383420 shutdown_connections 2024-04-26T18:13:32.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.722+0000 7f3deb3fb700 1 --2- 172.21.15.31:0/2502383420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3de410fed0 0x7f3de41122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:32.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.722+0000 7f3deb3fb700 1 --2- 172.21.15.31:0/2502383420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de406b130 0x7f3de406b510 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:32.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.722+0000 7f3deb3fb700 1 --2- 172.21.15.31:0/2502383420 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de406ba50 0x7f3de410f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:32.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.722+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2502383420 >> 172.21.15.31:0/2502383420 conn(0x7f3de40792d0 msgr2=0x7f3de40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:32.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.722+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2502383420 shutdown_connections 2024-04-26T18:13:32.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.722+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2502383420 wait complete. 2024-04-26T18:13:32.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.723+0000 7f3deb3fb700 1 Processor -- start 2024-04-26T18:13:32.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.723+0000 7f3deb3fb700 1 -- start start 2024-04-26T18:13:32.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.723+0000 7f3deb3fb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de406b130 0x7f3de419bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:32.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.723+0000 7f3deb3fb700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de406ba50 0x7f3de419c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:32.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.723+0000 7f3deb3fb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3de410fed0 0x7f3de41a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:32.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.723+0000 7f3deb3fb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3de41157d0 con 0x7f3de406b130 2024-04-26T18:13:32.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.723+0000 7f3deb3fb700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f3de4115650 con 0x7f3de406ba50 2024-04-26T18:13:32.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.723+0000 7f3deb3fb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3de4115950 con 0x7f3de410fed0 2024-04-26T18:13:32.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.723+0000 7f3de9197700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de406b130 0x7f3de419bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:32.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.723+0000 7f3de9197700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de406b130 0x7f3de419bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38526/0 (socket says 172.21.15.31:38526) 2024-04-26T18:13:32.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.723+0000 7f3de9197700 1 -- 172.21.15.31:0/2127145533 learned_addr learned my addr 172.21.15.31:0/2127145533 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:13:32.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3de9998700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3de410fed0 0x7f3de41a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:32.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3de8996700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de406ba50 0x7f3de419c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:32.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3de9197700 1 -- 172.21.15.31:0/2127145533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3de410fed0 msgr2=0x7f3de41a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:32.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3de9197700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3de410fed0 0x7f3de41a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:32.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3de9197700 1 -- 172.21.15.31:0/2127145533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de406ba50 msgr2=0x7f3de419c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:32.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3de9197700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de406ba50 0x7f3de419c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:32.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3de9197700 1 -- 172.21.15.31:0/2127145533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de41a0d30 con 0x7f3de406b130 2024-04-26T18:13:32.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3de8996700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de406ba50 0x7f3de419c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T18:13:32.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3de9998700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3de410fed0 0x7f3de41a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T18:13:32.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3de9197700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de406b130 0x7f3de419bd20 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3de000b2e0 tx=0x7f3de000b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:32.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3dda7fc700 1 -- 172.21.15.31:0/2127145533 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de0026070 con 0x7f3de406b130 2024-04-26T18:13:32.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3dda7fc700 1 -- 172.21.15.31:0/2127145533 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3de000fbc0 con 0x7f3de406b130 2024-04-26T18:13:32.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2127145533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de41a0fc0 con 0x7f3de406b130 2024-04-26T18:13:32.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3dda7fc700 1 -- 172.21.15.31:0/2127145533 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de0017650 con 0x7f3de406b130 2024-04-26T18:13:32.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.724+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2127145533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de41a88a0 con 0x7f3de406b130 2024-04-26T18:13:32.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.725+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2127145533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de406a110 con 0x7f3de406b130 2024-04-26T18:13:32.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.726+0000 7f3dda7fc700 1 -- 172.21.15.31:0/2127145533 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3de0029070 con 0x7f3de406b130 2024-04-26T18:13:32.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.727+0000 7f3dda7fc700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3dd00a3db0 0x7f3dd00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:32.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.727+0000 7f3de8996700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3dd00a3db0 0x7f3dd00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:32.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.727+0000 7f3dda7fc700 1 -- 172.21.15.31:0/2127145533 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8656+0+0 (secure 0 0 0) 0x7f3de00d17d0 con 0x7f3de406b130 2024-04-26T18:13:32.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.728+0000 7f3de8996700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3dd00a3db0 0x7f3dd00a6270 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4005fd0 tx=0x7f3dd4005e20 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:32.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.732+0000 7f3dda7fc700 1 -- 172.21.15.31:0/2127145533 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3de005dff0 con 0x7f3de406b130 2024-04-26T18:13:32.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.988+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2127145533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3de402e490 con 0x7f3de406b130 2024-04-26T18:13:32.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.991+0000 7f3dda7fc700 1 -- 172.21.15.31:0/2127145533 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v57) ==== 161+0+0 (secure 0 0 0) 0x7f3de0013070 con 0x7f3de406b130 2024-04-26T18:13:32.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.994+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2127145533 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3dd00a3db0 msgr2=0x7f3dd00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:32.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.994+0000 7f3deb3fb700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3dd00a3db0 0x7f3dd00a6270 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4005fd0 tx=0x7f3dd4005e20 comp rx=0 tx=0).stop 2024-04-26T18:13:32.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.994+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2127145533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de406b130 msgr2=0x7f3de419bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:32.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.994+0000 7f3deb3fb700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de406b130 0x7f3de419bd20 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3de000b2e0 tx=0x7f3de000b6c0 comp rx=0 tx=0).stop 2024-04-26T18:13:32.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.994+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2127145533 shutdown_connections 2024-04-26T18:13:32.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.994+0000 7f3deb3fb700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f3dd00a3db0 0x7f3dd00a6270 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:32.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.994+0000 7f3deb3fb700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3de410fed0 0x7f3de41a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:32.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.994+0000 7f3deb3fb700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de406b130 0x7f3de419bd20 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:32.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.994+0000 7f3deb3fb700 1 --2- 172.21.15.31:0/2127145533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3de406ba50 0x7f3de419c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:32.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.994+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2127145533 >> 172.21.15.31:0/2127145533 conn(0x7f3de40792d0 msgr2=0x7f3de4111420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:32.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.994+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2127145533 shutdown_connections 2024-04-26T18:13:32.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:32.994+0000 7f3deb3fb700 1 -- 172.21.15.31:0/2127145533 wait complete. 2024-04-26T18:13:32.996 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T18:13:33.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[68799]: ignoring --setuser ceph since I am not root 2024-04-26T18:13:33.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[68799]: ignoring --setgroup ceph since I am not root 2024-04-26T18:13:33.267 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-mds[68803]: -- 172.21.15.186:0/2017942422 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ba2f34ba40 con 0x55ba2f31ec00 2024-04-26T18:13:33.268 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[68799]: starting mds.c at 2024-04-26T18:13:33.268 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-f[63067]: ignoring --setuser ceph since I am not root 2024-04-26T18:13:33.268 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-f[63067]: ignoring --setgroup ceph since I am not root 2024-04-26T18:13:33.268 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-mds[63071]: -- 172.21.15.186:0/3017767766 <== mon.2 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5616273d3a40 con 0x5616273a6800 2024-04-26T18:13:33.268 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-f[63067]: starting mds.f at 2024-04-26T18:13:33.268 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-i[63686]: ignoring --setuser ceph since I am not root 2024-04-26T18:13:33.269 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-i[63686]: ignoring --setgroup ceph since I am not root 2024-04-26T18:13:33.269 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-mds[63690]: -- 172.21.15.186:0/3726134700 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559066b01a40 con 0x559066ad4800 2024-04-26T18:13:33.269 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-i[63686]: starting mds.i at 2024-04-26T18:13:33.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:32 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a[29160]: 2024-04-26T18:13:32.989+0000 7f48be5fb700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T18:13:33.385 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[67844]: ignoring --setuser ceph since I am not root 2024-04-26T18:13:33.386 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[67844]: ignoring --setgroup ceph since I am not root 2024-04-26T18:13:33.386 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-mds[67848]: -- 172.21.15.97:0/2886347443 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5652adf55a40 con 0x5652adf29000 2024-04-26T18:13:33.386 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[67844]: starting mds.b at 2024-04-26T18:13:33.386 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-e[62438]: ignoring --setuser ceph since I am not root 2024-04-26T18:13:33.386 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-e[62438]: ignoring --setgroup ceph since I am not root 2024-04-26T18:13:33.386 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-mds[62442]: -- 172.21.15.97:0/945748684 <== mon.2 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b82b24da40 con 0x55b82b221000 2024-04-26T18:13:33.386 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-e[62438]: starting mds.e at 2024-04-26T18:13:33.387 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-h[62756]: ignoring --setuser ceph since I am not root 2024-04-26T18:13:33.387 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-h[62756]: ignoring --setgroup ceph since I am not root 2024-04-26T18:13:33.387 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-mds[62760]: -- 172.21.15.97:0/2183397259 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56050377fa40 con 0x560503752800 2024-04-26T18:13:33.387 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-h[62756]: starting mds.h at 2024-04-26T18:13:33.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:33 smithi031 ceph-mon[29164]: pgmap v536: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 84 KiB/s wr, 12 op/s 2024-04-26T18:13:33.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:33 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2127145533' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T18:13:33.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:33 smithi031 ceph-mon[29164]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T18:13:33.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:33 smithi031 ceph-mon[29164]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T18:13:33.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:33 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/2127145533' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T18:13:33.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:33 smithi031 ceph-mon[29164]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T18:13:33.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:33 smithi031 ceph-mon[29164]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T18:13:33.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T18:13:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-mon[32797]: pgmap v536: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 84 KiB/s wr, 12 op/s 2024-04-26T18:13:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2127145533' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T18:13:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-mon[32797]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T18:13:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-mon[32797]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T18:13:33.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/2127145533' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T18:13:33.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-mon[32797]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T18:13:33.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:33 smithi186 ceph-mon[32797]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T18:13:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-mon[32419]: pgmap v536: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 84 KiB/s wr, 12 op/s 2024-04-26T18:13:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2127145533' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T18:13:33.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-mon[32419]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T18:13:33.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-mon[32419]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T18:13:33.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/2127145533' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T18:13:33.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-mon[32419]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T18:13:33.885 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:33 smithi097 ceph-mon[32419]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T18:13:33.901 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:13:35.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:34 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.186:6834/1314809480,v1:172.21.15.186:6835/1314809480] up:boot 2024-04-26T18:13:35.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:34 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.97:6834/3984090626,v1:172.21.15.97:6835/3984090626] up:boot 2024-04-26T18:13:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:34 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.186:6837/2628380261,v1:172.21.15.186:6839/2628380261] up:boot 2024-04-26T18:13:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:34 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.186:6836/1823167201,v1:172.21.15.186:6838/1823167201] up:boot 2024-04-26T18:13:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:34 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.97:6836/3389025609,v1:172.21.15.97:6837/3389025609] up:boot 2024-04-26T18:13:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:34 smithi186 ceph-mon[32797]: mds.? [v2:172.21.15.97:6838/1480513679,v1:172.21.15.97:6839/1480513679] up:boot 2024-04-26T18:13:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:34 smithi186 ceph-mon[32797]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-26T18:13:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T18:13:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T18:13:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T18:13:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T18:13:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T18:13:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:34 smithi186 ceph-mon[32797]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T18:13:35.268 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:34 smithi186 ceph-mon[32797]: pgmap v538: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 75 KiB/s wr, 10 op/s 2024-04-26T18:13:35.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:34 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.186:6834/1314809480,v1:172.21.15.186:6835/1314809480] up:boot 2024-04-26T18:13:35.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:34 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.97:6834/3984090626,v1:172.21.15.97:6835/3984090626] up:boot 2024-04-26T18:13:35.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:34 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.186:6837/2628380261,v1:172.21.15.186:6839/2628380261] up:boot 2024-04-26T18:13:35.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:34 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.186:6836/1823167201,v1:172.21.15.186:6838/1823167201] up:boot 2024-04-26T18:13:35.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:34 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.97:6836/3389025609,v1:172.21.15.97:6837/3389025609] up:boot 2024-04-26T18:13:35.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:34 smithi031 ceph-mon[29164]: mds.? [v2:172.21.15.97:6838/1480513679,v1:172.21.15.97:6839/1480513679] up:boot 2024-04-26T18:13:35.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:34 smithi031 ceph-mon[29164]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-26T18:13:35.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T18:13:35.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T18:13:35.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T18:13:35.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T18:13:35.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T18:13:35.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:34 smithi031 ceph-mon[29164]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T18:13:35.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:34 smithi031 ceph-mon[29164]: pgmap v538: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 75 KiB/s wr, 10 op/s 2024-04-26T18:13:35.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:34 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.186:6834/1314809480,v1:172.21.15.186:6835/1314809480] up:boot 2024-04-26T18:13:35.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:34 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.97:6834/3984090626,v1:172.21.15.97:6835/3984090626] up:boot 2024-04-26T18:13:35.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:34 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.186:6837/2628380261,v1:172.21.15.186:6839/2628380261] up:boot 2024-04-26T18:13:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:34 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.186:6836/1823167201,v1:172.21.15.186:6838/1823167201] up:boot 2024-04-26T18:13:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:34 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.97:6836/3389025609,v1:172.21.15.97:6837/3389025609] up:boot 2024-04-26T18:13:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:34 smithi097 ceph-mon[32419]: mds.? [v2:172.21.15.97:6838/1480513679,v1:172.21.15.97:6839/1480513679] up:boot 2024-04-26T18:13:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:34 smithi097 ceph-mon[32419]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-26T18:13:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T18:13:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T18:13:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T18:13:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T18:13:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T18:13:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:34 smithi097 ceph-mon[32419]: from='mgr.24826 172.21.15.31:0/671868214' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T18:13:35.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:34 smithi097 ceph-mon[32419]: pgmap v538: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 75 KiB/s wr, 10 op/s 2024-04-26T18:13:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.607+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/618541911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e41094c0 msgr2=0x7fe9e41098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.607+0000 7fe9eaf3b700 1 --2- 172.21.15.31:0/618541911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e41094c0 0x7fe9e41098a0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009990 tx=0x7fe9d8009ca0 comp rx=0 tx=0).stop 2024-04-26T18:13:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.607+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/618541911 shutdown_connections 2024-04-26T18:13:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.607+0000 7fe9eaf3b700 1 --2- 172.21.15.31:0/618541911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9e4103e30 0x7fe9e4110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.607+0000 7fe9eaf3b700 1 --2- 172.21.15.31:0/618541911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e41094c0 0x7fe9e41098a0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.607+0000 7fe9eaf3b700 1 --2- 172.21.15.31:0/618541911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e4103470 0x7fe9e41038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.607+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/618541911 >> 172.21.15.31:0/618541911 conn(0x7fe9e407a580 msgr2=0x7fe9e407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.607+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/618541911 shutdown_connections 2024-04-26T18:13:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.607+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/618541911 wait complete. 2024-04-26T18:13:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9eaf3b700 1 Processor -- start 2024-04-26T18:13:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9eaf3b700 1 -- start start 2024-04-26T18:13:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9eaf3b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9e4103470 0x7fe9e419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9eaf3b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e4103e30 0x7fe9e419fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9eaf3b700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e41094c0 0x7fe9e4199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9eaf3b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e41135f0 con 0x7fe9e4103e30 2024-04-26T18:13:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9eaf3b700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e4113470 con 0x7fe9e41094c0 2024-04-26T18:13:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9eaf3b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e4113770 con 0x7fe9e4103470 2024-04-26T18:13:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9e3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e4103e30 0x7fe9e419fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9e94d8700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e41094c0 0x7fe9e4199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9e3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e4103e30 0x7fe9e419fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48672/0 (socket says 172.21.15.31:48672) 2024-04-26T18:13:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9e3fff700 1 -- 172.21.15.31:0/3046537634 learned_addr learned my addr 172.21.15.31:0/3046537634 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:13:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9e8cd7700 1 --2- 172.21.15.31:0/3046537634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9e4103470 0x7fe9e419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9e94d8700 1 -- 172.21.15.31:0/3046537634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9e4103470 msgr2=0x7fe9e419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9e94d8700 1 --2- 172.21.15.31:0/3046537634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9e4103470 0x7fe9e419f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9e94d8700 1 -- 172.21.15.31:0/3046537634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e4103e30 msgr2=0x7fe9e419fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9e94d8700 1 --2- 172.21.15.31:0/3046537634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e4103e30 0x7fe9e419fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9e94d8700 1 -- 172.21.15.31:0/3046537634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e419a0d0 con 0x7fe9e41094c0 2024-04-26T18:13:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.608+0000 7fe9e3fff700 1 --2- 172.21.15.31:0/3046537634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e4103e30 0x7fe9e419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T18:13:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.609+0000 7fe9e94d8700 1 --2- 172.21.15.31:0/3046537634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e41094c0 0x7fe9e4199970 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc007f10 tx=0x7fe9dc00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.609+0000 7fe9e1ffb700 1 -- 172.21.15.31:0/3046537634 <== mon.2 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc004db0 con 0x7fe9e41094c0 2024-04-26T18:13:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.609+0000 7fe9e1ffb700 1 -- 172.21.15.31:0/3046537634 <== mon.2 v2:172.21.15.97:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9dc009e20 con 0x7fe9e41094c0 2024-04-26T18:13:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.609+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/3046537634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e419a3c0 con 0x7fe9e41094c0 2024-04-26T18:13:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.609+0000 7fe9e1ffb700 1 -- 172.21.15.31:0/3046537634 <== mon.2 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc015400 con 0x7fe9e41094c0 2024-04-26T18:13:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.609+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/3046537634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e4075650 con 0x7fe9e41094c0 2024-04-26T18:13:35.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.610+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/3046537634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9e410b930 con 0x7fe9e41094c0 2024-04-26T18:13:35.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.611+0000 7fe9e1ffb700 1 -- 172.21.15.31:0/3046537634 <== mon.2 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe9dc005260 con 0x7fe9e41094c0 2024-04-26T18:13:35.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.612+0000 7fe9e1ffb700 1 --2- 172.21.15.31:0/3046537634 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe9d40a3e30 0x7fe9d40a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:35.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.612+0000 7fe9e8cd7700 1 --2- 172.21.15.31:0/3046537634 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe9d40a3e30 0x7fe9d40a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:35.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.612+0000 7fe9e1ffb700 1 -- 172.21.15.31:0/3046537634 <== mon.2 v2:172.21.15.97:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8870+0+0 (secure 0 0 0) 0x7fe9dc0d01d0 con 0x7fe9e41094c0 2024-04-26T18:13:35.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.612+0000 7fe9e8cd7700 1 --2- 172.21.15.31:0/3046537634 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe9d40a3e30 0x7fe9d40a62f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800b2e0 tx=0x7fe9d801a040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:35.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.617+0000 7fe9e1ffb700 1 -- 172.21.15.31:0/3046537634 <== mon.2 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe9dc010070 con 0x7fe9e41094c0 2024-04-26T18:13:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:35.874+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/3046537634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fe9e419aea0 con 0x7fe9e41094c0 2024-04-26T18:13:36.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:36.010+0000 7fe9e1ffb700 1 -- 172.21.15.31:0/3046537634 <== mon.2 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v59) ==== 104+0+0 (secure 0 0 0) 0x7fe9dc09c710 con 0x7fe9e41094c0 2024-04-26T18:13:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:36.013+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/3046537634 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe9d40a3e30 msgr2=0x7fe9d40a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:36.013+0000 7fe9eaf3b700 1 --2- 172.21.15.31:0/3046537634 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe9d40a3e30 0x7fe9d40a62f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800b2e0 tx=0x7fe9d801a040 comp rx=0 tx=0).stop 2024-04-26T18:13:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:36.013+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/3046537634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e41094c0 msgr2=0x7fe9e4199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:36.013+0000 7fe9eaf3b700 1 --2- 172.21.15.31:0/3046537634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e41094c0 0x7fe9e4199970 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc007f10 tx=0x7fe9dc00d3b0 comp rx=0 tx=0).stop 2024-04-26T18:13:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:36.013+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/3046537634 shutdown_connections 2024-04-26T18:13:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:36.013+0000 7fe9eaf3b700 1 --2- 172.21.15.31:0/3046537634 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7fe9d40a3e30 0x7fe9d40a62f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:36.013+0000 7fe9eaf3b700 1 --2- 172.21.15.31:0/3046537634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e41094c0 0x7fe9e4199970 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:36.013+0000 7fe9eaf3b700 1 --2- 172.21.15.31:0/3046537634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9e4103470 0x7fe9e419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:36.013+0000 7fe9eaf3b700 1 --2- 172.21.15.31:0/3046537634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e4103e30 0x7fe9e419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:36.013+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/3046537634 >> 172.21.15.31:0/3046537634 conn(0x7fe9e407a580 msgr2=0x7fe9e41021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:36.013+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/3046537634 shutdown_connections 2024-04-26T18:13:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:36.013+0000 7fe9eaf3b700 1 -- 172.21.15.31:0/3046537634 wait complete. 2024-04-26T18:13:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:36 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/3046537634' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T18:13:36.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:36 smithi186 ceph-mon[32797]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T18:13:36.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:36 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/3046537634' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T18:13:36.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:36 smithi031 ceph-mon[29164]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T18:13:36.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:36 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/3046537634' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T18:13:36.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:36 smithi097 ceph-mon[32419]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T18:13:36.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T18:13:36.924 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:13:37.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:37 smithi186 ceph-mon[32797]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T18:13:37.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:37 smithi186 ceph-mon[32797]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T18:13:37.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:37 smithi186 ceph-mon[32797]: Cluster is now healthy 2024-04-26T18:13:37.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:37 smithi186 ceph-mon[32797]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T18:13:37.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:37 smithi186 ceph-mon[32797]: fsmap 12 up:standby 2024-04-26T18:13:37.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:37 smithi186 ceph-mon[32797]: pgmap v539: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 75 KiB/s wr, 10 op/s 2024-04-26T18:13:37.280 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:37 smithi031 ceph-mon[29164]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T18:13:37.280 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:37 smithi031 ceph-mon[29164]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T18:13:37.280 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:37 smithi031 ceph-mon[29164]: Cluster is now healthy 2024-04-26T18:13:37.280 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:37 smithi031 ceph-mon[29164]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T18:13:37.280 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:37 smithi031 ceph-mon[29164]: fsmap 12 up:standby 2024-04-26T18:13:37.280 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:37 smithi031 ceph-mon[29164]: pgmap v539: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 75 KiB/s wr, 10 op/s 2024-04-26T18:13:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:37 smithi097 ceph-mon[32419]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T18:13:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:37 smithi097 ceph-mon[32419]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T18:13:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:37 smithi097 ceph-mon[32419]: Cluster is now healthy 2024-04-26T18:13:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:37 smithi097 ceph-mon[32419]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T18:13:37.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:37 smithi097 ceph-mon[32419]: fsmap 12 up:standby 2024-04-26T18:13:37.385 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:37 smithi097 ceph-mon[32419]: pgmap v539: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 75 KiB/s wr, 10 op/s 2024-04-26T18:13:38.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.615+0000 7f917a4b5700 1 -- 172.21.15.31:0/4158346469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91741034a0 msgr2=0x7f9174103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.615+0000 7f917a4b5700 1 --2- 172.21.15.31:0/4158346469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91741034a0 0x7f9174103920 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f915c009990 tx=0x7f915c009ca0 comp rx=0 tx=0).stop 2024-04-26T18:13:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.615+0000 7f917a4b5700 1 -- 172.21.15.31:0/4158346469 shutdown_connections 2024-04-26T18:13:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.615+0000 7f917a4b5700 1 --2- 172.21.15.31:0/4158346469 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91741094e0 0x7f91741098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.615+0000 7f917a4b5700 1 --2- 172.21.15.31:0/4158346469 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9174103e60 0x7f91741102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.615+0000 7f917a4b5700 1 --2- 172.21.15.31:0/4158346469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91741034a0 0x7f9174103920 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.615+0000 7f917a4b5700 1 -- 172.21.15.31:0/4158346469 >> 172.21.15.31:0/4158346469 conn(0x7f917407a590 msgr2=0x7f917407a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.615+0000 7f917a4b5700 1 -- 172.21.15.31:0/4158346469 shutdown_connections 2024-04-26T18:13:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.615+0000 7f917a4b5700 1 -- 172.21.15.31:0/4158346469 wait complete. 2024-04-26T18:13:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.616+0000 7f917a4b5700 1 Processor -- start 2024-04-26T18:13:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.616+0000 7f917a4b5700 1 -- start start 2024-04-26T18:13:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.616+0000 7f917a4b5700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91741034a0 0x7f917419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.616+0000 7f917a4b5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9174103e60 0x7f917419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.616+0000 7f917a4b5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91741094e0 0x7f91741998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.616+0000 7f917a4b5700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9174113660 con 0x7f9174103e60 2024-04-26T18:13:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.616+0000 7f917a4b5700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f91741134e0 con 0x7f91741034a0 2024-04-26T18:13:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.616+0000 7f917a4b5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f91741137e0 con 0x7f91741094e0 2024-04-26T18:13:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.616+0000 7f91737fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9174103e60 0x7f917419fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.616+0000 7f91737fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9174103e60 0x7f917419fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48694/0 (socket says 172.21.15.31:48694) 2024-04-26T18:13:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.616+0000 7f91737fe700 1 -- 172.21.15.31:0/945021412 learned_addr learned my addr 172.21.15.31:0/945021412 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:13:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.616+0000 7f9178a52700 1 --2- 172.21.15.31:0/945021412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91741094e0 0x7f91741998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.616+0000 7f9173fff700 1 --2- 172.21.15.31:0/945021412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91741034a0 0x7f917419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.617+0000 7f91737fe700 1 -- 172.21.15.31:0/945021412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91741094e0 msgr2=0x7f91741998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.617+0000 7f91737fe700 1 --2- 172.21.15.31:0/945021412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91741094e0 0x7f91741998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.617+0000 7f91737fe700 1 -- 172.21.15.31:0/945021412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91741034a0 msgr2=0x7f917419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.617+0000 7f91737fe700 1 --2- 172.21.15.31:0/945021412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91741034a0 0x7f917419f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.617+0000 7f91737fe700 1 -- 172.21.15.31:0/945021412 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917419a040 con 0x7f9174103e60 2024-04-26T18:13:38.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.617+0000 7f91737fe700 1 --2- 172.21.15.31:0/945021412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9174103e60 0x7f917419fc60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f915c005910 tx=0x7f915c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:38.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.617+0000 7f91717fa700 1 -- 172.21.15.31:0/945021412 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915c027070 con 0x7f9174103e60 2024-04-26T18:13:38.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.617+0000 7f91717fa700 1 -- 172.21.15.31:0/945021412 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f915c004020 con 0x7f9174103e60 2024-04-26T18:13:38.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.617+0000 7f917a4b5700 1 -- 172.21.15.31:0/945021412 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f917419a2d0 con 0x7f9174103e60 2024-04-26T18:13:38.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.617+0000 7f91717fa700 1 -- 172.21.15.31:0/945021412 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915c00f460 con 0x7f9174103e60 2024-04-26T18:13:38.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.617+0000 7f917a4b5700 1 -- 172.21.15.31:0/945021412 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91741a65f0 con 0x7f9174103e60 2024-04-26T18:13:38.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.618+0000 7f917a4b5700 1 -- 172.21.15.31:0/945021412 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f917410b950 con 0x7f9174103e60 2024-04-26T18:13:38.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.619+0000 7f91717fa700 1 -- 172.21.15.31:0/945021412 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f915c02a070 con 0x7f9174103e60 2024-04-26T18:13:38.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.620+0000 7f91717fa700 1 --2- 172.21.15.31:0/945021412 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f91600a3db0 0x7f91600a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:38.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.620+0000 7f91717fa700 1 -- 172.21.15.31:0/945021412 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8870+0+0 (secure 0 0 0) 0x7f915c0d1a90 con 0x7f9174103e60 2024-04-26T18:13:38.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.620+0000 7f9173fff700 1 --2- 172.21.15.31:0/945021412 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f91600a3db0 0x7f91600a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:38.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.620+0000 7f9173fff700 1 --2- 172.21.15.31:0/945021412 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f91600a3db0 0x7f91600a6270 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9164005fd0 tx=0x7f916400af00 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:38.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.624+0000 7f91717fa700 1 -- 172.21.15.31:0/945021412 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f915c05eff0 con 0x7f9174103e60 2024-04-26T18:13:38.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:38.834+0000 7f917a4b5700 1 -- 172.21.15.31:0/945021412 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9174104b80 con 0x7f9174103e60 2024-04-26T18:13:39.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:39.157+0000 7f91717fa700 1 -- 172.21.15.31:0/945021412 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v115) ==== 181+0+0 (secure 0 0 0) 0x7f915c013070 con 0x7f9174103e60 2024-04-26T18:13:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:39.159+0000 7f917a4b5700 1 -- 172.21.15.31:0/945021412 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f91600a3db0 msgr2=0x7f91600a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:39.159+0000 7f917a4b5700 1 --2- 172.21.15.31:0/945021412 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f91600a3db0 0x7f91600a6270 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9164005fd0 tx=0x7f916400af00 comp rx=0 tx=0).stop 2024-04-26T18:13:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:39.159+0000 7f917a4b5700 1 -- 172.21.15.31:0/945021412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9174103e60 msgr2=0x7f917419fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:39.159+0000 7f917a4b5700 1 --2- 172.21.15.31:0/945021412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9174103e60 0x7f917419fc60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f915c005910 tx=0x7f915c0047b0 comp rx=0 tx=0).stop 2024-04-26T18:13:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:39.160+0000 7f917a4b5700 1 -- 172.21.15.31:0/945021412 shutdown_connections 2024-04-26T18:13:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:39.160+0000 7f917a4b5700 1 --2- 172.21.15.31:0/945021412 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f91600a3db0 0x7f91600a6270 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:39.160+0000 7f917a4b5700 1 --2- 172.21.15.31:0/945021412 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f91741034a0 0x7f917419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:39.160+0000 7f917a4b5700 1 --2- 172.21.15.31:0/945021412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91741094e0 0x7f91741998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:39.160+0000 7f917a4b5700 1 --2- 172.21.15.31:0/945021412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9174103e60 0x7f917419fc60 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:39.160+0000 7f917a4b5700 1 -- 172.21.15.31:0/945021412 >> 172.21.15.31:0/945021412 conn(0x7f917407a590 msgr2=0x7f9174102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:39.160+0000 7f917a4b5700 1 -- 172.21.15.31:0/945021412 shutdown_connections 2024-04-26T18:13:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:39.160+0000 7f917a4b5700 1 -- 172.21.15.31:0/945021412 wait complete. 2024-04-26T18:13:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-04-26T18:13:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:39 smithi186 ceph-mon[32797]: pgmap v540: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 7 op/s 2024-04-26T18:13:39.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:39 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/945021412' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T18:13:39.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:39 smithi031 ceph-mon[29164]: pgmap v540: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 7 op/s 2024-04-26T18:13:39.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:39 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/945021412' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T18:13:39.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:39 smithi097 ceph-mon[32419]: pgmap v540: 97 pgs: 97 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 7 op/s 2024-04-26T18:13:39.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:39 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/945021412' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T18:13:39.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1 shell --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T18:13:40.159 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T18:13:40.472 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:40 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/945021412' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T18:13:40.472 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:40 smithi031 ceph-mon[29164]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T18:13:40.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:40 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/945021412' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T18:13:40.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:40 smithi186 ceph-mon[32797]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T18:13:40.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:40 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/945021412' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T18:13:40.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:40 smithi097 ceph-mon[32419]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T18:13:41.443 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:41 smithi031 ceph-mon[29164]: pgmap v542: 65 pgs: 65 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T18:13:41.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:41 smithi186 ceph-mon[32797]: pgmap v542: 65 pgs: 65 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T18:13:41.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:41 smithi097 ceph-mon[32419]: pgmap v542: 65 pgs: 65 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T18:13:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.665+0000 7f1e18b19700 1 -- 172.21.15.31:0/2447872898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e14103e30 msgr2=0x7f1e14110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.665+0000 7f1e18b19700 1 --2- 172.21.15.31:0/2447872898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e14103e30 0x7f1e14110290 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c009990 tx=0x7f1e0c009ca0 comp rx=0 tx=0).stop 2024-04-26T18:13:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.665+0000 7f1e18b19700 1 -- 172.21.15.31:0/2447872898 shutdown_connections 2024-04-26T18:13:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.665+0000 7f1e18b19700 1 --2- 172.21.15.31:0/2447872898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e14103e30 0x7f1e14110290 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.665+0000 7f1e18b19700 1 --2- 172.21.15.31:0/2447872898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e141094c0 0x7f1e141098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.665+0000 7f1e18b19700 1 --2- 172.21.15.31:0/2447872898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1e14103470 0x7f1e141038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.665+0000 7f1e18b19700 1 -- 172.21.15.31:0/2447872898 >> 172.21.15.31:0/2447872898 conn(0x7f1e1407a580 msgr2=0x7f1e1407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:41.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.665+0000 7f1e18b19700 1 -- 172.21.15.31:0/2447872898 shutdown_connections 2024-04-26T18:13:41.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.665+0000 7f1e18b19700 1 -- 172.21.15.31:0/2447872898 wait complete. 2024-04-26T18:13:41.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.666+0000 7f1e18b19700 1 Processor -- start 2024-04-26T18:13:41.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.666+0000 7f1e18b19700 1 -- start start 2024-04-26T18:13:41.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.666+0000 7f1e18b19700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e14103470 0x7f1e1419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:41.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.666+0000 7f1e18b19700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e14103e30 0x7f1e1419fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:41.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.666+0000 7f1e18b19700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1e141094c0 0x7f1e141998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:41.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.666+0000 7f1e18b19700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1e141135c0 con 0x7f1e14103e30 2024-04-26T18:13:41.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.666+0000 7f1e18b19700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 -- 0x7f1e14113440 con 0x7f1e141094c0 2024-04-26T18:13:41.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.666+0000 7f1e18b19700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1e14113740 con 0x7f1e14103470 2024-04-26T18:13:41.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.666+0000 7f1e11d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e14103e30 0x7f1e1419fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:41.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.666+0000 7f1e11d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e14103e30 0x7f1e1419fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48710/0 (socket says 172.21.15.31:48710) 2024-04-26T18:13:41.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.666+0000 7f1e11d9b700 1 -- 172.21.15.31:0/1591150372 learned_addr learned my addr 172.21.15.31:0/1591150372 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-26T18:13:41.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.666+0000 7f1e12d9d700 1 --2- 172.21.15.31:0/1591150372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1e141094c0 0x7f1e141998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:41.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.667+0000 7f1e1259c700 1 --2- 172.21.15.31:0/1591150372 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e14103470 0x7f1e1419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:41.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.667+0000 7f1e11d9b700 1 -- 172.21.15.31:0/1591150372 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e14103470 msgr2=0x7f1e1419f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:41.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.667+0000 7f1e11d9b700 1 --2- 172.21.15.31:0/1591150372 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e14103470 0x7f1e1419f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:41.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.667+0000 7f1e11d9b700 1 -- 172.21.15.31:0/1591150372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1e141094c0 msgr2=0x7f1e141998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:41.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.667+0000 7f1e11d9b700 1 --2- 172.21.15.31:0/1591150372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1e141094c0 0x7f1e141998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:41.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.667+0000 7f1e11d9b700 1 -- 172.21.15.31:0/1591150372 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e1419a000 con 0x7f1e14103e30 2024-04-26T18:13:41.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.667+0000 7f1e11d9b700 1 --2- 172.21.15.31:0/1591150372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e14103e30 0x7f1e1419fc20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00d950 tx=0x7f1dfc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:41.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.667+0000 7f1e0b7fe700 1 -- 172.21.15.31:0/1591150372 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc016070 con 0x7f1e14103e30 2024-04-26T18:13:41.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.667+0000 7f1e18b19700 1 -- 172.21.15.31:0/1591150372 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e1419a2f0 con 0x7f1e14103e30 2024-04-26T18:13:41.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.667+0000 7f1e0b7fe700 1 -- 172.21.15.31:0/1591150372 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dfc009df0 con 0x7f1e14103e30 2024-04-26T18:13:41.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.667+0000 7f1e0b7fe700 1 -- 172.21.15.31:0/1591150372 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc00f3c0 con 0x7f1e14103e30 2024-04-26T18:13:41.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.667+0000 7f1e18b19700 1 -- 172.21.15.31:0/1591150372 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e141a65b0 con 0x7f1e14103e30 2024-04-26T18:13:41.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.668+0000 7f1e18b19700 1 -- 172.21.15.31:0/1591150372 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e1406a110 con 0x7f1e14103e30 2024-04-26T18:13:41.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.669+0000 7f1e0b7fe700 1 -- 172.21.15.31:0/1591150372 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1dfc019070 con 0x7f1e14103e30 2024-04-26T18:13:41.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.670+0000 7f1e0b7fe700 1 --2- 172.21.15.31:0/1591150372 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1e000a3e30 0x7f1e000a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T18:13:41.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.670+0000 7f1e1259c700 1 --2- 172.21.15.31:0/1591150372 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1e000a3e30 0x7f1e000a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T18:13:41.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.670+0000 7f1e0b7fe700 1 -- 172.21.15.31:0/1591150372 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8374+0+0 (secure 0 0 0) 0x7f1dfc0d0330 con 0x7f1e14103e30 2024-04-26T18:13:41.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.670+0000 7f1e1259c700 1 --2- 172.21.15.31:0/1591150372 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1e000a3e30 0x7f1e000a62f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1e1419a9e0 tx=0x7f1e04009040 comp rx=0 tx=0).ready entity=mgr.24826 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T18:13:41.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.675+0000 7f1e0b7fe700 1 -- 172.21.15.31:0/1591150372 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1dfc09ca00 con 0x7f1e14103e30 2024-04-26T18:13:41.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:41.882+0000 7f1e18b19700 1 -- 172.21.15.31:0/1591150372 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1e1402e490 con 0x7f1e14103e30 2024-04-26T18:13:42.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:42.180+0000 7f1e0b7fe700 1 -- 172.21.15.31:0/1591150372 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v116) ==== 169+0+0 (secure 0 0 0) 0x7f1dfc09c300 con 0x7f1e14103e30 2024-04-26T18:13:42.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:42.182+0000 7f1e18b19700 1 -- 172.21.15.31:0/1591150372 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1e000a3e30 msgr2=0x7f1e000a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:42.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:42.182+0000 7f1e18b19700 1 --2- 172.21.15.31:0/1591150372 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1e000a3e30 0x7f1e000a62f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1e1419a9e0 tx=0x7f1e04009040 comp rx=0 tx=0).stop 2024-04-26T18:13:42.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:42.182+0000 7f1e18b19700 1 -- 172.21.15.31:0/1591150372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e14103e30 msgr2=0x7f1e1419fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T18:13:42.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:42.182+0000 7f1e18b19700 1 --2- 172.21.15.31:0/1591150372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e14103e30 0x7f1e1419fc20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00d950 tx=0x7f1dfc00dc60 comp rx=0 tx=0).stop 2024-04-26T18:13:42.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:42.183+0000 7f1e18b19700 1 -- 172.21.15.31:0/1591150372 shutdown_connections 2024-04-26T18:13:42.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:42.183+0000 7f1e18b19700 1 --2- 172.21.15.31:0/1591150372 >> [v2:172.21.15.31:6800/3595544994,v1:172.21.15.31:6801/3595544994] conn(0x7f1e000a3e30 0x7f1e000a62f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:42.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:42.183+0000 7f1e18b19700 1 --2- 172.21.15.31:0/1591150372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1e141094c0 0x7f1e141998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:42.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:42.183+0000 7f1e18b19700 1 --2- 172.21.15.31:0/1591150372 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e14103470 0x7f1e1419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:42.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:42.183+0000 7f1e18b19700 1 --2- 172.21.15.31:0/1591150372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e14103e30 0x7f1e1419fc20 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T18:13:42.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:42.183+0000 7f1e18b19700 1 -- 172.21.15.31:0/1591150372 >> 172.21.15.31:0/1591150372 conn(0x7f1e1407a580 msgr2=0x7f1e141020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T18:13:42.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:42.183+0000 7f1e18b19700 1 -- 172.21.15.31:0/1591150372 shutdown_connections 2024-04-26T18:13:42.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-26T18:13:42.183+0000 7f1e18b19700 1 -- 172.21.15.31:0/1591150372 wait complete. 2024-04-26T18:13:42.186 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-04-26T18:13:42.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:42 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1591150372' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T18:13:42.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:42 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1591150372' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T18:13:42.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:42 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1591150372' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T18:13:43.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T18:13:43.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T18:13:43.105 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T18:13:43.134 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-26T18:13:43.134 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T18:13:43.156 DEBUG:teuthology.orchestra.run.smithi097:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T18:13:43.178 DEBUG:teuthology.orchestra.run.smithi186:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T18:13:43.194 INFO:tasks.cephadm:Stopping all daemons... 2024-04-26T18:13:43.194 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-26T18:13:43.195 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.a 2024-04-26T18:13:43.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:43 smithi186 ceph-mon[32797]: pgmap v543: 65 pgs: 65 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T18:13:43.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:43 smithi186 ceph-mon[32797]: from='client.? 172.21.15.31:0/1591150372' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T18:13:43.267 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:43 smithi186 ceph-mon[32797]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T18:13:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:43 smithi097 ceph-mon[32419]: pgmap v543: 65 pgs: 65 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T18:13:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:43 smithi097 ceph-mon[32419]: from='client.? 172.21.15.31:0/1591150372' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T18:13:43.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:43 smithi097 ceph-mon[32419]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T18:13:43.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:43 smithi031 ceph-mon[29164]: pgmap v543: 65 pgs: 65 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T18:13:43.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:43 smithi031 ceph-mon[29164]: from='client.? 172.21.15.31:0/1591150372' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T18:13:43.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:43 smithi031 ceph-mon[29164]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T18:13:43.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:43 smithi031 systemd[1]: Stopping Ceph mon.a for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:13:43.941 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a[29160]: 2024-04-26T18:13:43.685+0000 7f48c3e06700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:13:43.941 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a[29160]: 2024-04-26T18:13:43.685+0000 7f48c3e06700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-26T18:13:44.208 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:43 smithi031 podman[99383]: 2024-04-26 18:13:43.941745552 +0000 UTC m=+0.455437339 container died bd136c7b0b4e18ff5e53e302625de5bf7259e4adcbaf5b1565680493ea4d6b14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T18:13:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:44 smithi031 podman[99383]: 2024-04-26 18:13:44.308026075 +0000 UTC m=+0.821717871 container cleanup bd136c7b0b4e18ff5e53e302625de5bf7259e4adcbaf5b1565680493ea4d6b14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:13:44.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 26 18:13:44 smithi031 bash[99383]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-a 2024-04-26T18:13:44.796 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.a.service' 2024-04-26T18:13:44.864 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:13:44.864 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-26T18:13:44.865 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-26T18:13:44.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.b 2024-04-26T18:13:45.280 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:44 smithi097 systemd[1]: Stopping Ceph mon.b for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:13:45.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:45 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-b[32415]: 2024-04-26T18:13:45.279+0000 7fe1ef0b4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:13:45.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:45 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-b[32415]: 2024-04-26T18:13:45.279+0000 7fe1ef0b4700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-26T18:13:45.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:45 smithi097 podman[70428]: 2024-04-26 18:13:45.381382052 +0000 UTC m=+0.235293142 container died 81484a0a13bd3f3e8d890687339145d0166b978843c2dcdf87904bc30ea60ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-26T18:13:46.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:45 smithi097 podman[70428]: 2024-04-26 18:13:45.763356804 +0000 UTC m=+0.617267905 container cleanup 81484a0a13bd3f3e8d890687339145d0166b978843c2dcdf87904bc30ea60ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T18:13:46.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Apr 26 18:13:45 smithi097 bash[70428]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-b 2024-04-26T18:13:46.151 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.b.service' 2024-04-26T18:13:46.194 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:13:46.195 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-26T18:13:46.195 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-26T18:13:46.195 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.c 2024-04-26T18:13:46.517 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:46 smithi186 systemd[1]: Stopping Ceph mon.c for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:13:46.857 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:46 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-c[32793]: 2024-04-26T18:13:46.565+0000 7f4448c4a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:13:46.857 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:46 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-c[32793]: 2024-04-26T18:13:46.565+0000 7f4448c4a700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-26T18:13:46.858 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:46 smithi186 podman[71383]: 2024-04-26 18:13:46.616618483 +0000 UTC m=+0.234895419 container died 7c71567ce1a458c23c00f78464871d4818a88b7e82e63695d0d889b263593fbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-c, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-26T18:13:47.169 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:46 smithi186 podman[71383]: 2024-04-26 18:13:46.949122073 +0000 UTC m=+0.567398998 container cleanup 7c71567ce1a458c23c00f78464871d4818a88b7e82e63695d0d889b263593fbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T18:13:47.169 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:46 smithi186 bash[71383]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-c 2024-04-26T18:13:47.415 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.c.service' 2024-04-26T18:13:47.436 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:47 smithi186 podman[71397]: 2024-04-26 18:13:47.169979546 +0000 UTC m=+0.550269354 container remove 7c71567ce1a458c23c00f78464871d4818a88b7e82e63695d0d889b263593fbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-26T18:13:47.436 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:47 smithi186 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.c.service: Deactivated successfully. 2024-04-26T18:13:47.436 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:47 smithi186 systemd[1]: Stopped Ceph mon.c for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T18:13:47.436 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 18:13:47 smithi186 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mon.c.service: Consumed 15.272s CPU time. 2024-04-26T18:13:47.461 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:13:47.462 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-26T18:13:47.462 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-26T18:13:47.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mgr.x 2024-04-26T18:13:47.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 18:13:47 smithi031 systemd[1]: Stopping Ceph mgr.x for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:13:49.060 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 18:13:48 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T18:13:48.762+0000 7f51914a8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:13:49.060 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 18:13:48 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-26T18:13:48.762+0000 7f51914a8700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-26T18:13:49.060 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 18:13:48 smithi031 podman[99500]: 2024-04-26 18:13:48.80291241 +0000 UTC m=+1.147913560 container died 7f2efebd461ef656468903d0ae1c18ccf025bd0917ef162313d42efa2a7fd1b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-26T18:13:49.499 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 18:13:49 smithi031 podman[99500]: 2024-04-26 18:13:49.18086524 +0000 UTC m=+1.525866429 container cleanup 7f2efebd461ef656468903d0ae1c18ccf025bd0917ef162313d42efa2a7fd1b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc94a2a924f4a25e7c0317e85c91b85bf7cac0b1, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-26T18:13:49.499 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 26 18:13:49 smithi031 bash[99500]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-x 2024-04-26T18:13:49.679 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-26T18:13:49.716 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:13:49.716 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-26T18:13:49.716 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-26T18:13:49.716 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mgr.y 2024-04-26T18:13:50.134 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 18:13:49 smithi097 systemd[1]: Stopping Ceph mgr.y for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:13:51.384 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 18:13:51 smithi097 podman[70553]: 2024-04-26 18:13:51.076534082 +0000 UTC m=+1.167965595 container died d1c415b7a384d01efce34f8308d3cab5aa73f55051b70d93e40aabdba2e1cdba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-26T18:13:51.884 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 18:13:51 smithi097 podman[70553]: 2024-04-26 18:13:51.735994979 +0000 UTC m=+1.827426499 container cleanup d1c415b7a384d01efce34f8308d3cab5aa73f55051b70d93e40aabdba2e1cdba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-26T18:13:51.884 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 26 18:13:51 smithi097 bash[70553]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-y 2024-04-26T18:13:52.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-26T18:13:52.205 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:13:52.205 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-26T18:13:52.205 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-26T18:13:52.206 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mgr.z 2024-04-26T18:13:52.517 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 18:13:52 smithi186 systemd[1]: Stopping Ceph mgr.z for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:13:52.966 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 18:13:52 smithi186 podman[71505]: 2024-04-26 18:13:52.679947002 +0000 UTC m=+0.282169225 container died e4e8d064514ae8881acad3cd8fd0dc94004d35d08ba9d4836e5e08fa35c2c0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-26T18:13:53.267 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 18:13:53 smithi186 podman[71505]: 2024-04-26 18:13:53.06584767 +0000 UTC m=+0.668069904 container cleanup e4e8d064514ae8881acad3cd8fd0dc94004d35d08ba9d4836e5e08fa35c2c0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:13:53.268 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 18:13:53 smithi186 bash[71505]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mgr-z 2024-04-26T18:13:53.643 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-26T18:13:53.677 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:13:53.677 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-26T18:13:53.678 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-26T18:13:53.678 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.0 2024-04-26T18:13:54.016 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 18:13:53 smithi031 systemd[1]: Stopping Ceph osd.0 for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:13:55.310 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 18:13:54 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-0[41697]: 2024-04-26T18:13:54.878+0000 7f5e5bf02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:13:55.310 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 18:13:54 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-0[41697]: 2024-04-26T18:13:54.878+0000 7f5e5bf02700 -1 osd.0 116 *** Got signal Terminated *** 2024-04-26T18:13:55.310 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 18:13:54 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-0[41697]: 2024-04-26T18:13:54.878+0000 7f5e5bf02700 -1 osd.0 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T18:14:00.704 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 18:14:00 smithi031 podman[99621]: 2024-04-26 18:14:00.432902049 +0000 UTC m=+6.559694909 container died facefcfadbae4f50e0977e819e0dcf7e4ad5e1a72f654f90b2829ce11fd660f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-04-26T18:14:00.978 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 18:14:00 smithi031 podman[99621]: 2024-04-26 18:14:00.769396665 +0000 UTC m=+6.896189532 container cleanup facefcfadbae4f50e0977e819e0dcf7e4ad5e1a72f654f90b2829ce11fd660f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux ) 2024-04-26T18:14:00.979 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 18:14:00 smithi031 bash[99621]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-0 2024-04-26T18:14:01.310 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 18:14:00 smithi031 podman[99640]: 2024-04-26 18:14:00.979319662 +0000 UTC m=+0.547031333 container remove facefcfadbae4f50e0977e819e0dcf7e4ad5e1a72f654f90b2829ce11fd660f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T18:14:01.691 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 18:14:01 smithi031 podman[99735]: 2024-04-26 18:14:01.320035778 +0000 UTC m=+0.018917259 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T18:14:01.691 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 18:14:01 smithi031 podman[99735]: 2024-04-26 18:14:01.447228162 +0000 UTC m=+0.146109636 container create 050d46dff90c33d75b88f754e39d4fb52ad3fdd2463111707b824947eef97b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-26T18:14:01.691 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 18:14:01 smithi031 podman[99735]: 2024-04-26 18:14:01.691480868 +0000 UTC m=+0.390362337 container init 050d46dff90c33d75b88f754e39d4fb52ad3fdd2463111707b824947eef97b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-26T18:14:02.012 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 18:14:01 smithi031 podman[99735]: 2024-04-26 18:14:01.695017977 +0000 UTC m=+0.393899445 container start 050d46dff90c33d75b88f754e39d4fb52ad3fdd2463111707b824947eef97b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T18:14:02.012 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 18:14:01 smithi031 podman[99735]: 2024-04-26 18:14:01.791579284 +0000 UTC m=+0.490460758 container attach 050d46dff90c33d75b88f754e39d4fb52ad3fdd2463111707b824947eef97b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:14:02.303 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 26 18:14:02 smithi031 podman[99735]: 2024-04-26 18:14:02.012993583 +0000 UTC m=+0.711875050 container died 050d46dff90c33d75b88f754e39d4fb52ad3fdd2463111707b824947eef97b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T18:14:02.613 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.0.service' 2024-04-26T18:14:02.655 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:14:02.655 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-26T18:14:02.655 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-26T18:14:02.655 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.1 2024-04-26T18:14:02.958 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 18:14:02 smithi097 systemd[1]: Stopping Ceph osd.1 for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:14:03.384 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 18:14:02 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-1[37019]: 2024-04-26T18:14:02.957+0000 7f11d661a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:14:03.384 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 18:14:02 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-1[37019]: 2024-04-26T18:14:02.957+0000 7f11d661a700 -1 osd.1 116 *** Got signal Terminated *** 2024-04-26T18:14:03.384 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 18:14:02 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-1[37019]: 2024-04-26T18:14:02.957+0000 7f11d661a700 -1 osd.1 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T18:14:08.884 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 18:14:08 smithi097 podman[70677]: 2024-04-26 18:14:08.592183387 +0000 UTC m=+5.750562038 container died b6f8eea343edbd0db0d83eb0e1c80f4b564ba3b2736afb05a35742437789dbf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS) 2024-04-26T18:14:09.155 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 18:14:08 smithi097 podman[70677]: 2024-04-26 18:14:08.984098083 +0000 UTC m=+6.142476729 container cleanup b6f8eea343edbd0db0d83eb0e1c80f4b564ba3b2736afb05a35742437789dbf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:14:09.155 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 18:14:08 smithi097 bash[70677]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-1 2024-04-26T18:14:09.574 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 18:14:09 smithi097 podman[70695]: 2024-04-26 18:14:09.244426943 +0000 UTC m=+0.650894826 container remove b6f8eea343edbd0db0d83eb0e1c80f4b564ba3b2736afb05a35742437789dbf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T18:14:09.830 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 18:14:09 smithi097 podman[70791]: 2024-04-26 18:14:09.478255853 +0000 UTC m=+0.018172124 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T18:14:09.831 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 18:14:09 smithi097 podman[70791]: 2024-04-26 18:14:09.595433733 +0000 UTC m=+0.135349995 container create 2449d3d7a1ae2c0cb9305ec5c18a60e93f60428127080f903c3a6b943e919d87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-04-26T18:14:09.831 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 18:14:09 smithi097 podman[70791]: 2024-04-26 18:14:09.831249065 +0000 UTC m=+0.371165327 container init 2449d3d7a1ae2c0cb9305ec5c18a60e93f60428127080f903c3a6b943e919d87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-26T18:14:10.135 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 18:14:09 smithi097 podman[70791]: 2024-04-26 18:14:09.834789899 +0000 UTC m=+0.374706167 container start 2449d3d7a1ae2c0cb9305ec5c18a60e93f60428127080f903c3a6b943e919d87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-26T18:14:10.135 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 18:14:09 smithi097 podman[70791]: 2024-04-26 18:14:09.923124374 +0000 UTC m=+0.463040636 container attach 2449d3d7a1ae2c0cb9305ec5c18a60e93f60428127080f903c3a6b943e919d87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T18:14:10.462 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 26 18:14:10 smithi097 podman[70791]: 2024-04-26 18:14:10.146846057 +0000 UTC m=+0.686762328 container died 2449d3d7a1ae2c0cb9305ec5c18a60e93f60428127080f903c3a6b943e919d87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-26T18:14:10.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.1.service' 2024-04-26T18:14:10.821 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:14:10.821 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-26T18:14:10.821 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-26T18:14:10.821 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.2 2024-04-26T18:14:11.138 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 18:14:10 smithi186 systemd[1]: Stopping Ceph osd.2 for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:14:11.396 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 18:14:11 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-2[37964]: 2024-04-26T18:14:11.137+0000 7f1620945700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:14:11.396 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 18:14:11 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-2[37964]: 2024-04-26T18:14:11.137+0000 7f1620945700 -1 osd.2 116 *** Got signal Terminated *** 2024-04-26T18:14:11.396 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 18:14:11 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-2[37964]: 2024-04-26T18:14:11.137+0000 7f1620945700 -1 osd.2 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T18:14:17.114 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 18:14:16 smithi186 podman[71628]: 2024-04-26 18:14:16.840718173 +0000 UTC m=+5.823334365 container died 7eb21129363b7af4ae75b3e1dc98ee18fa59a111cc50b2bc25cea84b64a2a9b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-26T18:14:17.482 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 18:14:17 smithi186 podman[71628]: 2024-04-26 18:14:17.213697872 +0000 UTC m=+6.196314102 container cleanup 7eb21129363b7af4ae75b3e1dc98ee18fa59a111cc50b2bc25cea84b64a2a9b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-26T18:14:17.482 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 18:14:17 smithi186 bash[71628]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-2 2024-04-26T18:14:17.767 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 18:14:17 smithi186 podman[71656]: 2024-04-26 18:14:17.482957534 +0000 UTC m=+0.643978344 container remove 7eb21129363b7af4ae75b3e1dc98ee18fa59a111cc50b2bc25cea84b64a2a9b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-26T18:14:18.121 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 18:14:17 smithi186 podman[71749]: 2024-04-26 18:14:17.727759107 +0000 UTC m=+0.018481329 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T18:14:18.121 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 18:14:17 smithi186 podman[71749]: 2024-04-26 18:14:17.842966658 +0000 UTC m=+0.133688872 container create 3c5a69b4f0541da5001432d385dbbb9c081e57e8c14efc21a6aa5694218a88cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-26T18:14:18.412 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 18:14:18 smithi186 podman[71749]: 2024-04-26 18:14:18.121652762 +0000 UTC m=+0.412374975 container init 3c5a69b4f0541da5001432d385dbbb9c081e57e8c14efc21a6aa5694218a88cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:14:18.412 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 18:14:18 smithi186 podman[71749]: 2024-04-26 18:14:18.125013703 +0000 UTC m=+0.415735925 container start 3c5a69b4f0541da5001432d385dbbb9c081e57e8c14efc21a6aa5694218a88cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:14:18.412 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 18:14:18 smithi186 podman[71749]: 2024-04-26 18:14:18.221641281 +0000 UTC m=+0.512363509 container attach 3c5a69b4f0541da5001432d385dbbb9c081e57e8c14efc21a6aa5694218a88cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T18:14:18.767 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 18:14:18 smithi186 podman[71749]: 2024-04-26 18:14:18.413301576 +0000 UTC m=+0.704023789 container died 3c5a69b4f0541da5001432d385dbbb9c081e57e8c14efc21a6aa5694218a88cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-26T18:14:19.093 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.2.service' 2024-04-26T18:14:19.135 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:14:19.136 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-26T18:14:19.136 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-26T18:14:19.136 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.3 2024-04-26T18:14:19.456 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 18:14:19 smithi031 systemd[1]: Stopping Ceph osd.3 for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:14:19.810 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 18:14:19 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-3[48391]: 2024-04-26T18:14:19.455+0000 7f31048bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:14:19.810 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 18:14:19 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-3[48391]: 2024-04-26T18:14:19.455+0000 7f31048bb700 -1 osd.3 116 *** Got signal Terminated *** 2024-04-26T18:14:19.810 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 18:14:19 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-3[48391]: 2024-04-26T18:14:19.455+0000 7f31048bb700 -1 osd.3 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T18:14:21.560 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:21 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:21.469+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:21.560 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:21 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:21.295+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:22.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:22 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:22.467+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:22.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:22 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:22.310+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:23.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:22 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:22.951+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:23.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:22 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:22.795+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:23.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:23 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:23.487+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:23.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:23 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:23.330+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:24.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:23 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:23.932+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:24.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:23 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:23.774+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:24.472 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:24 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:24.298+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:24.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:24 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:24.471+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:25.017 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:24.960+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:25.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:24.752+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:25.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:24 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:24.945+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:25.489 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 18:14:25 smithi031 podman[99866]: 2024-04-26 18:14:25.07675356 +0000 UTC m=+5.742401836 container died 56fb33f9971f9e5db2048113249fbec6751428e85912c1f89e34c1d9bef5904f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-26T18:14:25.490 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:25 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:25.303+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:25.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:25 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:25.520+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:25.810 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 18:14:25 smithi031 podman[99866]: 2024-04-26 18:14:25.581043595 +0000 UTC m=+6.246691870 container cleanup 56fb33f9971f9e5db2048113249fbec6751428e85912c1f89e34c1d9bef5904f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-04-26T18:14:25.810 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 18:14:25 smithi031 bash[99866]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-3 2024-04-26T18:14:26.018 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:25.981+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:26.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:25.736+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:26.152 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 18:14:25 smithi031 podman[99884]: 2024-04-26 18:14:25.815716241 +0000 UTC m=+0.736912257 container remove 56fb33f9971f9e5db2048113249fbec6751428e85912c1f89e34c1d9bef5904f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T18:14:26.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:25 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:25.994+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:26.427 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:26.345+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:26.427 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 18:14:26 smithi031 podman[99975]: 2024-04-26 18:14:26.056401907 +0000 UTC m=+0.018769968 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T18:14:26.427 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 18:14:26 smithi031 podman[99975]: 2024-04-26 18:14:26.158321022 +0000 UTC m=+0.120689068 container create b31ca03ae456def7e59d503922eadaaec9a91a055b2bfbbe8be2078b9fdb8723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-26T18:14:26.427 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 18:14:26 smithi031 podman[99975]: 2024-04-26 18:14:26.427766628 +0000 UTC m=+0.390134675 container init b31ca03ae456def7e59d503922eadaaec9a91a055b2bfbbe8be2078b9fdb8723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T18:14:26.739 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:26 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:26.553+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:26.739 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 18:14:26 smithi031 podman[99975]: 2024-04-26 18:14:26.43116041 +0000 UTC m=+0.393528465 container start b31ca03ae456def7e59d503922eadaaec9a91a055b2bfbbe8be2078b9fdb8723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T18:14:26.739 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 18:14:26 smithi031 podman[99975]: 2024-04-26 18:14:26.519310671 +0000 UTC m=+0.481678722 container attach b31ca03ae456def7e59d503922eadaaec9a91a055b2bfbbe8be2078b9fdb8723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T18:14:27.055 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 26 18:14:26 smithi031 podman[99975]: 2024-04-26 18:14:26.739771294 +0000 UTC m=+0.702139339 container died b31ca03ae456def7e59d503922eadaaec9a91a055b2bfbbe8be2078b9fdb8723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-26T18:14:27.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:26 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:26.975+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:27.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:26 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:26.784+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:27.341 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.3.service' 2024-04-26T18:14:27.382 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:14:27.382 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-26T18:14:27.382 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-26T18:14:27.382 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.4 2024-04-26T18:14:27.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:26 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:26.952+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:27.682 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:27 smithi097 systemd[1]: Stopping Ceph osd.4 for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:14:27.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:27 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:27.541+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:27.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:27 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:27.371+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:28.017 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:27 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:27.961+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:28.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:27 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:27.739+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:28.134 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:27 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:27.918+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:28.135 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:27 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4[43823]: 2024-04-26T18:14:27.682+0000 7f905692c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:14:28.135 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:27 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4[43823]: 2024-04-26T18:14:27.682+0000 7f905692c700 -1 osd.4 116 *** Got signal Terminated *** 2024-04-26T18:14:28.135 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:27 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4[43823]: 2024-04-26T18:14:27.682+0000 7f905692c700 -1 osd.4 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T18:14:28.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:28 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:28.366+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:28.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:28 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:28.515+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:29.143 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:28 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:28.943+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:29.143 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:28 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:28.770+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:29.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:28 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:28.906+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:29.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:29 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:29.538+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:29.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:29 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:29.357+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:29.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:29 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:29.357+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:30.017 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:29 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:29.941+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:30.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:29 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:29.741+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:30.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:30 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:30.077+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:30.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:29 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:29.923+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:30.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:29 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:29.923+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:30.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:30 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:30.325+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:30.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:30 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:30.325+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:30.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:30 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:30.542+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:30.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:30 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:30.542+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:31.017 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:30 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:30.957+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:31.017 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:30 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:30.957+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.546276+0000 front 2024-04-26T18:14:07.546291+0000 (oldest deadline 2024-04-26T18:14:30.446201+0000) 2024-04-26T18:14:31.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:30 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:30.746+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:31.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:30 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:30.746+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:31.134 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:31 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:31.109+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:31.134 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:30 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:30.880+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:31.135 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:30 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:30.880+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:31.767 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:31 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:31.476+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:31.767 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:31 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:31.713+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:31.768 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:31 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:31.713+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:31.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:31 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:31.314+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:31.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:31 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:31.314+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:31.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:31 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:31.543+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:31.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:31 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:31.543+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:32.094 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:31 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4[43823]: 2024-04-26T18:14:31.834+0000 7f9052f20700 -1 osd.4 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:05.902944+0000 front 2024-04-26T18:14:05.902869+0000 (oldest deadline 2024-04-26T18:14:31.803037+0000) 2024-04-26T18:14:32.094 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:31 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:31.919+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:32.094 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:31 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:31.919+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:32.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:31 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:31.984+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:32.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:31 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:31.984+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.546276+0000 front 2024-04-26T18:14:07.546291+0000 (oldest deadline 2024-04-26T18:14:30.446201+0000) 2024-04-26T18:14:32.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:32 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:32.094+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:32.757 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:32 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:32.495+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:32.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:32 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:32.361+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:32.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:32 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:32.361+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:32.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:32 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:32.591+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:32.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:32 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:32.591+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:33.017 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:32 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:32.981+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:33.017 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:32 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:32.981+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.546276+0000 front 2024-04-26T18:14:07.546291+0000 (oldest deadline 2024-04-26T18:14:30.446201+0000) 2024-04-26T18:14:33.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:32 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:32.757+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:33.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:32 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:32.757+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:33.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:33 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:33.100+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:33.385 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:33 smithi097 podman[70900]: 2024-04-26 18:14:33.179900327 +0000 UTC m=+5.609672243 container died d399f4a5e5ea1c87e433403bed2ed8f80478728c555c171a8a6d866b8e20a090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=) 2024-04-26T18:14:33.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:32 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:32.933+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:33.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:32 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:32.933+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:33.797 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:33 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:33.531+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:33.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:33 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:33.606+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:33.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:33 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:33.606+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:33.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:33 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:33.406+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:33.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:33 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:33.406+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:33.834 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:33 smithi097 podman[70900]: 2024-04-26 18:14:33.583049212 +0000 UTC m=+6.012821125 container cleanup d399f4a5e5ea1c87e433403bed2ed8f80478728c555c171a8a6d866b8e20a090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-26T18:14:33.834 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:33 smithi097 bash[70900]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4 2024-04-26T18:14:34.109 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:33 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:33.933+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:34.109 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:33 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:33.933+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:34.109 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:33 smithi097 podman[70918]: 2024-04-26 18:14:33.83447883 +0000 UTC m=+0.656767269 container remove d399f4a5e5ea1c87e433403bed2ed8f80478728c555c171a8a6d866b8e20a090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T18:14:34.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:34 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:34.026+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:34.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:34 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:34.026+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.546276+0000 front 2024-04-26T18:14:07.546291+0000 (oldest deadline 2024-04-26T18:14:30.446201+0000) 2024-04-26T18:14:34.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:33 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:33.798+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:34.268 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:33 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:33.798+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:34.385 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:34 smithi097 podman[71009]: 2024-04-26 18:14:34.096521235 +0000 UTC m=+0.017713725 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T18:14:34.385 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:34 smithi097 podman[71009]: 2024-04-26 18:14:34.210460423 +0000 UTC m=+0.131652910 container create 3948aec59844ed9586f7958b45d9201b52b5963fe5eb399df8133c213083a25a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T18:14:34.386 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:34 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:34.109+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:34.756 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:34 smithi097 podman[71009]: 2024-04-26 18:14:34.463643052 +0000 UTC m=+0.384835542 container init 3948aec59844ed9586f7958b45d9201b52b5963fe5eb399df8133c213083a25a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5) 2024-04-26T18:14:34.756 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:34 smithi097 podman[71009]: 2024-04-26 18:14:34.466911687 +0000 UTC m=+0.388104169 container start 3948aec59844ed9586f7958b45d9201b52b5963fe5eb399df8133c213083a25a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T18:14:34.757 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:34 smithi097 podman[71009]: 2024-04-26 18:14:34.555199522 +0000 UTC m=+0.476392020 container attach 3948aec59844ed9586f7958b45d9201b52b5963fe5eb399df8133c213083a25a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T18:14:34.778 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:34 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:34.525+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:34.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:34 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:34.610+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:34.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:34 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:34.610+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:34.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:34 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:34.433+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:34.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:34 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:34.433+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:35.057 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:34 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:34.966+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:35.057 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:34 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:34.967+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:35.058 INFO:journalctl@ceph.osd.4.smithi097.stdout:Apr 26 18:14:34 smithi097 podman[71009]: 2024-04-26 18:14:34.756818958 +0000 UTC m=+0.678011427 container died 3948aec59844ed9586f7958b45d9201b52b5963fe5eb399df8133c213083a25a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T18:14:35.062 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:34 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:34.776+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:35.062 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:34 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:34.776+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:35.342 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:35 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:35.156+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:35.359 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.4.service' 2024-04-26T18:14:35.398 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:14:35.398 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-26T18:14:35.398 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-26T18:14:35.398 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.5 2024-04-26T18:14:35.424 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:35 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:35.063+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:35.424 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:35 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:35.063+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.546276+0000 front 2024-04-26T18:14:07.546291+0000 (oldest deadline 2024-04-26T18:14:30.446201+0000) 2024-04-26T18:14:35.704 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:35 smithi186 systemd[1]: Stopping Ceph osd.5 for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:14:35.704 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:35 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:35.567+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:35.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:35 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:35.415+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:35.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:35 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:35.415+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:35.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:35 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:35.566+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:35.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:35 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:35.566+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:36.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:35 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:35.825+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:36.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:35 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:35.825+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:36.018 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:35 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:35.704+0000 7f4f8aeb4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:14:36.018 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:35 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:35.704+0000 7f4f8aeb4700 -1 osd.5 116 *** Got signal Terminated *** 2024-04-26T18:14:36.018 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:35 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:35.704+0000 7f4f8aeb4700 -1 osd.5 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T18:14:36.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:36 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:36.118+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:36.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:36 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:36.015+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:36.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:36 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:36.015+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:36.518 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:36 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:36.055+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:36.518 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:36 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:36.055+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.546276+0000 front 2024-04-26T18:14:07.546291+0000 (oldest deadline 2024-04-26T18:14:30.446201+0000) 2024-04-26T18:14:36.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:36 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:36.447+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:36.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:36 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:36.447+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:36.811 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:36 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:36.594+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:36.811 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:36 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:36.594+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:36.858 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:36 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:36.557+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:37.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:37 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:37.035+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:37.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:37 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:37.035+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.546276+0000 front 2024-04-26T18:14:07.546291+0000 (oldest deadline 2024-04-26T18:14:30.446201+0000) 2024-04-26T18:14:37.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:36 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:36.858+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:37.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:36 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:36.858+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:37.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:37 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:37.087+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:37.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:37 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:37.006+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:37.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:37 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:37.006+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:37.767 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:37 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:37.513+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:37.767 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:37 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:37.513+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:37.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:37 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:37.448+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:37.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:37 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:37.448+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:37.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:37 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:37.619+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:37.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:37 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:37.619+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:38.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:38 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:38.015+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:38.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:38 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:38.015+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.546276+0000 front 2024-04-26T18:14:07.546291+0000 (oldest deadline 2024-04-26T18:14:30.446201+0000) 2024-04-26T18:14:38.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:37 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:37.814+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:38.268 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:37 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:37.814+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:38.268 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:37 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:37.814+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:38.285 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:38 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:38.073+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:38.286 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:38 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:38.019+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:38.286 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:38 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:38.019+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:38.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:38 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:38.447+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:38.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:38 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:38.447+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:38.811 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:38 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:38.589+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:38.811 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:38 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:38.589+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:38.812 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:38 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:38.532+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:38.812 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:38 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:38.532+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:39.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:38 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:38.982+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:39.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:38 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:38.982+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.546276+0000 front 2024-04-26T18:14:07.546291+0000 (oldest deadline 2024-04-26T18:14:30.446201+0000) 2024-04-26T18:14:39.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:38 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:38.982+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:15.147047+0000 front 2024-04-26T18:14:15.147121+0000 (oldest deadline 2024-04-26T18:14:38.047213+0000) 2024-04-26T18:14:39.268 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:38 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:38.811+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:39.268 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:38 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:38.811+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:39.268 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:38 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:38.811+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:39.298 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:39 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:39.112+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:39.298 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:39 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:39.112+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:39.298 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:39 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:39.034+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:39.298 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:39 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:39.034+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:39.298 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:39 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:39.034+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:39.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:39 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:39.480+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:39.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:39 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:39.480+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:39.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:39 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:39.480+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:39.811 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:39 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:39.572+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:39.811 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:39 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:39.572+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:39.811 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:39 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:39.572+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.672593+0000 front 2024-04-26T18:14:13.672550+0000 (oldest deadline 2024-04-26T18:14:38.972411+0000) 2024-04-26T18:14:39.851 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:39 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:39.579+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:39.851 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:39 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:39.579+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:40.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:39 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:39.949+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.444268+0000 front 2024-04-26T18:13:56.444277+0000 (oldest deadline 2024-04-26T18:14:22.344101+0000) 2024-04-26T18:14:40.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:39 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:39.949+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.546276+0000 front 2024-04-26T18:14:07.546291+0000 (oldest deadline 2024-04-26T18:14:30.446201+0000) 2024-04-26T18:14:40.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:39 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5[44760]: 2024-04-26T18:14:39.949+0000 7f4f834a0700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:15.147047+0000 front 2024-04-26T18:14:15.147121+0000 (oldest deadline 2024-04-26T18:14:38.047213+0000) 2024-04-26T18:14:40.268 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:39 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:39.851+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:40.268 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:39 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:39.851+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:40.268 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:39 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:39.851+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:40.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:40 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:40.072+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:40.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:40 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:40.072+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:40.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:40 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:40.037+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:40.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:40 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:40.037+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:40.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:40 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:40.037+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:40.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:40 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:40.478+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:40.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:40 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:40.478+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:40.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:40 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:40.478+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:40.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:40 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:40.568+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:40.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:40 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:40.568+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:40.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:40 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:40.568+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.672593+0000 front 2024-04-26T18:14:13.672550+0000 (oldest deadline 2024-04-26T18:14:38.972411+0000) 2024-04-26T18:14:40.863 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:40 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:40.555+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:40.864 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:40 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:40.555+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:41.189 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:40 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:40.864+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:41.190 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:40 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:40.864+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:41.190 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:40 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:40.864+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:41.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:41 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:41.032+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:41.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:41 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:41.032+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:41.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:41 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:41.085+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:41.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:41 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:41.085+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:41.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:41 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:41.085+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:41.488 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:41 smithi186 podman[71853]: 2024-04-26 18:14:41.190592019 +0000 UTC m=+5.598407288 container died 8f8387fbd45474314840dcfcad9cc0945a3397796ad3d69754d3e701634bca63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T18:14:41.767 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:41 smithi186 podman[71853]: 2024-04-26 18:14:41.597095135 +0000 UTC m=+6.004910402 container cleanup 8f8387fbd45474314840dcfcad9cc0945a3397796ad3d69754d3e701634bca63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-26T18:14:41.767 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:41 smithi186 bash[71853]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5 2024-04-26T18:14:41.767 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:41 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:41.516+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:41.768 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:41 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:41.516+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:41.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:41 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:41.482+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:41.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:41 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:41.482+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:41.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:41 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:41.482+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:41.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:41 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:41.601+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:41.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:41 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:41.601+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:41.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:41 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:41.601+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.672593+0000 front 2024-04-26T18:14:13.672550+0000 (oldest deadline 2024-04-26T18:14:38.972411+0000) 2024-04-26T18:14:42.212 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:41 smithi186 podman[71871]: 2024-04-26 18:14:41.798659823 +0000 UTC m=+0.602963483 container remove 8f8387fbd45474314840dcfcad9cc0945a3397796ad3d69754d3e701634bca63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-26T18:14:42.213 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:41 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:41.861+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:42.213 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:41 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:41.861+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:42.213 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:41 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:41.861+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:42.385 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:42 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:42.001+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:42.385 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:42 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:42.001+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:42.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:42 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:42.074+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:42.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:42 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:42.074+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:42.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:42 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:42.074+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:42.476 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:42 smithi186 podman[71959]: 2024-04-26 18:14:42.116699618 +0000 UTC m=+0.018339285 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T18:14:42.476 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:42 smithi186 podman[71959]: 2024-04-26 18:14:42.233104563 +0000 UTC m=+0.134744226 container create b1c9b06eb2787da9f5eaf0a8eccf05a5b02705f8623d6ab25b032cadd044b568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:14:42.477 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:42 smithi186 podman[71959]: 2024-04-26 18:14:42.477309902 +0000 UTC m=+0.378949565 container init b1c9b06eb2787da9f5eaf0a8eccf05a5b02705f8623d6ab25b032cadd044b568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-26T18:14:42.767 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:42 smithi186 podman[71959]: 2024-04-26 18:14:42.480513027 +0000 UTC m=+0.382152691 container start b1c9b06eb2787da9f5eaf0a8eccf05a5b02705f8623d6ab25b032cadd044b568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-26T18:14:42.768 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:42 smithi186 podman[71959]: 2024-04-26 18:14:42.586050287 +0000 UTC m=+0.487689968 container attach b1c9b06eb2787da9f5eaf0a8eccf05a5b02705f8623d6ab25b032cadd044b568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-26T18:14:42.768 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:42 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:42.477+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:42.768 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:42 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:42.477+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:42.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:42 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:42.531+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:42.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:42 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:42.531+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:42.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:42 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:42.531+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:42.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:42 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:42.617+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:42.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:42 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:42.617+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:42.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:42 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:42.617+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.672593+0000 front 2024-04-26T18:14:13.672550+0000 (oldest deadline 2024-04-26T18:14:38.972411+0000) 2024-04-26T18:14:43.097 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 18:14:42 smithi186 podman[71959]: 2024-04-26 18:14:42.776779462 +0000 UTC m=+0.678419128 container died b1c9b06eb2787da9f5eaf0a8eccf05a5b02705f8623d6ab25b032cadd044b568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T18:14:43.097 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:42 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:42.871+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:43.097 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:42 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:42.871+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:43.097 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:42 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:42.871+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:43.383 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.5.service' 2024-04-26T18:14:43.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:43 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:43.032+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:43.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:43 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:43.032+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:43.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:43 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:43.037+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:43.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:43 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:43.037+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:43.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:43 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:43.037+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:43.462 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:14:43.463 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-26T18:14:43.463 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-26T18:14:43.463 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.6 2024-04-26T18:14:43.767 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:43.526+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:43.767 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:43.526+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:43.783 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:43 smithi031 systemd[1]: Stopping Ceph osd.6 for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:14:43.785 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:43.587+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:43.785 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:43.587+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:43.785 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:43.587+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.672593+0000 front 2024-04-26T18:14:13.672550+0000 (oldest deadline 2024-04-26T18:14:38.972411+0000) 2024-04-26T18:14:43.786 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:43.503+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:43.786 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:43.503+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:43.786 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:43.503+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:44.060 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:43.782+0000 7f144de93700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:14:44.060 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:43.782+0000 7f144de93700 -1 osd.6 116 *** Got signal Terminated *** 2024-04-26T18:14:44.060 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:43 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:43.782+0000 7f144de93700 -1 osd.6 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T18:14:44.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:43.886+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:44.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:43.886+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:44.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:43 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:43.886+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:44.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:44 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:44.020+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:44.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:44 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:44.020+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:44.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:44 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:44.064+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:44.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:44 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:44.064+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:44.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:44 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:44.064+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:44.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:44 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:44.629+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:44.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:44 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:44.629+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:44.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:44 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:44.629+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.672593+0000 front 2024-04-26T18:14:13.672550+0000 (oldest deadline 2024-04-26T18:14:38.972411+0000) 2024-04-26T18:14:44.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:44 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:44.509+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:44.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:44 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:44.509+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:44.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:44 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:44.509+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:44.861 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:44 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:44.563+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:44.861 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:44 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:44.563+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:45.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:44 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:44.861+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:45.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:44 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:44.861+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:45.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:44 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:44.861+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:45.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:45 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:45.014+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:45.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:45 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:45.014+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:45.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:45 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:45.071+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:45.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:45 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:45.072+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:45.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:45 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:45.072+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:45.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:45 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:45.586+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:45.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:45 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:45.586+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:45.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:45 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:45.586+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.672593+0000 front 2024-04-26T18:14:13.672550+0000 (oldest deadline 2024-04-26T18:14:38.972411+0000) 2024-04-26T18:14:45.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:45 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:45.519+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:45.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:45 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:45.519+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:45.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:45 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:45.519+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:45.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:45 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:45.603+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:45.900 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:45 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:45.603+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:46.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:45 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:45.900+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:46.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:45 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:45.900+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:46.268 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:45 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:45.900+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:46.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:46 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:46.037+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:46.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:46 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:46.037+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:46.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:46 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:46.116+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:46.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:46 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:46.116+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:46.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:46 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:46.116+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:46.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:46 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:46.116+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:14:46.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:46 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:46.580+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:46.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:46 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:46.580+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:46.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:46 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:46.580+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.672593+0000 front 2024-04-26T18:14:13.672550+0000 (oldest deadline 2024-04-26T18:14:38.972411+0000) 2024-04-26T18:14:46.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:46 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:46.580+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:23.673630+0000 front 2024-04-26T18:14:23.673653+0000 (oldest deadline 2024-04-26T18:14:45.973770+0000) 2024-04-26T18:14:46.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:46 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:46.471+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:46.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:46 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:46.471+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:46.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:46 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:46.471+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:46.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:46 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:46.471+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:14:46.886 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:46 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:46.584+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:46.886 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:46 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:46.584+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:47.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:46 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:46.887+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:47.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:46 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:46.887+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:47.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:46 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:46.887+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:47.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:47 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:47.004+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:47.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:47 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:47.004+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:47.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:47 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:47.004+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.539851+0000 front 2024-04-26T18:14:22.539734+0000 (oldest deadline 2024-04-26T18:14:46.039721+0000) 2024-04-26T18:14:47.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:47 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:47.146+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:47.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:47 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:47.146+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:47.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:47 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:47.146+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:47.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:47 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:47.146+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:14:47.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:47 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:47.557+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:47.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:47 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:47.557+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:47.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:47 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:47.557+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.672593+0000 front 2024-04-26T18:14:13.672550+0000 (oldest deadline 2024-04-26T18:14:38.972411+0000) 2024-04-26T18:14:47.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:47 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:47.557+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:23.673630+0000 front 2024-04-26T18:14:23.673653+0000 (oldest deadline 2024-04-26T18:14:45.973770+0000) 2024-04-26T18:14:47.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:47 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:47.496+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:47.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:47 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:47.496+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:47.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:47 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:47.496+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:47.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:47 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:47.496+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:14:48.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:47 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:47.616+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:48.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:47 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:47.616+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:48.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:47 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:47.843+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:48.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:47 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:47.843+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:48.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:47 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:47.843+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:48.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:47 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:47.955+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:48.384 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:47 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:47.955+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:48.385 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:47 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:47.955+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.539851+0000 front 2024-04-26T18:14:22.539734+0000 (oldest deadline 2024-04-26T18:14:46.039721+0000) 2024-04-26T18:14:48.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:48 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:48.172+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:48.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:48 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:48.172+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:48.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:48 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:48.172+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:48.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:48 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:48.172+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:14:48.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:48 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:48.517+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.169388+0000 front 2024-04-26T18:13:59.169329+0000 (oldest deadline 2024-04-26T18:14:21.469535+0000) 2024-04-26T18:14:48.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:48 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:48.517+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.971023+0000 front 2024-04-26T18:14:04.970827+0000 (oldest deadline 2024-04-26T18:14:30.270833+0000) 2024-04-26T18:14:48.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:48 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:48.517+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.672593+0000 front 2024-04-26T18:14:13.672550+0000 (oldest deadline 2024-04-26T18:14:38.972411+0000) 2024-04-26T18:14:48.810 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:48 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6[55410]: 2024-04-26T18:14:48.517+0000 7f144647f700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:23.673630+0000 front 2024-04-26T18:14:23.673653+0000 (oldest deadline 2024-04-26T18:14:45.973770+0000) 2024-04-26T18:14:48.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:48 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:48.535+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:48.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:48 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:48.535+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:48.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:48 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:48.535+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:48.811 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:48 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:48.535+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:14:48.879 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:48 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:48.596+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:48.879 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:48 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:48.596+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:48.879 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:48 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:48.596+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:14:49.208 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:48 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:48.929+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:49.208 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:48 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:48.929+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:49.208 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:48 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:48.929+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.539851+0000 front 2024-04-26T18:14:22.539734+0000 (oldest deadline 2024-04-26T18:14:46.039721+0000) 2024-04-26T18:14:49.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:48 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:48.879+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:49.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:48 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:48.879+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:49.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:48 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:48.879+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:49.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:49 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:49.208+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:49.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:49 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:49.208+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:49.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:49 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:49.208+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:49.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:49 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:49.208+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:14:49.742 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:49 smithi031 podman[100081]: 2024-04-26 18:14:49.460489032 +0000 UTC m=+5.799226519 container died 38a36a9ef15a1aab3bb60374e673b6433742d45d1a62031038ff9298b58014af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T18:14:49.742 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:49 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:49.488+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:49.742 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:49 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:49.488+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:49.742 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:49 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:49.488+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:49.743 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:49 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:49.488+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:14:49.832 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:49 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:49.577+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:49.832 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:49 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:49.577+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:49.832 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:49 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:49.577+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:14:50.059 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:49 smithi031 podman[100081]: 2024-04-26 18:14:49.832702349 +0000 UTC m=+6.171439828 container cleanup 38a36a9ef15a1aab3bb60374e673b6433742d45d1a62031038ff9298b58014af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T18:14:50.060 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:49 smithi031 bash[100081]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6 2024-04-26T18:14:50.236 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:49 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:49.953+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:50.236 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:49 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:49.953+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:50.236 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:49 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:49.953+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.539851+0000 front 2024-04-26T18:14:22.539734+0000 (oldest deadline 2024-04-26T18:14:46.039721+0000) 2024-04-26T18:14:50.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:49 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:49.832+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:50.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:49 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:49.832+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:50.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:49 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:49.832+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:50.442 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:50 smithi031 podman[100111]: 2024-04-26 18:14:50.100665512 +0000 UTC m=+0.630626154 container remove 38a36a9ef15a1aab3bb60374e673b6433742d45d1a62031038ff9298b58014af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-26T18:14:50.546 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:50 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:50.235+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:50.546 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:50 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:50.235+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:50.547 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:50 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:50.235+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:50.547 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:50 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:50.235+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:14:50.762 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:50 smithi031 podman[100203]: 2024-04-26 18:14:50.443312621 +0000 UTC m=+0.102006365 container create 3b0cda6a16217ae758f76ea867ea78e5fd1fa4bb5adb27b10de65a275135e6b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-26T18:14:50.762 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:50 smithi031 podman[100203]: 2024-04-26 18:14:50.359840199 +0000 UTC m=+0.018533968 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T18:14:50.762 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:50 smithi031 podman[100203]: 2024-04-26 18:14:50.678968277 +0000 UTC m=+0.337662022 container init 3b0cda6a16217ae758f76ea867ea78e5fd1fa4bb5adb27b10de65a275135e6b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T18:14:50.762 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:50 smithi031 podman[100203]: 2024-04-26 18:14:50.682218773 +0000 UTC m=+0.340912526 container start 3b0cda6a16217ae758f76ea867ea78e5fd1fa4bb5adb27b10de65a275135e6b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T18:14:50.762 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:50 smithi031 podman[100203]: 2024-04-26 18:14:50.762093056 +0000 UTC m=+0.420786813 container attach 3b0cda6a16217ae758f76ea867ea78e5fd1fa4bb5adb27b10de65a275135e6b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T18:14:50.762 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:50 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:50.487+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:50.762 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:50 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:50.487+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:50.763 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:50 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:50.487+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:50.763 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:50 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:50.487+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:14:50.872 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:50 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:50.612+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:50.872 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:50 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:50.612+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:50.873 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:50 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:50.612+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:14:51.018 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 26 18:14:50 smithi031 podman[100203]: 2024-04-26 18:14:50.986702634 +0000 UTC m=+0.645396395 container died 3b0cda6a16217ae758f76ea867ea78e5fd1fa4bb5adb27b10de65a275135e6b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-26T18:14:51.242 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:50 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:50.925+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:51.242 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:50 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:50.925+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:51.243 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:50 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:50.925+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.539851+0000 front 2024-04-26T18:14:22.539734+0000 (oldest deadline 2024-04-26T18:14:46.039721+0000) 2024-04-26T18:14:51.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:50 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:50.872+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:51.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:50 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:50.872+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:51.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:50 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:50.872+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:51.632 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:51 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:51.485+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:51.632 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:51 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:51.485+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:51.632 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:51 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:51.485+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:51.632 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:51 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:51.485+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:14:51.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:51 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:51.242+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:51.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:51 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:51.242+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:51.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:51 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:51.242+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:51.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:51 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:51.242+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:14:51.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.6.service' 2024-04-26T18:14:51.706 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:14:51.707 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-26T18:14:51.707 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-26T18:14:51.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.7 2024-04-26T18:14:52.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:51 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:51.639+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:52.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:51 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:51.639+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:52.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:51 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:51.639+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:14:52.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:51 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:51.827+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:52.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:51 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:51.827+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:52.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:51 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:51.827+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:52.021 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:51 smithi097 systemd[1]: Stopping Ceph osd.7 for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:14:52.021 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:51 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:51.904+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:52.021 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:51 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:51.904+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:52.022 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:51 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:51.904+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.539851+0000 front 2024-04-26T18:14:22.539734+0000 (oldest deadline 2024-04-26T18:14:46.039721+0000) 2024-04-26T18:14:52.022 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:52 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:52.020+0000 7f61beedd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:14:52.022 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:52 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:52.020+0000 7f61beedd700 -1 osd.7 116 *** Got signal Terminated *** 2024-04-26T18:14:52.022 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:52 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:52.020+0000 7f61beedd700 -1 osd.7 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T18:14:52.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:52 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:52.282+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:52.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:52 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:52.282+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:52.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:52 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:52.282+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:52.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:52 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:52.282+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:14:52.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:52 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:52.483+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:52.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:52 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:52.483+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:52.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:52 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:52.483+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:52.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:52 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:52.483+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:14:52.862 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:52 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:52.607+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:52.862 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:52 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:52.607+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:52.862 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:52 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:52.607+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:14:53.134 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:52 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:52.876+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:53.134 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:52 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:52.876+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:53.134 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:52 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:52.876+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.539851+0000 front 2024-04-26T18:14:22.539734+0000 (oldest deadline 2024-04-26T18:14:46.039721+0000) 2024-04-26T18:14:53.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:52 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:52.861+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:53.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:52 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:52.861+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:53.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:52 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:52.861+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:53.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:53 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:53.235+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:53.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:53 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:53.235+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:53.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:53 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:53.235+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:53.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:53 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:53.235+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:14:53.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:53 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:53.492+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:53.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:53 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:53.492+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:53.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:53 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:53.492+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:53.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:53 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:53.492+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:14:54.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:53 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:53.591+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:54.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:53 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:53.591+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:54.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:53 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:53.591+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:14:54.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:53 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:53.817+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:54.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:53 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:53.817+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:54.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:53 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:53.817+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:54.224 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:53 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:53.908+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:54.224 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:53 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:53.908+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:54.224 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:53 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:53.908+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.539851+0000 front 2024-04-26T18:14:22.539734+0000 (oldest deadline 2024-04-26T18:14:46.039721+0000) 2024-04-26T18:14:54.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:54 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:54.223+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:54.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:54 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:54.223+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:54.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:54 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:54.223+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:54.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:54 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:54.223+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:14:54.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:54 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:54.466+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:54.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:54 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:54.466+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:54.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:54 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:54.466+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:54.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:54 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:54.466+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:14:54.859 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:54 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:54.599+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:54.859 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:54 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:54.599+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:54.859 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:54 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:54.599+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:14:54.859 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:54 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:54.599+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.081859+0000 front 2024-04-26T18:14:31.081946+0000 (oldest deadline 2024-04-26T18:14:53.982064+0000) 2024-04-26T18:14:55.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:54 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:54.858+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:55.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:54 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:54.858+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:55.267 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:54 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:54.858+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:55.272 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:54 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:54.883+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:55.273 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:54 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:54.883+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:55.273 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:54 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:54.883+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.539851+0000 front 2024-04-26T18:14:22.539734+0000 (oldest deadline 2024-04-26T18:14:46.039721+0000) 2024-04-26T18:14:55.273 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:54 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:54.883+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.241154+0000 front 2024-04-26T18:14:31.241207+0000 (oldest deadline 2024-04-26T18:14:54.741321+0000) 2024-04-26T18:14:55.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:55 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:55.271+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:55.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:55 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:55.271+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:55.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:55 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:55.271+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:55.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:55 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:55.271+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:14:55.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:55 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:55.456+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:55.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:55 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:55.456+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:55.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:55 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:55.456+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:55.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:55 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:55.456+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:14:56.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:55 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:55.608+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:56.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:55 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:55.608+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:56.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:55 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:55.608+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:14:56.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:55 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:55.608+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.081859+0000 front 2024-04-26T18:14:31.081946+0000 (oldest deadline 2024-04-26T18:14:53.982064+0000) 2024-04-26T18:14:56.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:55 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:55.832+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:56.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:55 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:55.832+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:56.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:55 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:55.832+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:56.255 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:55 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:55.906+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:56.255 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:55 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:55.906+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:56.255 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:55 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:55.906+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.539851+0000 front 2024-04-26T18:14:22.539734+0000 (oldest deadline 2024-04-26T18:14:46.039721+0000) 2024-04-26T18:14:56.255 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:55 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:55.906+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.241154+0000 front 2024-04-26T18:14:31.241207+0000 (oldest deadline 2024-04-26T18:14:54.741321+0000) 2024-04-26T18:14:56.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:56 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:56.254+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:56.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:56 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:56.254+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:56.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:56 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:56.254+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:56.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:56 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:56.254+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:14:56.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:56 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:56.425+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:56.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:56 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:56.425+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:56.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:56 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:56.425+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:56.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:56 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:56.425+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:14:57.024 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:56 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:56.637+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:57.024 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:56 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:56.637+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:57.024 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:56 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:56.637+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:14:57.024 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:56 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:56.637+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.081859+0000 front 2024-04-26T18:14:31.081946+0000 (oldest deadline 2024-04-26T18:14:53.982064+0000) 2024-04-26T18:14:57.024 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:56 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:56.806+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:57.024 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:56 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:56.806+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:57.024 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:56 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:56.806+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:57.211 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:56 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:56.910+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:07.437043+0000 front 2024-04-26T18:14:07.437063+0000 (oldest deadline 2024-04-26T18:14:29.137198+0000) 2024-04-26T18:14:57.211 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:56 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:56.910+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.238260+0000 front 2024-04-26T18:14:13.238197+0000 (oldest deadline 2024-04-26T18:14:38.538234+0000) 2024-04-26T18:14:57.211 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:56 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:56.910+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.539851+0000 front 2024-04-26T18:14:22.539734+0000 (oldest deadline 2024-04-26T18:14:46.039721+0000) 2024-04-26T18:14:57.211 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:56 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7[50798]: 2024-04-26T18:14:56.910+0000 7f61bb4d1700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.241154+0000 front 2024-04-26T18:14:31.241207+0000 (oldest deadline 2024-04-26T18:14:54.741321+0000) 2024-04-26T18:14:57.584 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:57 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:57.207+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:57.584 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:57 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:57.207+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:57.585 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:57 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:57.207+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:57.585 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:57 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:57.207+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:14:57.585 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:57 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:57.207+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:14:57.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:57 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:57.411+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:57.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:57 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:57.411+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:57.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:57 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:57.411+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:57.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:57 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:57.411+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:14:57.884 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:57 smithi097 podman[71128]: 2024-04-26 18:14:57.58486364 +0000 UTC m=+5.682016705 container died 8bfc575ef7ff7d2acd35c621573b03e15dd09db1c22e12ee0d451fc93c642e46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T18:14:58.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:57 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:57.649+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:58.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:57 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:57.649+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:58.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:57 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:57.649+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:14:58.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:57 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:57.649+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.081859+0000 front 2024-04-26T18:14:31.081946+0000 (oldest deadline 2024-04-26T18:14:53.982064+0000) 2024-04-26T18:14:58.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:57 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:57.825+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:58.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:57 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:57.825+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:58.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:57 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:57.825+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:58.019 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:57 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:57.825+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:14:58.160 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:57 smithi097 podman[71128]: 2024-04-26 18:14:57.899455209 +0000 UTC m=+5.996608279 container cleanup 8bfc575ef7ff7d2acd35c621573b03e15dd09db1c22e12ee0d451fc93c642e46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T18:14:58.160 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:57 smithi097 bash[71128]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7 2024-04-26T18:14:58.160 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:58 smithi097 podman[71145]: 2024-04-26 18:14:58.147827771 +0000 UTC m=+0.561430137 container remove 8bfc575ef7ff7d2acd35c621573b03e15dd09db1c22e12ee0d451fc93c642e46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T18:14:58.161 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:58 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:58.159+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:58.161 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:58 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:58.159+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:58.161 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:58 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:58.159+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:58.161 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:58 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:58.159+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:14:58.161 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:58 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:58.159+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:14:58.717 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:58 smithi097 podman[71235]: 2024-04-26 18:14:58.352497919 +0000 UTC m=+0.017765054 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T18:14:58.717 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:58 smithi097 podman[71235]: 2024-04-26 18:14:58.457031379 +0000 UTC m=+0.122298502 container create 9b5caece317a0d8f321333f546a6489d8f01eaf77be41b454a8220a2a32c899f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-26T18:14:58.717 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:58 smithi097 podman[71235]: 2024-04-26 18:14:58.717784336 +0000 UTC m=+0.383051463 container init 9b5caece317a0d8f321333f546a6489d8f01eaf77be41b454a8220a2a32c899f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:14:58.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:58 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:58.387+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:58.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:58 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:58.387+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:58.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:58 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:58.387+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:58.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:58 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:58.387+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:14:59.010 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:58 smithi097 podman[71235]: 2024-04-26 18:14:58.721168194 +0000 UTC m=+0.386435319 container start 9b5caece317a0d8f321333f546a6489d8f01eaf77be41b454a8220a2a32c899f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T18:14:59.010 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:58 smithi097 podman[71235]: 2024-04-26 18:14:58.80118036 +0000 UTC m=+0.466447504 container attach 9b5caece317a0d8f321333f546a6489d8f01eaf77be41b454a8220a2a32c899f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD) 2024-04-26T18:14:59.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:58 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:58.638+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:59.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:58 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:58.638+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:59.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:58 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:58.638+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:14:59.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:58 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:58.638+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.081859+0000 front 2024-04-26T18:14:31.081946+0000 (oldest deadline 2024-04-26T18:14:53.982064+0000) 2024-04-26T18:14:59.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:58 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:58.778+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:59.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:58 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:58.778+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:59.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:58 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:58.778+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:59.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:58 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:58.778+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:14:59.281 INFO:journalctl@ceph.osd.7.smithi097.stdout:Apr 26 18:14:59 smithi097 podman[71235]: 2024-04-26 18:14:59.010917331 +0000 UTC m=+0.676184457 container died 9b5caece317a0d8f321333f546a6489d8f01eaf77be41b454a8220a2a32c899f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-26T18:14:59.281 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:59 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:59.112+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:14:59.281 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:59 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:59.112+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:14:59.281 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:59 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:59.112+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:14:59.281 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:59 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:59.112+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:14:59.281 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:14:59 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:14:59.112+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:14:59.581 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.7.service' 2024-04-26T18:14:59.618 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:14:59.618 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-26T18:14:59.618 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-26T18:14:59.618 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.8 2024-04-26T18:14:59.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:59 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:59.427+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:14:59.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:59 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:59.427+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:14:59.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:59 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:59.427+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:14:59.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:14:59 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:14:59.427+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:14:59.961 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:59 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:59.803+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:14:59.961 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:59 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:59.803+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:14:59.961 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:59 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:59.804+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:14:59.961 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:14:59 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:14:59.804+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:14:59.962 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:59 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:59.656+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:14:59.962 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:59 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:59.656+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:14:59.962 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:59 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:59.656+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:14:59.962 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:59 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:59.656+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.081859+0000 front 2024-04-26T18:14:31.081946+0000 (oldest deadline 2024-04-26T18:14:53.982064+0000) 2024-04-26T18:14:59.962 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:59 smithi186 systemd[1]: Stopping Ceph osd.8 for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:14:59.962 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:59 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:59.960+0000 7fb882b84700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:14:59.962 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:59 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:59.960+0000 7fb882b84700 -1 osd.8 116 *** Got signal Terminated *** 2024-04-26T18:14:59.962 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:14:59 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:14:59.960+0000 7fb882b84700 -1 osd.8 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T18:15:00.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:00 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:00.157+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:00.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:00 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:00.157+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:00.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:00 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:00.157+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:00.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:00 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:00.157+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:00.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:00 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:00.157+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:00.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:00 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:00.402+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:15:00.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:00 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:00.402+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:15:00.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:00 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:00.402+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:15:00.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:00 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:00.402+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:15:01.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:00 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:00.696+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:15:01.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:00 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:00.696+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:15:01.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:00 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:00.696+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:15:01.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:00 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:00.696+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.081859+0000 front 2024-04-26T18:14:31.081946+0000 (oldest deadline 2024-04-26T18:14:53.982064+0000) 2024-04-26T18:15:01.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:00 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:00.762+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:01.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:00 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:00.762+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:01.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:00 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:00.762+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:01.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:00 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:00.762+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:01.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:01 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:01.171+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:01.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:01 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:01.171+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:01.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:01 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:01.171+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:01.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:01 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:01.171+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:01.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:01 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:01.171+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:01.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:01 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:01.396+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:15:01.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:01 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:01.396+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:15:01.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:01 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:01.396+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:15:01.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:01 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:01.396+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:15:02.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:01 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:01.669+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:15:02.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:01 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:01.669+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:15:02.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:01 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:01.669+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:15:02.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:01 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:01.669+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.081859+0000 front 2024-04-26T18:14:31.081946+0000 (oldest deadline 2024-04-26T18:14:53.982064+0000) 2024-04-26T18:15:02.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:01 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:01.747+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:02.019 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:01 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:01.747+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:02.019 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:01 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:01.747+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:02.019 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:01 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:01.747+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:02.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:02 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:02.205+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:02.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:02 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:02.205+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:02.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:02 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:02.205+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:02.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:02 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:02.205+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:02.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:02 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:02.205+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:02.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:02 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:02.409+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:15:02.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:02 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:02.409+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:15:02.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:02 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:02.409+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:15:02.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:02 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:02.409+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:15:03.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:02 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:02.643+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:15:03.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:02 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:02.643+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:15:03.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:02 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:02.643+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:15:03.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:02 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:02.643+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.081859+0000 front 2024-04-26T18:14:31.081946+0000 (oldest deadline 2024-04-26T18:14:53.982064+0000) 2024-04-26T18:15:03.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:02 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:02.643+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:38.682993+0000 front 2024-04-26T18:14:38.683012+0000 (oldest deadline 2024-04-26T18:15:02.083334+0000) 2024-04-26T18:15:03.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:02 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:02.742+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:03.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:02 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:02.742+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:03.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:02 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:02.742+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:03.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:02 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:02.742+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:03.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:02 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:02.742+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:03.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:03 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:03.203+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:03.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:03 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:03.203+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:03.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:03 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:03.203+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:03.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:03 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:03.203+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:03.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:03 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:03.203+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:03.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:03 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:03.203+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:03.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:03 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:03.371+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:15:03.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:03 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:03.371+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:15:03.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:03 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:03.371+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:15:03.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:03 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:03.371+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:15:04.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:03.617+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:15:04.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:03.617+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:15:04.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:03.617+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:15:04.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:03.617+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.081859+0000 front 2024-04-26T18:14:31.081946+0000 (oldest deadline 2024-04-26T18:14:53.982064+0000) 2024-04-26T18:15:04.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:03.617+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:38.682993+0000 front 2024-04-26T18:14:38.683012+0000 (oldest deadline 2024-04-26T18:15:02.083334+0000) 2024-04-26T18:15:04.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:03.715+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:04.019 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:03.715+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:04.019 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:03.715+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:04.019 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:03.715+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:04.019 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:03 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:03.715+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:04.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:04 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:04.221+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:04.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:04 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:04.221+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:04.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:04 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:04.221+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:04.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:04 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:04.221+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:04.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:04 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:04.221+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:04.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:04 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:04.221+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:04.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:04 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:04.398+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:15:04.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:04 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:04.398+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:15:04.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:04 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:04.398+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:15:04.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:04 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:04.398+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:15:05.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:04 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:04.587+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.578250+0000 front 2024-04-26T18:14:06.578128+0000 (oldest deadline 2024-04-26T18:14:31.278056+0000) 2024-04-26T18:15:05.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:04 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:04.587+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:11.278383+0000 front 2024-04-26T18:14:11.278458+0000 (oldest deadline 2024-04-26T18:14:37.178477+0000) 2024-04-26T18:15:05.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:04 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:04.587+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:22.380516+0000 front 2024-04-26T18:14:22.380531+0000 (oldest deadline 2024-04-26T18:14:48.180962+0000) 2024-04-26T18:15:05.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:04 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:04.587+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.081859+0000 front 2024-04-26T18:14:31.081946+0000 (oldest deadline 2024-04-26T18:14:53.982064+0000) 2024-04-26T18:15:05.018 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:04 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-26T18:15:04.587+0000 7fb87f178700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:38.682993+0000 front 2024-04-26T18:14:38.683012+0000 (oldest deadline 2024-04-26T18:15:02.083334+0000) 2024-04-26T18:15:05.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:04 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:04.727+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:05.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:04 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:04.727+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:05.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:04 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:04.727+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:05.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:04 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:04.727+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:05.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:04 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:04.727+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:05.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:05.248+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:05.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:05.248+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:05.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:05.248+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:05.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:05.248+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:05.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:05.248+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:05.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:05 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:05.248+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:05.767 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:05 smithi186 podman[72063]: 2024-04-26 18:15:05.503874779 +0000 UTC m=+5.687455535 container died eb37c8feddbdfa05de705f95a8ca84ed883151536cfb53e51c77a11528ea4976 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-26T18:15:05.767 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:05 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:05.712+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:05.768 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:05 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:05.712+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:05.768 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:05 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:05.712+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:05.768 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:05 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:05.712+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:05.768 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:05 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:05.712+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:05.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:05 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:05.435+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:15:05.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:05 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:05.435+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:15:05.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:05 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:05.435+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:15:05.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:05 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:05.435+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:15:06.144 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:05 smithi186 podman[72063]: 2024-04-26 18:15:05.901924065 +0000 UTC m=+6.085504809 container cleanup eb37c8feddbdfa05de705f95a8ca84ed883151536cfb53e51c77a11528ea4976 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5) 2024-04-26T18:15:06.144 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:05 smithi186 bash[72063]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8 2024-04-26T18:15:06.473 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:06 smithi186 podman[72080]: 2024-04-26 18:15:06.145100888 +0000 UTC m=+0.643402636 container remove eb37c8feddbdfa05de705f95a8ca84ed883151536cfb53e51c77a11528ea4976 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, ceph=True) 2024-04-26T18:15:06.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:06 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:06.232+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:06.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:06 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:06.232+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:06.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:06 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:06.232+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:06.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:06 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:06.232+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:06.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:06 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:06.232+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:06.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:06 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:06.232+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:06.748 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:06 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:06.669+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:06.748 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:06 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:06.669+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:06.748 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:06 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:06.669+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:06.748 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:06 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:06.669+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:06.748 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:06 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:06.669+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:06.748 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:06 smithi186 podman[72172]: 2024-04-26 18:15:06.376616917 +0000 UTC m=+0.018090629 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T18:15:06.748 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:06 smithi186 podman[72172]: 2024-04-26 18:15:06.49601344 +0000 UTC m=+0.137487143 container create 5580d20dff7f84338fc63ca49a0b49fe196a5556fe8918195a922a6d1204e992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-26T18:15:06.748 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:06 smithi186 podman[72172]: 2024-04-26 18:15:06.748613796 +0000 UTC m=+0.390087498 container init 5580d20dff7f84338fc63ca49a0b49fe196a5556fe8918195a922a6d1204e992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T18:15:06.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:06 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:06.431+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:15:06.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:06 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:06.431+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:15:06.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:06 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:06.431+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:15:06.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:06 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:06.431+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:15:06.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:06 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:06.431+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:40.469250+0000 front 2024-04-26T18:14:40.469325+0000 (oldest deadline 2024-04-26T18:15:06.369177+0000) 2024-04-26T18:15:07.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:06 smithi186 podman[72172]: 2024-04-26 18:15:06.753085651 +0000 UTC m=+0.394559355 container start 5580d20dff7f84338fc63ca49a0b49fe196a5556fe8918195a922a6d1204e992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-26T18:15:07.017 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:06 smithi186 podman[72172]: 2024-04-26 18:15:06.857098962 +0000 UTC m=+0.498572683 container attach 5580d20dff7f84338fc63ca49a0b49fe196a5556fe8918195a922a6d1204e992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-26T18:15:07.328 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 18:15:07 smithi186 podman[72172]: 2024-04-26 18:15:07.055827672 +0000 UTC m=+0.697301367 container died 5580d20dff7f84338fc63ca49a0b49fe196a5556fe8918195a922a6d1204e992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-26T18:15:07.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:07 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:07.204+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:07.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:07 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:07.204+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:07.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:07 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:07.204+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:07.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:07 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:07.204+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:07.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:07 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:07.204+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:07.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:07 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:07.204+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:07.747 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.8.service' 2024-04-26T18:15:07.785 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:15:07.785 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-26T18:15:07.785 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-26T18:15:07.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.9 2024-04-26T18:15:07.809 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:07 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:07.390+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:15:07.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:07 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:07.390+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:15:07.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:07 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:07.390+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:15:07.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:07 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:07.390+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:15:07.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:07 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:07.390+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:40.469250+0000 front 2024-04-26T18:14:40.469325+0000 (oldest deadline 2024-04-26T18:15:06.369177+0000) 2024-04-26T18:15:08.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:07 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:07.673+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:08.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:07 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:07.673+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:08.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:07 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:07.673+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:08.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:07 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:07.673+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:08.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:07 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:07.673+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:08.109 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:07 smithi031 systemd[1]: Stopping Ceph osd.9 for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:15:08.413 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:08 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:08.108+0000 7f5d5dca1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:15:08.414 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:08 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:08.108+0000 7f5d5dca1700 -1 osd.9 116 *** Got signal Terminated *** 2024-04-26T18:15:08.414 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:08 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:08.108+0000 7f5d5dca1700 -1 osd.9 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T18:15:08.535 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:08 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:08.162+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:08.536 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:08 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:08.162+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:08.536 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:08 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:08.162+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:08.536 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:08 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:08.162+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:08.536 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:08 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:08.162+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:08.536 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:08 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:08.162+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:08.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:08 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:08.413+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:15:08.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:08 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:08.413+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:15:08.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:08 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:08.413+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:15:08.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:08 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:08.413+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:15:08.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:08 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:08.413+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:40.469250+0000 front 2024-04-26T18:14:40.469325+0000 (oldest deadline 2024-04-26T18:15:06.369177+0000) 2024-04-26T18:15:09.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:08 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:08.649+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:09.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:08 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:08.649+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:09.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:08 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:08.649+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:09.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:08 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:08.649+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:09.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:08 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:08.649+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:09.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:09 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:09.151+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:09.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:09 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:09.151+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:09.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:09 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:09.151+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:09.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:09 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:09.151+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:09.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:09 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:09.151+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:09.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:09 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:09.151+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:09.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:09 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:09.451+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:15:09.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:09 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:09.451+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:15:09.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:09 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:09.451+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:15:09.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:09 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:09.451+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:15:09.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:09 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:09.451+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:40.469250+0000 front 2024-04-26T18:14:40.469325+0000 (oldest deadline 2024-04-26T18:15:06.369177+0000) 2024-04-26T18:15:10.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:09 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:09.608+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:10.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:09 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:09.608+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:10.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:09 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:09.608+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:10.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:09 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:09.608+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:10.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:09 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:09.608+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:10.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:10 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:10.124+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:10.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:10 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:10.124+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:10.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:10 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:10.124+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:10.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:10 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:10.124+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:10.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:10 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:10.124+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:10.385 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:10 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:10.124+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:10.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:10 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:10.493+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:15:10.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:10 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:10.493+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:15:10.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:10 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:10.493+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:15:10.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:10 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:10.493+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:15:10.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:10 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:10.493+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:40.469250+0000 front 2024-04-26T18:14:40.469325+0000 (oldest deadline 2024-04-26T18:15:06.369177+0000) 2024-04-26T18:15:11.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:10 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:10.653+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:11.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:10 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:10.653+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:11.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:10 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:10.653+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:11.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:10 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:10.653+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:11.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:10 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:10.653+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:11.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:11 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:11.154+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:11.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:11 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:11.154+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:11.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:11 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:11.154+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:11.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:11 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:11.154+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:11.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:11 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:11.154+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:11.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:11 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:11.154+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:11.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:11 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:11.455+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:15:11.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:11 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:11.455+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:15:11.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:11 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:11.455+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:15:11.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:11 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:11.455+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:15:11.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:11 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:11.455+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:40.469250+0000 front 2024-04-26T18:14:40.469325+0000 (oldest deadline 2024-04-26T18:15:06.369177+0000) 2024-04-26T18:15:11.918 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:11 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:11.603+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:11.918 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:11 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:11.603+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:11.918 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:11 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:11.603+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:11.918 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:11 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:11.603+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:11.918 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:11 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:11.603+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:12.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:12 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:12.190+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:12.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:12 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:12.190+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:12.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:12 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:12.190+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:12.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:12 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:12.190+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:12.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:12 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:12.190+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:12.636 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:12 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:12.190+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:12.636 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:12 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:12.190+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.927725+0000 front 2024-04-26T18:14:45.927818+0000 (oldest deadline 2024-04-26T18:15:11.227394+0000) 2024-04-26T18:15:12.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:12 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:12.438+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:54.762593+0000 front 2024-04-26T18:13:54.762551+0000 (oldest deadline 2024-04-26T18:14:20.662764+0000) 2024-04-26T18:15:12.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:12 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:12.438+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.164253+0000 front 2024-04-26T18:14:04.164279+0000 (oldest deadline 2024-04-26T18:14:28.864145+0000) 2024-04-26T18:15:12.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:12 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:12.438+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:13.565215+0000 front 2024-04-26T18:14:13.565308+0000 (oldest deadline 2024-04-26T18:14:39.465100+0000) 2024-04-26T18:15:12.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:12 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:12.438+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:21.166435+0000 front 2024-04-26T18:14:21.166554+0000 (oldest deadline 2024-04-26T18:14:45.866545+0000) 2024-04-26T18:15:12.810 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:12 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9[62463]: 2024-04-26T18:15:12.438+0000 7f5d5628d700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:40.469250+0000 front 2024-04-26T18:14:40.469325+0000 (oldest deadline 2024-04-26T18:15:06.369177+0000) 2024-04-26T18:15:12.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:12 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:12.578+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:12.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:12 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:12.578+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:12.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:12 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:12.578+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:12.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:12 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:12.578+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:12.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:12 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:12.578+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:12.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:12 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:12.578+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:13.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:13 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:13.182+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:13.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:13 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:13.182+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:13.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:13 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:13.182+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:13.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:13 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:13.182+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:13.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:13 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:13.182+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:13.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:13 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:13.182+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:13.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:13 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:13.182+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.927725+0000 front 2024-04-26T18:14:45.927818+0000 (oldest deadline 2024-04-26T18:15:11.227394+0000) 2024-04-26T18:15:13.992 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:13 smithi031 podman[100308]: 2024-04-26 18:15:13.657801814 +0000 UTC m=+5.663253337 container died b73b20d2bdb08deb158e2edf35a465eb4deddabe34e750f1e1ad6d13ca64a15c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T18:15:14.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:13 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:13.564+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:14.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:13 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:13.564+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:14.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:13 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:13.564+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:14.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:13 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:13.564+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:14.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:13 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:13.564+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:14.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:13 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:13.564+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:14.310 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:14 smithi031 podman[100308]: 2024-04-26 18:15:14.083880357 +0000 UTC m=+6.089331889 container cleanup b73b20d2bdb08deb158e2edf35a465eb4deddabe34e750f1e1ad6d13ca64a15c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-26T18:15:14.310 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:14 smithi031 bash[100308]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9 2024-04-26T18:15:14.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:14 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:14.212+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:14.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:14 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:14.212+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:14.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:14 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:14.212+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:14.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:14 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:14.212+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:14.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:14 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:14.212+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:14.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:14 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:14.212+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:14.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:14 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:14.212+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.927725+0000 front 2024-04-26T18:14:45.927818+0000 (oldest deadline 2024-04-26T18:15:11.227394+0000) 2024-04-26T18:15:14.752 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:14 smithi031 podman[100326]: 2024-04-26 18:15:14.318272862 +0000 UTC m=+0.652114257 container remove b73b20d2bdb08deb158e2edf35a465eb4deddabe34e750f1e1ad6d13ca64a15c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-26T18:15:15.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:14 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:14.528+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:15.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:14 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:14.528+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:15.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:14 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:14.528+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:15.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:14 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:14.528+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:15.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:14 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:14.528+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:15.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:14 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:14.528+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:15.054 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:14 smithi031 podman[100416]: 2024-04-26 18:15:14.752673298 +0000 UTC m=+0.109458556 container create 11894d490054d8a4d6f2a29ea6a46f821afc75a2b25a1f901254932c292cdcb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-04-26T18:15:15.054 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:14 smithi031 podman[100416]: 2024-04-26 18:15:14.66177554 +0000 UTC m=+0.018560771 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T18:15:15.054 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:14 smithi031 podman[100416]: 2024-04-26 18:15:14.980583701 +0000 UTC m=+0.337368920 container init 11894d490054d8a4d6f2a29ea6a46f821afc75a2b25a1f901254932c292cdcb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-26T18:15:15.054 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:14 smithi031 podman[100416]: 2024-04-26 18:15:14.984206936 +0000 UTC m=+0.340992170 container start 11894d490054d8a4d6f2a29ea6a46f821afc75a2b25a1f901254932c292cdcb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-26T18:15:15.054 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:15 smithi031 podman[100416]: 2024-04-26 18:15:15.054701544 +0000 UTC m=+0.411486785 container attach 11894d490054d8a4d6f2a29ea6a46f821afc75a2b25a1f901254932c292cdcb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-04-26T18:15:15.310 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:15 smithi031 podman[100416]: 2024-04-26 18:15:15.28274982 +0000 UTC m=+0.639535070 container died 11894d490054d8a4d6f2a29ea6a46f821afc75a2b25a1f901254932c292cdcb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T18:15:15.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:15 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:15.251+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:15.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:15 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:15.251+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:15.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:15 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:15.251+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:15.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:15 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:15.251+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:15.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:15 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:15.251+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:15.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:15 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:15.251+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:15.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:15 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:15.251+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.927725+0000 front 2024-04-26T18:14:45.927818+0000 (oldest deadline 2024-04-26T18:15:11.227394+0000) 2024-04-26T18:15:15.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.9.service' 2024-04-26T18:15:15.866 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:15 smithi031 podman[100442]: 2024-04-26 18:15:15.782962473 +0000 UTC m=+0.494345994 container remove 11894d490054d8a4d6f2a29ea6a46f821afc75a2b25a1f901254932c292cdcb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-26T18:15:15.866 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:15 smithi031 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.9.service: Deactivated successfully. 2024-04-26T18:15:15.866 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:15 smithi031 systemd[1]: Stopped Ceph osd.9 for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T18:15:15.866 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 26 18:15:15 smithi031 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.9.service: Consumed 6min 20.409s CPU time. 2024-04-26T18:15:15.882 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:15:15.882 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-26T18:15:15.882 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-26T18:15:15.882 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.10 2024-04-26T18:15:16.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:15 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:15.539+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:16.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:15 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:15.539+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:16.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:15 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:15.539+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:16.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:15 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:15.539+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:16.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:15 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:15.539+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:16.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:15 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:15.539+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:16.200 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:15 smithi097 systemd[1]: Stopping Ceph osd.10 for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:15:16.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:16 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:16.200+0000 7f4cd781b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:15:16.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:16 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:16.200+0000 7f4cd781b700 -1 osd.10 116 *** Got signal Terminated *** 2024-04-26T18:15:16.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:16 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:16.200+0000 7f4cd781b700 -1 osd.10 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T18:15:16.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:16 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:16.236+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:16.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:16 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:16.236+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:16.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:16 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:16.236+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:16.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:16 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:16.236+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:16.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:16 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:16.236+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:16.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:16 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:16.236+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:16.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:16 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:16.236+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.927725+0000 front 2024-04-26T18:14:45.927818+0000 (oldest deadline 2024-04-26T18:15:11.227394+0000) 2024-04-26T18:15:17.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:16 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:16.519+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:17.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:16 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:16.519+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:17.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:16 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:16.519+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:17.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:16 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:16.519+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:17.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:16 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:16.519+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:17.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:16 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:16.519+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:17.636 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:17 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:17.211+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:17.637 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:17 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:17.211+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:17.637 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:17 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:17.211+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:17.637 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:17 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:17.211+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:17.637 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:17 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:17.211+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:17.637 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:17 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:17.211+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:17.637 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:17 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:17.211+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.927725+0000 front 2024-04-26T18:14:45.927818+0000 (oldest deadline 2024-04-26T18:15:11.227394+0000) 2024-04-26T18:15:17.767 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:17 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:17.478+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:17.768 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:17 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:17.478+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:17.768 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:17 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:17.478+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:17.768 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:17 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:17.478+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:17.768 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:17 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:17.478+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:17.768 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:17 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:17.478+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:18.589 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:18 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:18.173+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:18.589 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:18 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:18.173+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:18.589 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:18 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:18.173+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:18.589 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:18 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:18.173+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:18.589 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:18 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:18.173+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:18.589 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:18 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:18.173+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:18.589 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:18 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:18.173+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.927725+0000 front 2024-04-26T18:14:45.927818+0000 (oldest deadline 2024-04-26T18:15:11.227394+0000) 2024-04-26T18:15:19.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:18 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:18.525+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:19.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:18 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:18.525+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:19.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:18 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:18.525+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:19.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:18 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:18.525+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:19.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:18 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:18.525+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:19.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:18 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:18.525+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:19.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:19 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:19.200+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:19.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:19 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:19.200+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:19.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:19 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:19.200+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:19.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:19 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:19.200+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:19.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:19 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:19.200+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:19.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:19 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:19.200+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:19.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:19 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:19.200+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.927725+0000 front 2024-04-26T18:14:45.927818+0000 (oldest deadline 2024-04-26T18:15:11.227394+0000) 2024-04-26T18:15:20.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:19 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:19.531+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:20.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:19 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:19.531+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:20.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:19 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:19.531+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:20.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:19 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:19.531+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:20.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:19 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:19.531+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:20.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:19 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:19.531+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:20.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:20 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:20.247+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:59.320250+0000 front 2024-04-26T18:13:59.320018+0000 (oldest deadline 2024-04-26T18:14:24.620107+0000) 2024-04-26T18:15:20.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:20 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:20.247+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:06.320974+0000 front 2024-04-26T18:14:06.321105+0000 (oldest deadline 2024-04-26T18:14:29.221204+0000) 2024-04-26T18:15:20.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:20 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:20.247+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.422753+0000 front 2024-04-26T18:14:14.422675+0000 (oldest deadline 2024-04-26T18:14:38.522674+0000) 2024-04-26T18:15:20.634 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:20 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:20.247+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-26T18:14:20.223966+0000 front 2024-04-26T18:14:20.224039+0000 (oldest deadline 2024-04-26T18:14:45.523715+0000) 2024-04-26T18:15:20.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:20 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:20.247+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:31.325248+0000 front 2024-04-26T18:14:31.325262+0000 (oldest deadline 2024-04-26T18:14:56.525840+0000) 2024-04-26T18:15:20.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:20 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:20.247+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:36.526487+0000 front 2024-04-26T18:14:36.526392+0000 (oldest deadline 2024-04-26T18:15:02.426314+0000) 2024-04-26T18:15:20.635 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:20 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10[57827]: 2024-04-26T18:15:20.247+0000 7f4cd3e0f700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.927725+0000 front 2024-04-26T18:14:45.927818+0000 (oldest deadline 2024-04-26T18:15:11.227394+0000) 2024-04-26T18:15:21.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:20 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:20.568+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:21.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:20 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:20.568+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:21.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:20 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:20.568+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:21.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:20 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:20.568+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:21.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:20 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:20.568+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:21.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:20 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:20.568+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:22.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:21 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:21.597+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:22.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:21 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:21.597+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:22.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:21 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:21.597+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:22.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:21 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:21.597+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:22.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:21 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:21.597+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:22.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:21 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:21.597+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:22.066 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:21 smithi097 podman[71352]: 2024-04-26 18:15:21.81584203 +0000 UTC m=+5.743214539 container died cedae32b8a0e72aa503e1ff870b292af7bb3e0671d58a599cfc3261b98aee4db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-26T18:15:22.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:22 smithi097 podman[71352]: 2024-04-26 18:15:22.191217995 +0000 UTC m=+6.118590517 container cleanup cedae32b8a0e72aa503e1ff870b292af7bb3e0671d58a599cfc3261b98aee4db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T18:15:22.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:22 smithi097 bash[71352]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10 2024-04-26T18:15:22.734 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:22 smithi097 podman[71370]: 2024-04-26 18:15:22.417483322 +0000 UTC m=+0.600201908 container remove cedae32b8a0e72aa503e1ff870b292af7bb3e0671d58a599cfc3261b98aee4db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T18:15:23.297 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:22 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:22.625+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:23.297 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:22 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:22.625+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:23.297 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:22 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:22.625+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:23.297 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:22 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:22.625+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:23.297 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:22 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:22.625+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:23.297 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:22 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:22.625+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:23.297 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:22 smithi097 podman[71461]: 2024-04-26 18:15:22.734959587 +0000 UTC m=+0.094368821 container create 6f6a79cf0313451ea3fab5a921fe9a1e69ee422de99803edf04262e102e54c60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-26T18:15:23.297 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:22 smithi097 podman[71461]: 2024-04-26 18:15:22.659008608 +0000 UTC m=+0.018417855 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T18:15:23.297 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:22 smithi097 podman[71461]: 2024-04-26 18:15:22.962841057 +0000 UTC m=+0.322250299 container init 6f6a79cf0313451ea3fab5a921fe9a1e69ee422de99803edf04262e102e54c60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-26T18:15:23.297 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:22 smithi097 podman[71461]: 2024-04-26 18:15:22.965855922 +0000 UTC m=+0.325265157 container start 6f6a79cf0313451ea3fab5a921fe9a1e69ee422de99803edf04262e102e54c60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:15:23.297 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:23 smithi097 podman[71461]: 2024-04-26 18:15:23.037737451 +0000 UTC m=+0.397146695 container attach 6f6a79cf0313451ea3fab5a921fe9a1e69ee422de99803edf04262e102e54c60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-26T18:15:23.384 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:23 smithi097 podman[71461]: 2024-04-26 18:15:23.256942925 +0000 UTC m=+0.616352160 container died 6f6a79cf0313451ea3fab5a921fe9a1e69ee422de99803edf04262e102e54c60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-26T18:15:23.794 INFO:journalctl@ceph.osd.10.smithi097.stdout:Apr 26 18:15:23 smithi097 podman[71487]: 2024-04-26 18:15:23.774408631 +0000 UTC m=+0.512118303 container remove 6f6a79cf0313451ea3fab5a921fe9a1e69ee422de99803edf04262e102e54c60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T18:15:23.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.10.service' 2024-04-26T18:15:23.836 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:15:23.836 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-26T18:15:23.837 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-26T18:15:23.837 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.11 2024-04-26T18:15:23.885 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:23 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:23.617+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:23.886 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:23 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:23.617+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:23.886 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:23 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:23.617+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:23.886 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:23 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:23.617+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:23.886 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:23 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:23.617+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:23.886 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:23 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:23.617+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:23.886 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:23 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:23.617+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6820 osd.7 since back 2024-04-26T18:14:56.294335+0000 front 2024-04-26T18:14:56.294371+0000 (oldest deadline 2024-04-26T18:15:23.294764+0000) 2024-04-26T18:15:24.146 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:23 smithi186 systemd[1]: Stopping Ceph osd.11 for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:15:24.517 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:24.146+0000 7f2e236db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:15:24.517 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:24.146+0000 7f2e236db700 -1 osd.11 116 *** Got signal Terminated *** 2024-04-26T18:15:24.517 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:24.146+0000 7f2e236db700 -1 osd.11 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T18:15:25.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:24.574+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:25.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:24.574+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:25.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:24.574+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:25.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:24.574+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:25.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:24.574+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:25.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:24.574+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:25.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:24 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:24.574+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6820 osd.7 since back 2024-04-26T18:14:56.294335+0000 front 2024-04-26T18:14:56.294371+0000 (oldest deadline 2024-04-26T18:15:23.294764+0000) 2024-04-26T18:15:26.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:25.573+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:26.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:25.573+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:26.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:25.573+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:26.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:25.573+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:26.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:25.573+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:26.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:25.573+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:26.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:25 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:25.573+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6820 osd.7 since back 2024-04-26T18:14:56.294335+0000 front 2024-04-26T18:14:56.294371+0000 (oldest deadline 2024-04-26T18:15:23.294764+0000) 2024-04-26T18:15:27.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:26 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:26.604+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:27.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:26 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:26.604+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:27.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:26 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:26.604+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:27.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:26 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:26.604+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:27.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:26 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:26.604+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:27.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:26 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:26.604+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:27.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:26 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:26.604+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6820 osd.7 since back 2024-04-26T18:14:56.294335+0000 front 2024-04-26T18:14:56.294371+0000 (oldest deadline 2024-04-26T18:15:23.294764+0000) 2024-04-26T18:15:28.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:27 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:27.640+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:28.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:27 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:27.640+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:28.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:27 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:27.640+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:28.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:27 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:27.640+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:28.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:27 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:27.640+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:28.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:27 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:27.640+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:28.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:27 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:27.640+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6820 osd.7 since back 2024-04-26T18:14:56.294335+0000 front 2024-04-26T18:14:56.294371+0000 (oldest deadline 2024-04-26T18:15:23.294764+0000) 2024-04-26T18:15:29.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:28 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:28.685+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-26T18:13:56.785297+0000 front 2024-04-26T18:13:56.785282+0000 (oldest deadline 2024-04-26T18:14:22.085293+0000) 2024-04-26T18:15:29.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:28 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:28.685+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6804 osd.1 since back 2024-04-26T18:14:04.386642+0000 front 2024-04-26T18:14:04.386515+0000 (oldest deadline 2024-04-26T18:14:30.286478+0000) 2024-04-26T18:15:29.017 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:28 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:28.685+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T18:14:14.387816+0000 front 2024-04-26T18:14:14.387505+0000 (oldest deadline 2024-04-26T18:14:37.188165+0000) 2024-04-26T18:15:29.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:28 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:28.685+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6812 osd.4 since back 2024-04-26T18:14:32.391075+0000 front 2024-04-26T18:14:32.391038+0000 (oldest deadline 2024-04-26T18:14:57.091001+0000) 2024-04-26T18:15:29.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:28 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:28.685+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T18:14:37.091419+0000 front 2024-04-26T18:14:37.091440+0000 (oldest deadline 2024-04-26T18:15:01.791363+0000) 2024-04-26T18:15:29.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:28 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:28.685+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-26T18:14:45.792993+0000 front 2024-04-26T18:14:45.793059+0000 (oldest deadline 2024-04-26T18:15:11.692937+0000) 2024-04-26T18:15:29.018 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:28 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11[58346]: 2024-04-26T18:15:28.685+0000 7f2e1fccf700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.97:6820 osd.7 since back 2024-04-26T18:14:56.294335+0000 front 2024-04-26T18:14:56.294371+0000 (oldest deadline 2024-04-26T18:15:23.294764+0000) 2024-04-26T18:15:30.139 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:29 smithi186 podman[72289]: 2024-04-26 18:15:29.874404491 +0000 UTC m=+5.841880319 container died 5413abcf1060dfaf57783cde37bdd6384e655120bee97555da4dfd2a11eea293 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-26T18:15:30.517 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:30 smithi186 podman[72289]: 2024-04-26 18:15:30.221615878 +0000 UTC m=+6.189091697 container cleanup 5413abcf1060dfaf57783cde37bdd6384e655120bee97555da4dfd2a11eea293 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-26T18:15:30.517 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:30 smithi186 bash[72289]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11 2024-04-26T18:15:30.881 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:30 smithi186 podman[72307]: 2024-04-26 18:15:30.539724495 +0000 UTC m=+0.664237403 container remove 5413abcf1060dfaf57783cde37bdd6384e655120bee97555da4dfd2a11eea293 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, ceph=True) 2024-04-26T18:15:31.184 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:30 smithi186 podman[72396]: 2024-04-26 18:15:30.882373649 +0000 UTC m=+0.112432914 container create ed307803ac28e2cc3c43ef1641e8aef0ea28a23c3d842bb1dc4cc9d662a5dd0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T18:15:31.184 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:30 smithi186 podman[72396]: 2024-04-26 18:15:30.788039717 +0000 UTC m=+0.018099000 image pull 3a6bde214c5b0b19fc991f4bf5457f79505becbb4a5f8542663f01cefd0dd555 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc 2024-04-26T18:15:31.184 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:31 smithi186 podman[72396]: 2024-04-26 18:15:31.109771551 +0000 UTC m=+0.339830817 container init ed307803ac28e2cc3c43ef1641e8aef0ea28a23c3d842bb1dc4cc9d662a5dd0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T18:15:31.184 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:31 smithi186 podman[72396]: 2024-04-26 18:15:31.112788913 +0000 UTC m=+0.342848187 container start ed307803ac28e2cc3c43ef1641e8aef0ea28a23c3d842bb1dc4cc9d662a5dd0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True) 2024-04-26T18:15:31.184 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:31 smithi186 podman[72396]: 2024-04-26 18:15:31.18468525 +0000 UTC m=+0.414744519 container attach ed307803ac28e2cc3c43ef1641e8aef0ea28a23c3d842bb1dc4cc9d662a5dd0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-26T18:15:31.517 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 18:15:31 smithi186 podman[72396]: 2024-04-26 18:15:31.407060666 +0000 UTC m=+0.637119925 container died ed307803ac28e2cc3c43ef1641e8aef0ea28a23c3d842bb1dc4cc9d662a5dd0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:15:31.990 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@osd.11.service' 2024-04-26T18:15:32.028 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:15:32.028 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-26T18:15:32.028 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-26T18:15:32.029 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.a 2024-04-26T18:15:32.310 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 18:15:32 smithi031 systemd[1]: Stopping Ceph mds.a for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:15:32.810 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 18:15:32 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-a[69511]: 2024-04-26T18:15:32.316+0000 7f6f6c034700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:15:32.810 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 18:15:32 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-a[69511]: 2024-04-26T18:15:32.316+0000 7f6f6c034700 -1 mds.a *** got signal Terminated *** 2024-04-26T18:15:36.200 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 18:15:35 smithi031 podman[100520]: 2024-04-26 18:15:35.949126019 +0000 UTC m=+3.719372668 container died 1647cb60c9742be5339601fff6fcb2fbcdebca9a053d6a23b0351b8f6902dfc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T18:15:36.560 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 18:15:36 smithi031 podman[100520]: 2024-04-26 18:15:36.300043756 +0000 UTC m=+4.070290404 container cleanup 1647cb60c9742be5339601fff6fcb2fbcdebca9a053d6a23b0351b8f6902dfc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-26T18:15:36.560 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 26 18:15:36 smithi031 bash[100520]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-a 2024-04-26T18:15:36.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.a.service' 2024-04-26T18:15:36.768 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:15:36.768 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-26T18:15:36.769 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-26T18:15:36.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.d 2024-04-26T18:15:37.310 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 18:15:36 smithi031 systemd[1]: Stopping Ceph mds.d for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:15:37.310 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 18:15:37 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-d[68217]: 2024-04-26T18:15:37.077+0000 7fe6ecf15700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:15:37.310 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 18:15:37 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-d[68217]: 2024-04-26T18:15:37.077+0000 7fe6ecf15700 -1 mds.d *** got signal Terminated *** 2024-04-26T18:15:41.352 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 18:15:41 smithi031 podman[100645]: 2024-04-26 18:15:41.083533598 +0000 UTC m=+4.084441973 container died d63a664ed7d0df63aa0905e4999a2cf33362ba55e5df3ae79c87d75b08918792 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-26T18:15:41.352 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 18:15:41 smithi031 podman[100645]: 2024-04-26 18:15:41.353097411 +0000 UTC m=+4.354005786 container cleanup d63a664ed7d0df63aa0905e4999a2cf33362ba55e5df3ae79c87d75b08918792 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-26T18:15:41.620 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 26 18:15:41 smithi031 bash[100645]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-d 2024-04-26T18:15:41.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.d.service' 2024-04-26T18:15:41.819 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:15:41.819 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-26T18:15:41.819 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-26T18:15:41.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.g 2024-04-26T18:15:42.131 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 18:15:41 smithi031 systemd[1]: Stopping Ceph mds.g for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:15:42.560 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 18:15:42 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-g[69043]: 2024-04-26T18:15:42.131+0000 7f8a6f7d9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:15:42.560 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 18:15:42 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-g[69043]: 2024-04-26T18:15:42.131+0000 7f8a6f7d9700 -1 mds.g *** got signal Terminated *** 2024-04-26T18:15:43.682 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 18:15:43 smithi031 podman[100766]: 2024-04-26 18:15:43.415145179 +0000 UTC m=+1.374167287 container died c4605ee34e42730939ce4424e2e6d8b1bbea89b1d5b25bd746129b315ba334ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , ceph=True) 2024-04-26T18:15:44.057 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 18:15:43 smithi031 podman[100766]: 2024-04-26 18:15:43.798502858 +0000 UTC m=+1.757524968 container cleanup c4605ee34e42730939ce4424e2e6d8b1bbea89b1d5b25bd746129b315ba334ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-26T18:15:44.057 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 26 18:15:43 smithi031 bash[100766]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-g 2024-04-26T18:15:44.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.g.service' 2024-04-26T18:15:44.269 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:15:44.269 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-26T18:15:44.269 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-26T18:15:44.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.j 2024-04-26T18:15:44.355 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 18:15:44 smithi031 systemd[1]: Stopping Ceph mds.j for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:15:44.901 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 18:15:44 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-j[67792]: 2024-04-26T18:15:44.643+0000 7f8008767700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:15:44.901 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 18:15:44 smithi031 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-j[67792]: 2024-04-26T18:15:44.643+0000 7f8008767700 -1 mds.j *** got signal Terminated *** 2024-04-26T18:15:46.060 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 18:15:45 smithi031 podman[100897]: 2024-04-26 18:15:45.801729811 +0000 UTC m=+1.297982033 container died e44088701a82347d470f592e4c1ba965d4c6e194ac131b787a28ac7d3a8cbb1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0) 2024-04-26T18:15:46.436 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 18:15:46 smithi031 podman[100897]: 2024-04-26 18:15:46.093953755 +0000 UTC m=+1.590205979 container cleanup e44088701a82347d470f592e4c1ba965d4c6e194ac131b787a28ac7d3a8cbb1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-26T18:15:46.436 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 18:15:46 smithi031 bash[100897]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-j 2024-04-26T18:15:46.436 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 26 18:15:46 smithi031 podman[100912]: 2024-04-26 18:15:46.319951692 +0000 UTC m=+0.513154271 container remove e44088701a82347d470f592e4c1ba965d4c6e194ac131b787a28ac7d3a8cbb1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-26T18:15:46.455 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.j.service' 2024-04-26T18:15:46.489 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:15:46.490 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-26T18:15:46.490 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-26T18:15:46.490 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.b 2024-04-26T18:15:46.842 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:15:46 smithi097 systemd[1]: Stopping Ceph mds.b for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:15:47.134 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:15:46 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[67844]: 2024-04-26T18:15:46.841+0000 7fe133684700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:15:47.134 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:15:46 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b[67844]: 2024-04-26T18:15:46.841+0000 7fe133684700 -1 mds.b *** got signal Terminated *** 2024-04-26T18:15:49.134 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:15:49 smithi097 podman[71565]: 2024-04-26 18:15:49.07299607 +0000 UTC m=+2.388582308 container died f9ca1792ea8814f8d198be5cf29b068bc73313db17d7c48eeebe786df3a84ab0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T18:15:49.634 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:15:49 smithi097 podman[71565]: 2024-04-26 18:15:49.417340889 +0000 UTC m=+2.732927125 container cleanup f9ca1792ea8814f8d198be5cf29b068bc73313db17d7c48eeebe786df3a84ab0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:15:49.634 INFO:journalctl@ceph.mds.b.smithi097.stdout:Apr 26 18:15:49 smithi097 bash[71565]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-b 2024-04-26T18:15:49.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.b.service' 2024-04-26T18:15:49.863 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:15:49.864 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-26T18:15:49.864 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-26T18:15:49.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.e 2024-04-26T18:15:50.384 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 18:15:49 smithi097 systemd[1]: Stopping Ceph mds.e for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:15:50.384 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 18:15:50 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-e[62438]: 2024-04-26T18:15:50.178+0000 7faa1b80f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:15:50.384 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 18:15:50 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-e[62438]: 2024-04-26T18:15:50.178+0000 7faa1b80f700 -1 mds.e *** got signal Terminated *** 2024-04-26T18:15:53.384 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 18:15:53 smithi097 podman[71698]: 2024-04-26 18:15:53.068898599 +0000 UTC m=+2.984484971 container died c07ebece9e02cbfa6b7e5299f34c35da50d46260e95ebd9201dcb99d28309f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, GIT_BRANCH=HEAD) 2024-04-26T18:15:53.880 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 18:15:53 smithi097 podman[71698]: 2024-04-26 18:15:53.638143073 +0000 UTC m=+3.553729455 container cleanup c07ebece9e02cbfa6b7e5299f34c35da50d46260e95ebd9201dcb99d28309f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-26T18:15:53.880 INFO:journalctl@ceph.mds.e.smithi097.stdout:Apr 26 18:15:53 smithi097 bash[71698]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-e 2024-04-26T18:15:54.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.e.service' 2024-04-26T18:15:54.060 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:15:54.061 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-26T18:15:54.061 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-26T18:15:54.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.h 2024-04-26T18:15:54.384 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 18:15:54 smithi097 systemd[1]: Stopping Ceph mds.h for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:15:54.885 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 18:15:54 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-h[62756]: 2024-04-26T18:15:54.424+0000 7fea058ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:15:54.885 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 18:15:54 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-h[62756]: 2024-04-26T18:15:54.424+0000 7fea058ba700 -1 mds.h *** got signal Terminated *** 2024-04-26T18:15:57.376 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 18:15:57 smithi097 podman[71820]: 2024-04-26 18:15:57.06047123 +0000 UTC m=+2.787627471 container died 46498fc53496f589a6479c9e238aa338020e3a3b495c53607979146193715971 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-26T18:15:57.634 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 18:15:57 smithi097 podman[71820]: 2024-04-26 18:15:57.467257443 +0000 UTC m=+3.194413690 container cleanup 46498fc53496f589a6479c9e238aa338020e3a3b495c53607979146193715971 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-26T18:15:57.635 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 18:15:57 smithi097 bash[71820]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-h 2024-04-26T18:15:57.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.h.service' 2024-04-26T18:15:57.923 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 18:15:57 smithi097 podman[71834]: 2024-04-26 18:15:57.664645504 +0000 UTC m=+0.602061346 container remove 46498fc53496f589a6479c9e238aa338020e3a3b495c53607979146193715971 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-26T18:15:57.923 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 18:15:57 smithi097 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.h.service: Deactivated successfully. 2024-04-26T18:15:57.923 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 18:15:57 smithi097 systemd[1]: Stopped Ceph mds.h for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T18:15:57.923 INFO:journalctl@ceph.mds.h.smithi097.stdout:Apr 26 18:15:57 smithi097 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.h.service: Consumed 15.877s CPU time. 2024-04-26T18:15:57.934 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:15:57.934 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-26T18:15:57.934 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-26T18:15:57.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.k 2024-04-26T18:15:58.253 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 18:15:58 smithi097 systemd[1]: Stopping Ceph mds.k for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:15:58.634 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 18:15:58 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-k[62118]: 2024-04-26T18:15:58.253+0000 7fdaed382700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:15:58.634 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 18:15:58 smithi097 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-k[62118]: 2024-04-26T18:15:58.253+0000 7fdaed382700 -1 mds.k *** got signal Terminated *** 2024-04-26T18:16:00.884 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 18:16:00 smithi097 podman[71942]: 2024-04-26 18:16:00.62187217 +0000 UTC m=+2.469184415 container died 5f9a7c1d519c0a43ed2c1c1fc832f928bb46791ff41f4eeec4fcac62ce10346e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:16:01.384 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 18:16:01 smithi097 podman[71942]: 2024-04-26 18:16:01.062872163 +0000 UTC m=+2.910184418 container cleanup 5f9a7c1d519c0a43ed2c1c1fc832f928bb46791ff41f4eeec4fcac62ce10346e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T18:16:01.384 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 18:16:01 smithi097 bash[71942]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-k 2024-04-26T18:16:01.384 INFO:journalctl@ceph.mds.k.smithi097.stdout:Apr 26 18:16:01 smithi097 podman[71956]: 2024-04-26 18:16:01.221983255 +0000 UTC m=+0.599523325 container remove 5f9a7c1d519c0a43ed2c1c1fc832f928bb46791ff41f4eeec4fcac62ce10346e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-26T18:16:01.440 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.k.service' 2024-04-26T18:16:01.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:16:01.471 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-26T18:16:01.471 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-26T18:16:01.471 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.c 2024-04-26T18:16:01.767 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:16:01 smithi186 systemd[1]: Stopping Ceph mds.c for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:16:02.031 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:16:01 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[68799]: 2024-04-26T18:16:01.871+0000 7f8a71c7e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:16:02.031 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:16:01 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c[68799]: 2024-04-26T18:16:01.871+0000 7f8a71c7e700 -1 mds.c *** got signal Terminated *** 2024-04-26T18:16:05.489 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:16:05 smithi186 podman[72500]: 2024-04-26 18:16:05.049722291 +0000 UTC m=+3.387297596 container died d796aef9d2ae6eaff534cd50ccd9706563816a9c099056c412270a26fe34abb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.schema-version=1.0) 2024-04-26T18:16:05.767 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:16:05 smithi186 podman[72500]: 2024-04-26 18:16:05.563120708 +0000 UTC m=+3.900696004 container cleanup d796aef9d2ae6eaff534cd50ccd9706563816a9c099056c412270a26fe34abb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-26T18:16:05.767 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 18:16:05 smithi186 bash[72500]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-c 2024-04-26T18:16:05.981 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.c.service' 2024-04-26T18:16:06.015 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:16:06.015 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-26T18:16:06.015 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-26T18:16:06.015 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.f 2024-04-26T18:16:06.517 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 18:16:06 smithi186 systemd[1]: Stopping Ceph mds.f for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:16:06.517 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 18:16:06 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-f[63067]: 2024-04-26T18:16:06.333+0000 7f0923881700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:16:06.517 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 18:16:06 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-f[63067]: 2024-04-26T18:16:06.333+0000 7f0923881700 -1 mds.f *** got signal Terminated *** 2024-04-26T18:16:09.334 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 18:16:09 smithi186 podman[72634]: 2024-04-26 18:16:09.070959021 +0000 UTC m=+2.839954259 container died ad71e0cc5520dda922246035a6cfc17ef7e46020e7ad26856615543de5a3ee38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T18:16:09.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 18:16:09 smithi186 podman[72634]: 2024-04-26 18:16:09.417068548 +0000 UTC m=+3.186063784 container cleanup ad71e0cc5520dda922246035a6cfc17ef7e46020e7ad26856615543de5a3ee38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-26T18:16:09.676 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 18:16:09 smithi186 bash[72634]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-f 2024-04-26T18:16:09.808 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.f.service' 2024-04-26T18:16:09.840 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:16:09.840 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-26T18:16:09.840 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-26T18:16:09.840 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.i 2024-04-26T18:16:10.017 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 18:16:09 smithi186 systemd[1]: Stopping Ceph mds.i for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:16:10.517 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 18:16:10 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-i[63686]: 2024-04-26T18:16:10.144+0000 7f828f2ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:16:10.517 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 18:16:10 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-i[63686]: 2024-04-26T18:16:10.144+0000 7f828f2ba700 -1 mds.i *** got signal Terminated *** 2024-04-26T18:16:13.268 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 18:16:13 smithi186 podman[72754]: 2024-04-26 18:16:13.065904113 +0000 UTC m=+3.014313192 container died 44a6244684343aaf6ee6b1bfd5adafe29b7d40cbea2e642d07da70f94d39d7ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:16:13.588 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 18:16:13 smithi186 podman[72754]: 2024-04-26 18:16:13.379219999 +0000 UTC m=+3.327629075 container cleanup 44a6244684343aaf6ee6b1bfd5adafe29b7d40cbea2e642d07da70f94d39d7ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-fixes-bc94a2a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T18:16:13.588 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 18:16:13 smithi186 bash[72754]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-i 2024-04-26T18:16:13.808 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.i.service' 2024-04-26T18:16:13.840 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:16:13.841 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-26T18:16:13.841 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-26T18:16:13.841 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.l 2024-04-26T18:16:14.182 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 18:16:13 smithi186 systemd[1]: Stopping Ceph mds.l for 7e53e050-03f4-11ef-bc93-c7b262605968... 2024-04-26T18:16:14.517 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 18:16:14 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-l[62755]: 2024-04-26T18:16:14.181+0000 7f0d939b0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T18:16:14.517 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 18:16:14 smithi186 ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-l[62755]: 2024-04-26T18:16:14.181+0000 7f0d939b0700 -1 mds.l *** got signal Terminated *** 2024-04-26T18:16:18.017 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 18:16:17 smithi186 podman[72878]: 2024-04-26 18:16:17.748474962 +0000 UTC m=+3.689920868 container died 883a1e6b36df35e25c9e204b54338f4a8a38c06ddfed6640bfb04817c3d96418 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:16:18.375 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 18:16:18 smithi186 podman[72878]: 2024-04-26 18:16:18.124525825 +0000 UTC m=+4.065971723 container cleanup 883a1e6b36df35e25c9e204b54338f4a8a38c06ddfed6640bfb04817c3d96418 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:16:18.375 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 18:16:18 smithi186 bash[72878]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-l 2024-04-26T18:16:18.605 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.l.service' 2024-04-26T18:16:18.625 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 18:16:18 smithi186 podman[72892]: 2024-04-26 18:16:18.375739611 +0000 UTC m=+0.621428353 container remove 883a1e6b36df35e25c9e204b54338f4a8a38c06ddfed6640bfb04817c3d96418 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2aadaf96fcd676c1b4c3580ed8dd1b2a67fa40f49767a9535ff3eae823efe0dc, name=ceph-7e53e050-03f4-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-fixes-bc94a2a) 2024-04-26T18:16:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 18:16:18 smithi186 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.l.service: Deactivated successfully. 2024-04-26T18:16:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 18:16:18 smithi186 systemd[1]: Stopped Ceph mds.l for 7e53e050-03f4-11ef-bc93-c7b262605968. 2024-04-26T18:16:18.626 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 18:16:18 smithi186 systemd[1]: ceph-7e53e050-03f4-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.144s CPU time. 2024-04-26T18:16:18.636 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T18:16:18.636 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-26T18:16:18.637 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T18:16:18.838 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T18:16:21.417 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T18:16:21.698 INFO:teuthology.orchestra.run.smithi097.stdout:Deleting cluster with fsid: 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T18:16:24.204 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T18:16:24.396 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T18:16:26.762 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T18:16:26.796 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T18:16:26.828 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T18:16:26.853 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-26T18:16:26.856 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895/remote/smithi031/crash 2024-04-26T18:16:26.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/crash -- . 2024-04-26T18:16:26.927 INFO:teuthology.orchestra.run.smithi031.stderr:tar: /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T18:16:26.927 INFO:teuthology.orchestra.run.smithi031.stderr:tar: Error is not recoverable: exiting now 2024-04-26T18:16:26.929 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895/remote/smithi097/crash 2024-04-26T18:16:26.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/crash -- . 2024-04-26T18:16:26.967 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T18:16:26.967 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2024-04-26T18:16:26.969 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895/remote/smithi186/crash 2024-04-26T18:16:26.970 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/crash -- . 2024-04-26T18:16:27.003 INFO:teuthology.orchestra.run.smithi186.stderr:tar: /var/lib/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T18:16:27.003 INFO:teuthology.orchestra.run.smithi186.stderr:tar: Error is not recoverable: exiting now 2024-04-26T18:16:27.005 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-26T18:16:27.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-26T18:16:27.052 INFO:tasks.cephadm:Compressing logs... 2024-04-26T18:16:27.053 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T18:16:27.055 DEBUG:teuthology.orchestra.run.smithi097:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T18:16:27.058 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T18:16:27.079 INFO:teuthology.orchestra.run.smithi031.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T18:16:27.079 INFO:teuthology.orchestra.run.smithi031.stderr:: No such file or directory 2024-04-26T18:16:27.080 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.log 2024-04-26T18:16:27.080 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T18:16:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-26T18:16:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-26T18:16:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-26T18:16:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-26T18:16:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-26T18:16:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-26T18:16:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-26T18:16:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mgr.x.log: /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T18:16:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-26T18:16:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.6.log: /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T18:16:27.090 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T18:16:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-26T18:16:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.a.log: /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-26T18:16:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.d.log: /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.6% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-26T18:16:27.091 INFO:teuthology.orchestra.run.smithi031.stderr: 94.5% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-26T18:16:27.093 INFO:teuthology.orchestra.run.smithi186.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T18:16:27.093 INFO:teuthology.orchestra.run.smithi186.stderr:: No such file or directory 2024-04-26T18:16:27.093 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.log 2024-04-26T18:16:27.094 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-26T18:16:27.094 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-26T18:16:27.094 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T18:16:27.094 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-26T18:16:27.094 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-26T18:16:27.094 INFO:teuthology.orchestra.run.smithi186.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T18:16:27.094 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-26T18:16:27.094 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-26T18:16:27.094 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-26T18:16:27.094 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mgr.z.log: 86.9% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T18:16:27.094 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T18:16:27.094 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T18:16:27.095 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-26T18:16:27.095 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T18:16:27.096 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T18:16:27.096 INFO:teuthology.orchestra.run.smithi097.stderr:: No such file or directory 2024-04-26T18:16:27.096 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.log 2024-04-26T18:16:27.096 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T18:16:27.096 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-26T18:16:27.097 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-26T18:16:27.097 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-26T18:16:27.097 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-26T18:16:27.097 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-26T18:16:27.097 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-26T18:16:27.097 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-26T18:16:27.097 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T18:16:27.097 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-26T18:16:27.097 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T18:16:27.097 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-26T18:16:27.097 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-26T18:16:27.097 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mon.b.log: /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.4% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-26T18:16:27.098 INFO:teuthology.orchestra.run.smithi031.stderr: 84.1% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T18:16:27.099 INFO:teuthology.orchestra.run.smithi186.stderr: 92.3% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-26T18:16:27.099 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-26T18:16:27.099 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T18:16:27.114 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-26T18:16:27.141 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.l.log: 91.0% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T18:16:27.141 INFO:teuthology.orchestra.run.smithi186.stderr: 94.4% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-26T18:16:27.147 INFO:teuthology.orchestra.run.smithi097.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T18:16:27.152 INFO:teuthology.orchestra.run.smithi097.stderr: 83.7% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T18:16:27.158 INFO:teuthology.orchestra.run.smithi097.stderr: 90.5% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T18:16:27.158 INFO:teuthology.orchestra.run.smithi097.stderr: 86.9% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T18:16:27.189 INFO:teuthology.orchestra.run.smithi031.stderr: 94.5% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-26T18:16:27.196 INFO:teuthology.orchestra.run.smithi031.stderr: 94.5% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-26T18:16:27.216 INFO:teuthology.orchestra.run.smithi031.stderr: 90.3% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T18:16:27.219 INFO:teuthology.orchestra.run.smithi031.stderr: 93.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T18:16:27.249 INFO:teuthology.orchestra.run.smithi031.stderr: 91.1% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T18:16:27.323 INFO:teuthology.orchestra.run.smithi186.stderr: 93.6% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-26T18:16:27.389 INFO:teuthology.orchestra.run.smithi097.stderr: 92.2% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-26T18:16:27.401 INFO:teuthology.orchestra.run.smithi097.stderr: 94.5% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-26T18:16:27.410 INFO:teuthology.orchestra.run.smithi097.stderr: 91.0% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T18:16:27.495 INFO:teuthology.orchestra.run.smithi186.stderr: 92.8% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-26T18:16:27.635 INFO:teuthology.orchestra.run.smithi031.stderr: 90.3% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-26T18:16:27.665 INFO:teuthology.orchestra.run.smithi186.stderr: 91.9% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-26T18:16:27.913 INFO:teuthology.orchestra.run.smithi097.stderr: 91.9% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-26T18:16:28.453 INFO:teuthology.orchestra.run.smithi031.stderr: 90.2% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-26T18:16:30.051 INFO:teuthology.orchestra.run.smithi186.stderr: 94.5% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-26T18:16:32.809 INFO:teuthology.orchestra.run.smithi097.stderr: 93.9% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-26T18:16:38.161 INFO:teuthology.orchestra.run.smithi097.stderr: 93.8% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-26T18:17:05.816 INFO:teuthology.orchestra.run.smithi186.stderr: 91.4% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-26T18:17:09.291 INFO:teuthology.orchestra.run.smithi031.stderr: 91.2% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-26T18:17:13.038 INFO:teuthology.orchestra.run.smithi031.stderr: 91.2% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-26T18:17:16.780 INFO:teuthology.orchestra.run.smithi186.stderr: 91.4% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-26T18:17:17.041 INFO:teuthology.orchestra.run.smithi186.stderr: 91.3% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-26T18:17:18.169 INFO:teuthology.orchestra.run.smithi097.stderr: 91.5% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-26T18:17:19.602 INFO:teuthology.orchestra.run.smithi097.stderr: 91.3% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-26T18:17:21.402 INFO:teuthology.orchestra.run.smithi186.stderr: 91.2% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-26T18:17:21.404 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T18:17:21.404 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m54.331s 2024-04-26T18:17:21.404 INFO:teuthology.orchestra.run.smithi186.stderr:user 3m10.275s 2024-04-26T18:17:21.404 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m6.152s 2024-04-26T18:17:25.144 INFO:teuthology.orchestra.run.smithi097.stderr: 91.4% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-26T18:17:29.612 INFO:teuthology.orchestra.run.smithi031.stderr: 91.2% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-26T18:17:31.526 INFO:teuthology.orchestra.run.smithi031.stderr: 91.1% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-26T18:17:31.528 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-04-26T18:17:31.528 INFO:teuthology.orchestra.run.smithi031.stderr:real 1m4.459s 2024-04-26T18:17:31.529 INFO:teuthology.orchestra.run.smithi031.stderr:user 3m28.990s 2024-04-26T18:17:31.529 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m5.917s 2024-04-26T18:17:40.105 INFO:teuthology.orchestra.run.smithi097.stderr: 91.2% -- replaced with /var/log/ceph/7e53e050-03f4-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-26T18:17:40.107 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-26T18:17:40.107 INFO:teuthology.orchestra.run.smithi097.stderr:real 1m13.035s 2024-04-26T18:17:40.107 INFO:teuthology.orchestra.run.smithi097.stderr:user 3m31.506s 2024-04-26T18:17:40.107 INFO:teuthology.orchestra.run.smithi097.stderr:sys 0m6.601s 2024-04-26T18:17:40.108 INFO:tasks.cephadm:Archiving logs... 2024-04-26T18:17:40.108 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895/remote/smithi031/log 2024-04-26T18:17:40.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T18:18:17.986 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895/remote/smithi097/log 2024-04-26T18:18:17.987 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T18:18:57.232 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895/remote/smithi186/log 2024-04-26T18:18:57.234 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T18:19:31.569 INFO:tasks.cephadm:Removing cluster... 2024-04-26T18:19:31.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 --force 2024-04-26T18:19:31.772 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T18:19:32.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 --force 2024-04-26T18:19:32.522 INFO:teuthology.orchestra.run.smithi097.stdout:Deleting cluster with fsid: 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T18:19:33.135 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7e53e050-03f4-11ef-bc93-c7b262605968 --force 2024-04-26T18:19:33.332 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: 7e53e050-03f4-11ef-bc93-c7b262605968 2024-04-26T18:19:33.865 INFO:tasks.cephadm:Removing cephadm ... 2024-04-26T18:19:33.865 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T18:19:33.885 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T18:19:33.900 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T18:19:33.916 INFO:tasks.cephadm:Teardown complete 2024-04-26T18:19:33.916 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T18:19:33.928 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T18:19:33.928 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T18:19:33.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T18:19:33.944 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T18:19:34.104 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T18:19:34.115 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T18:19:34.116 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T18:19:34.119 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T18:19:34.121 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T18:19:34.132 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-04-26T18:19:34.133 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-04-26T18:19:34.134 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-04-26T18:19:34.136 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T18:19:34.136 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-04-26T18:19:34.136 INFO:teuthology.orchestra.run.smithi097.stdout:^* hv01.front.sepia.ceph.com 3 6 377 0 +1142us[+1163us] +/- 65ms 2024-04-26T18:19:34.136 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 3 -266us[ -245us] +/- 57ms 2024-04-26T18:19:34.137 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 65 -859us[ -842us] +/- 63ms 2024-04-26T18:19:34.137 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T18:19:34.139 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T18:19:34.139 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-04-26T18:19:34.139 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv01.front.sepia.ceph.com 3 6 377 2 +1140us[+1154us] +/- 65ms 2024-04-26T18:19:34.139 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 3 -265us[ -252us] +/- 57ms 2024-04-26T18:19:34.139 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 -846us[ -833us] +/- 64ms 2024-04-26T18:19:34.139 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T18:19:34.140 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T18:19:34.140 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-04-26T18:19:34.140 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv01.front.sepia.ceph.com 3 6 377 1 +1147us[+1168us] +/- 65ms 2024-04-26T18:19:34.140 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 4 -263us[ -243us] +/- 57ms 2024-04-26T18:19:34.140 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 -856us[ -837us] +/- 64ms 2024-04-26T18:19:34.140 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T18:19:34.141 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T18:19:34.176 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T18:19:34.176 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T18:19:34.194 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T18:19:34.299 DEBUG:teuthology.orchestra.run.smithi097:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T18:19:34.372 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T18:19:34.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T18:19:34.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T18:19:34.476 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T18:19:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T18:19:34.503 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T18:19:34.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T18:19:34.532 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T18:19:34.547 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T18:19:34.709 INFO:teuthology.task.internal:Duration was 2502.897775 seconds 2024-04-26T18:19:34.710 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T18:19:34.721 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T18:19:34.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T18:19:34.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T18:19:34.727 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T18:19:34.764 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T18:19:34.766 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T18:19:34.772 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T18:19:35.175 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T18:19:35.175 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-04-26T18:19:35.175 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T18:19:35.201 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi097.front.sepia.ceph.com 2024-04-26T18:19:35.201 DEBUG:teuthology.orchestra.run.smithi097:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T18:19:35.226 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T18:19:35.227 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T18:19:35.254 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T18:19:35.254 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T18:19:35.257 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T18:19:35.270 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T18:19:35.323 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T18:19:35.323 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T18:19:35.331 DEBUG:teuthology.orchestra.run.smithi097:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T18:19:35.340 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T18:19:35.999 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T18:19:36.013 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T18:19:36.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T18:19:36.037 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T18:19:36.068 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T18:19:36.094 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T18:19:36.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T18:19:36.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T18:19:36.121 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T18:19:36.140 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-04-26T18:19:36.143 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = core 2024-04-26T18:19:36.157 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-04-26T18:19:36.178 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T18:19:36.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T18:19:36.209 DEBUG:teuthology.orchestra.run.smithi097:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T18:19:36.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T18:19:36.224 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T18:19:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T18:19:36.238 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T18:19:36.251 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T18:19:36.252 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895/remote/smithi031 2024-04-26T18:19:36.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T18:19:36.306 DEBUG:teuthology.misc:Transferring archived files from smithi097:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895/remote/smithi097 2024-04-26T18:19:36.306 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T18:19:36.344 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895/remote/smithi186 2024-04-26T18:19:36.344 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T18:19:36.379 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T18:19:36.380 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T18:19:36.382 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T18:19:36.385 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T18:19:36.428 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T18:19:36.440 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T18:19:36.440 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T18:19:36.452 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T18:19:36.452 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T18:19:36.454 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T18:19:36.457 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T18:19:36.467 INFO:teuthology.orchestra.run.smithi031.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 18:19 /home/ubuntu/cephtest 2024-04-26T18:19:36.469 INFO:teuthology.orchestra.run.smithi097.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 18:19 /home/ubuntu/cephtest 2024-04-26T18:19:36.482 INFO:teuthology.orchestra.run.smithi186.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 18:19 /home/ubuntu/cephtest 2024-04-26T18:19:36.483 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T18:19:36.625 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T18:19:36.652 INFO:teuthology.nuke:Checking targets against current locks 2024-04-26T18:19:36.689 DEBUG:teuthology.nuke:shortname: smithi031 2024-04-26T18:19:36.690 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T18:19:36.731 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 17:32:23.309686', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKchA0vnuabghk5dSl6KV2FE+12Bwgp17BdGnkCb0tTWzpmNszg1u7Q7j8iJHp4Bhb4V/7TrUY4qk0M6mr0npEo='} 2024-04-26T18:19:36.744 DEBUG:teuthology.nuke:shortname: smithi097 2024-04-26T18:19:36.744 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T18:19:36.800 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 17:32:23.313337', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKjBYkFsJUrWXkYzp5wxoasVGvBqd1aYVjEqZ3dBP5RYMX2/SmE5wNdWfKxA7COTjobTnJWa0WsLXiZ2h3+7M/s='} 2024-04-26T18:19:36.813 DEBUG:teuthology.nuke:shortname: smithi186 2024-04-26T18:19:36.813 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T18:19:36.863 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_17:24:23-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674895', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 17:32:23.311679', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAANIvgyydTKYZ2uktji9hiKZsB4kJq9b3D4reCbloMNIgZoe1zzYPTwotBeHaFW30GdAiHd2UNK0aEqUJ3ly3Q='} 2024-04-26T18:19:36.871 INFO:teuthology.orchestra.console.smithi031:Power off 2024-04-26T18:19:36.871 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T18:19:36.914 INFO:teuthology.orchestra.console.smithi097:Power off 2024-04-26T18:19:36.914 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T18:19:36.949 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2024-04-26T18:19:36.949 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T18:19:36.974 INFO:teuthology.orchestra.console.smithi186:Power off 2024-04-26T18:19:36.974 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T18:19:36.982 DEBUG:teuthology.orchestra.console.smithi097:power off output: Chassis Power Control: Down/Off 2024-04-26T18:19:36.982 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T18:19:37.009 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-04-26T18:19:37.021 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-04-26T18:19:37.021 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T18:19:37.032 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-04-26T18:19:37.075 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-26T18:19:41.010 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T18:19:42.285 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T18:19:42.293 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T18:19:42.387 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-04-26T18:19:42.394 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-04-26T18:19:42.402 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-26T18:19:46.390 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T18:19:46.399 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T18:19:46.408 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T18:19:46.501 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2024-04-26T18:19:46.510 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is off 2024-04-26T18:19:46.518 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-04-26T18:19:46.602 INFO:teuthology.orchestra.console.smithi031:Power off completed 2024-04-26T18:19:46.611 INFO:teuthology.orchestra.console.smithi097:Power off completed 2024-04-26T18:19:46.618 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-04-26T18:19:46.724 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} duration: 2502.897775411606 failure_reason: 'error during quiesce thrashing: Error parsing quiesce response: Expecting value: line 1 column 1 (char 0)' flavor: default owner: scheduled_leonidus@teuthology status: fail success: false 2024-04-26T18:19:46.725 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T18:19:46.812 INFO:teuthology.run:FAIL